May 25 11:17:38.186522 ip-10-0-128-146 systemd[1]: Starting Kubernetes Kubelet...
May 25 11:17:38.682817 ip-10-0-128-146 kubenswrapper[2560]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
May 25 11:17:38.682817 ip-10-0-128-146 kubenswrapper[2560]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
May 25 11:17:38.682817 ip-10-0-128-146 kubenswrapper[2560]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
May 25 11:17:38.682817 ip-10-0-128-146 kubenswrapper[2560]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
May 25 11:17:38.682817 ip-10-0-128-146 kubenswrapper[2560]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
May 25 11:17:38.684789 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.684692    2560 server.go:212] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
May 25 11:17:38.689322 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.689305    2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 25 11:17:38.689322 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.689321    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 25 11:17:38.689400 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.689325    2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 25 11:17:38.689400 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.689328    2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 25 11:17:38.689400 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.689331    2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 25 11:17:38.689400 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.689334    2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 25 11:17:38.689400 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.689337    2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 25 11:17:38.689400 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.689341    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 25 11:17:38.689400 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.689345    2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 25 11:17:38.689400 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.689347    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 25 11:17:38.689400 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.689350    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 25 11:17:38.689400 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.689353    2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 25 11:17:38.689400 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.689356    2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 25 11:17:38.689400 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.689365    2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 25 11:17:38.689400 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.689368    2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 25 11:17:38.689400 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.689371    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 25 11:17:38.689400 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.689374    2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 25 11:17:38.689400 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.689376    2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 25 11:17:38.689400 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.689379    2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 25 11:17:38.689400 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.689381    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 25 11:17:38.689400 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.689384    2560 feature_gate.go:328] unrecognized feature gate: Example
May 25 11:17:38.689400 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.689387    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 25 11:17:38.689881 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.689389    2560 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 25 11:17:38.689881 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.689392    2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 25 11:17:38.689881 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.689394    2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 25 11:17:38.689881 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.689397    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 25 11:17:38.689881 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.689399    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 25 11:17:38.689881 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.689402    2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 25 11:17:38.689881 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.689405    2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 25 11:17:38.689881 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.689407    2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 25 11:17:38.689881 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.689410    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 25 11:17:38.689881 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.689414    2560 feature_gate.go:328] unrecognized feature gate: NewOLM
May 25 11:17:38.689881 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.689416    2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 25 11:17:38.689881 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.689419    2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 25 11:17:38.689881 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.689421    2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 25 11:17:38.689881 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.689424    2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 25 11:17:38.689881 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.689427    2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 25 11:17:38.689881 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.689429    2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 25 11:17:38.689881 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.689432    2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 25 11:17:38.689881 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.689434    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 25 11:17:38.689881 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.689437    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 25 11:17:38.689881 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.689439    2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 25 11:17:38.690567 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.689442    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 25 11:17:38.690567 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.689444    2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 25 11:17:38.690567 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.689447    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 25 11:17:38.690567 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.689449    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 25 11:17:38.690567 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.689452    2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 25 11:17:38.690567 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.689455    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 25 11:17:38.690567 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.689458    2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 25 11:17:38.690567 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.689460    2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 25 11:17:38.690567 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.689463    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 25 11:17:38.690567 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.689469    2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 25 11:17:38.690567 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.689472    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 25 11:17:38.690567 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.689476    2560 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 25 11:17:38.690567 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.689478    2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 25 11:17:38.690567 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.689482    2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 25 11:17:38.690567 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.689486    2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 25 11:17:38.690567 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.689488    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 25 11:17:38.690567 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.689491    2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 25 11:17:38.690567 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.689495    2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 25 11:17:38.691154 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.689497    2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 25 11:17:38.691154 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.689500    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 25 11:17:38.691154 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.689502    2560 feature_gate.go:328] unrecognized feature gate: Example2
May 25 11:17:38.691154 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.689505    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 25 11:17:38.691154 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.689508    2560 feature_gate.go:328] unrecognized feature gate: DualReplica
May 25 11:17:38.691154 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.689511    2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 25 11:17:38.691154 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.689513    2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 25 11:17:38.691154 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.689516    2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 25 11:17:38.691154 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.689518    2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 25 11:17:38.691154 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.689522    2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 25 11:17:38.691154 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.689525    2560 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 25 11:17:38.691154 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.689528    2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 25 11:17:38.691154 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.689532    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 25 11:17:38.691154 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.689534    2560 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 25 11:17:38.691154 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.689537    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 25 11:17:38.691154 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.689540    2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 25 11:17:38.691154 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.689542    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 25 11:17:38.691154 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.689545    2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 25 11:17:38.691154 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.689547    2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 25 11:17:38.691154 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.689550    2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 25 11:17:38.691662 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.689553    2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 25 11:17:38.691662 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.689556    2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 25 11:17:38.691662 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.689558    2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 25 11:17:38.691662 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.689561    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 25 11:17:38.691662 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.689563    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 25 11:17:38.691662 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.689566    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 25 11:17:38.692933 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.692916    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 25 11:17:38.692933 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.692933    2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 25 11:17:38.693014 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.692938    2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 25 11:17:38.693014 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.692945    2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 25 11:17:38.693014 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.692952    2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 25 11:17:38.693014 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.692957    2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 25 11:17:38.693014 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.692961    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 25 11:17:38.693014 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.692965    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 25 11:17:38.693014 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.692971    2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 25 11:17:38.693014 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.692976    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 25 11:17:38.693014 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.692980    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 25 11:17:38.693014 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.692983    2560 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 25 11:17:38.693014 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.692988    2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 25 11:17:38.693014 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.692991    2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 25 11:17:38.693014 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.692994    2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 25 11:17:38.693014 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.692996    2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 25 11:17:38.693014 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.692999    2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 25 11:17:38.693014 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.693002    2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 25 11:17:38.693014 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.693005    2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 25 11:17:38.693014 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.693007    2560 feature_gate.go:328] unrecognized feature gate: Example
May 25 11:17:38.693014 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.693009    2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 25 11:17:38.693498 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.693012    2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 25 11:17:38.693498 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.693015    2560 feature_gate.go:328] unrecognized feature gate: NewOLM
May 25 11:17:38.693498 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.693017    2560 feature_gate.go:328] unrecognized feature gate: Example2
May 25 11:17:38.693498 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.693020    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 25 11:17:38.693498 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.693023    2560 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 25 11:17:38.693498 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.693026    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 25 11:17:38.693498 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.693029    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 25 11:17:38.693498 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.693032    2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 25 11:17:38.693498 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.693034    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 25 11:17:38.693498 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.693037    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 25 11:17:38.693498 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.693040    2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 25 11:17:38.693498 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.693043    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 25 11:17:38.693498 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.693045    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 25 11:17:38.693498 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.693048    2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 25 11:17:38.693498 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.693050    2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 25 11:17:38.693498 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.693053    2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 25 11:17:38.693498 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.693055    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 25 11:17:38.693498 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.693058    2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 25 11:17:38.693498 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.693060    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 25 11:17:38.693498 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.693062    2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 25 11:17:38.693993 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.693065    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 25 11:17:38.693993 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.693067    2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 25 11:17:38.693993 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.693070    2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 25 11:17:38.693993 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.693072    2560 feature_gate.go:328] unrecognized feature gate: DualReplica
May 25 11:17:38.693993 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.693077    2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 25 11:17:38.693993 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.693081    2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 25 11:17:38.693993 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.693085    2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 25 11:17:38.693993 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.693088    2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 25 11:17:38.693993 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.693090    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 25 11:17:38.693993 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.693093    2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 25 11:17:38.693993 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.693095    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 25 11:17:38.693993 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.693098    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 25 11:17:38.693993 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.693101    2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 25 11:17:38.693993 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.693103    2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 25 11:17:38.693993 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.693106    2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 25 11:17:38.693993 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.693109    2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 25 11:17:38.693993 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.693111    2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 25 11:17:38.693993 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.693114    2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 25 11:17:38.693993 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.693116    2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 25 11:17:38.694476 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.693119    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 25 11:17:38.694476 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.693122    2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 25 11:17:38.694476 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.693125    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 25 11:17:38.694476 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.693127    2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 25 11:17:38.694476 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.693130    2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 25 11:17:38.694476 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.693133    2560 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 25 11:17:38.694476 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.693135    2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 25 11:17:38.694476 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.693138    2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 25 11:17:38.694476 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.693141    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 25 11:17:38.694476 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.693144    2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 25 11:17:38.694476 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.693147    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 25 11:17:38.694476 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.693149    2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 25 11:17:38.694476 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.693151    2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 25 11:17:38.694476 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.693154    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 25 11:17:38.694476 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.693157    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 25 11:17:38.694476 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.693160    2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 25 11:17:38.694476 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.693162    2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 25 11:17:38.694476 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.693165    2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 25 11:17:38.694476 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.693169    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 25 11:17:38.694476 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.693171    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 25 11:17:38.694982 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.693174    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 25 11:17:38.694982 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.693177    2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 25 11:17:38.694982 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.693179    2560 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 25 11:17:38.694982 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.693181    2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 25 11:17:38.694982 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.693184    2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 25 11:17:38.694982 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.693186    2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 25 11:17:38.694982 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.693288    2560 flags.go:64] FLAG: --address="0.0.0.0"
May 25 11:17:38.694982 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.693299    2560 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
May 25 11:17:38.694982 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.693309    2560 flags.go:64] FLAG: --anonymous-auth="true"
May 25 11:17:38.694982 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.693314    2560 flags.go:64] FLAG: --application-metrics-count-limit="100"
May 25 11:17:38.694982 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.693320    2560 flags.go:64] FLAG: --authentication-token-webhook="false"
May 25 11:17:38.694982 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.693323    2560 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
May 25 11:17:38.694982 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.693328    2560 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
May 25 11:17:38.694982 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.693332    2560 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
May 25 11:17:38.694982 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.693336    2560 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
May 25 11:17:38.694982 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.693339    2560 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
May 25 11:17:38.694982 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.693343    2560 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
May 25 11:17:38.694982 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.693346    2560 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
May 25 11:17:38.694982 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.693349    2560 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
May 25 11:17:38.694982 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.693352    2560 flags.go:64] FLAG: --cgroup-root=""
May 25 11:17:38.694982 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.693355    2560 flags.go:64] FLAG: --cgroups-per-qos="true"
May 25 11:17:38.694982 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.693358    2560 flags.go:64] FLAG: --client-ca-file=""
May 25 11:17:38.694982 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.693360    2560 flags.go:64] FLAG: --cloud-config=""
May 25 11:17:38.695552 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.693363    2560 flags.go:64] FLAG: --cloud-provider="external"
May 25 11:17:38.695552 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.693366    2560 flags.go:64] FLAG: --cluster-dns="[]"
May 25 11:17:38.695552 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.693371    2560 flags.go:64] FLAG: --cluster-domain=""
May 25 11:17:38.695552 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.693374    2560 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
May 25 11:17:38.695552 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.693393    2560 flags.go:64] FLAG: --config-dir=""
May 25 11:17:38.695552 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.693397    2560 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
May 25 11:17:38.695552 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.693402    2560 flags.go:64] FLAG: --container-log-max-files="5"
May 25 11:17:38.695552 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.693406    2560 flags.go:64] FLAG: --container-log-max-size="10Mi"
May 25 11:17:38.695552 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.693410    2560 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
May 25 11:17:38.695552 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.693414    2560 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
May 25 11:17:38.695552 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.693418    2560 flags.go:64] FLAG: --containerd-namespace="k8s.io"
May 25 11:17:38.695552 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.693421    2560 flags.go:64] FLAG: --contention-profiling="false"
May 25 11:17:38.695552 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.693424    2560 flags.go:64] FLAG: --cpu-cfs-quota="true"
May 25 11:17:38.695552 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.693427    2560 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
May 25 11:17:38.695552 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.693431    2560 flags.go:64] FLAG: --cpu-manager-policy="none"
May 25 11:17:38.695552 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.693434    2560 flags.go:64] FLAG: --cpu-manager-policy-options=""
May 25 11:17:38.695552 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.693438    2560 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
May 25 11:17:38.695552 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.693441    2560 flags.go:64] FLAG: --enable-controller-attach-detach="true"
May 25 11:17:38.695552 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.693444    2560 flags.go:64] FLAG: --enable-debugging-handlers="true"
May 25 11:17:38.695552 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.693447    2560 flags.go:64] FLAG: --enable-load-reader="false"
May 25 11:17:38.695552 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.693450    2560 flags.go:64] FLAG: --enable-server="true"
May 25 11:17:38.695552 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.693453    2560 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
May 25 11:17:38.695552 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.693459    2560 flags.go:64] FLAG: --event-burst="100"
May 25 11:17:38.695552 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.693462    2560 flags.go:64] FLAG: --event-qps="50"
May 25 11:17:38.695552 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.693465    2560 flags.go:64] FLAG: --event-storage-age-limit="default=0"
May 25 11:17:38.696228 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.693469    2560 flags.go:64] FLAG: --event-storage-event-limit="default=0"
May 25 11:17:38.696228 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.693472    2560 flags.go:64] FLAG: --eviction-hard=""
May 25 11:17:38.696228 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.693475    2560 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
May 25 11:17:38.696228 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.693478    2560 flags.go:64] FLAG: --eviction-minimum-reclaim=""
May 25 11:17:38.696228 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.693481    2560 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
May 25 11:17:38.696228 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.693484    2560 flags.go:64] FLAG: --eviction-soft=""
May 25 11:17:38.696228 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.693487    2560 flags.go:64] FLAG: --eviction-soft-grace-period=""
May 25 11:17:38.696228 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.693490    2560 flags.go:64] FLAG: --exit-on-lock-contention="false"
May 25 11:17:38.696228 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.693493    2560 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
May 25 11:17:38.696228 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.693496    2560 flags.go:64] FLAG: --experimental-mounter-path=""
May 25 11:17:38.696228 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.693499    2560 flags.go:64] FLAG: --fail-cgroupv1="false"
May 25 11:17:38.696228 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.693502    2560 flags.go:64] FLAG: --fail-swap-on="true"
May 25 11:17:38.696228 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.693506    2560 flags.go:64] FLAG: --feature-gates=""
May 25 11:17:38.696228 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.693511    2560 flags.go:64] FLAG: --file-check-frequency="20s"
May 25 11:17:38.696228 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.693514    2560 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
May 25 11:17:38.696228 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.693517    2560 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
May 25 11:17:38.696228 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.693522    2560 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
May 25 11:17:38.696228 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.693525    2560 flags.go:64] FLAG: --healthz-port="10248"
May 25 11:17:38.696228 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.693528    2560 flags.go:64] FLAG: --help="false"
May 25 11:17:38.696228 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.693531    2560 flags.go:64] FLAG: --hostname-override="ip-10-0-128-146.ec2.internal"
May 25 11:17:38.696228 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.693534    2560 flags.go:64] FLAG: --housekeeping-interval="10s"
May 25 11:17:38.696228 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.693537    2560 flags.go:64] FLAG: --http-check-frequency="20s"
May 25 11:17:38.696228 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.693540    2560 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
May 25 11:17:38.696786 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.693544    2560 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
May 25 11:17:38.696786 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.693547    2560 flags.go:64] FLAG: --image-gc-high-threshold="85"
May 25 11:17:38.696786 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.693550    2560 flags.go:64] FLAG: --image-gc-low-threshold="80"
May 25 11:17:38.696786 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.693552    2560 flags.go:64] FLAG: --image-service-endpoint=""
May 25 11:17:38.696786 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.693555    2560 flags.go:64] FLAG: --kernel-memcg-notification="false"
May 25 11:17:38.696786 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.693558    2560 flags.go:64] FLAG: --kube-api-burst="100"
May 25 11:17:38.696786 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.693561    2560 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
May 25 11:17:38.696786 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.693564    2560 flags.go:64] FLAG: --kube-api-qps="50"
May 25 11:17:38.696786 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.693567    2560 flags.go:64] FLAG: --kube-reserved=""
May 25 11:17:38.696786 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.693570    2560 flags.go:64] FLAG: --kube-reserved-cgroup=""
May 25 11:17:38.696786 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.693572    2560 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
May 25 11:17:38.696786 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.693575    2560 flags.go:64] FLAG: --kubelet-cgroups=""
May 25 11:17:38.696786 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.693578    2560 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
May 25 11:17:38.696786 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.693581    2560 flags.go:64] FLAG: --lock-file=""
May 25 11:17:38.696786 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.693584    2560 flags.go:64] FLAG: --log-cadvisor-usage="false"
May 25 11:17:38.696786 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.693587    2560 flags.go:64] FLAG: --log-flush-frequency="5s"
May 25 11:17:38.696786 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.693590    2560 flags.go:64] FLAG: --log-json-info-buffer-size="0"
May 25 11:17:38.696786 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.693595    2560 flags.go:64] FLAG: --log-json-split-stream="false"
May 25 11:17:38.696786 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.693598    2560 flags.go:64] FLAG: --log-text-info-buffer-size="0"
May 25 11:17:38.696786 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.693601    2560 flags.go:64] FLAG: --log-text-split-stream="false"
May 25 11:17:38.696786 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.693604    2560 flags.go:64] FLAG: --logging-format="text"
May 25 11:17:38.696786 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.693606    2560 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
May 25 11:17:38.696786 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.693610    2560 flags.go:64] FLAG: --make-iptables-util-chains="true"
May 25 11:17:38.696786 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.693613    2560 flags.go:64] FLAG: --manifest-url=""
May 25 11:17:38.697372 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.693616    2560 flags.go:64] FLAG: --manifest-url-header=""
May 25 11:17:38.697372 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.693621    2560 flags.go:64] FLAG: --max-housekeeping-interval="15s"
May 25 11:17:38.697372 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.693625    2560 flags.go:64] FLAG: --max-open-files="1000000"
May 25 11:17:38.697372 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.693630    2560 flags.go:64] FLAG: --max-pods="110"
May 25 11:17:38.697372 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.693633    2560 flags.go:64] FLAG: --maximum-dead-containers="-1"
May 25 11:17:38.697372 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.693636    2560 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
May 25 11:17:38.697372 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.693639    2560 flags.go:64] FLAG: --memory-manager-policy="None"
May 25 11:17:38.697372 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.693641    2560 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
May 25 11:17:38.697372 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.693644    2560 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
May 25 11:17:38.697372 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.693647    2560 flags.go:64] FLAG: --node-ip="0.0.0.0"
May 25 11:17:38.697372 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.693650    2560 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
May 25 11:17:38.697372 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.693659    2560 flags.go:64] FLAG: --node-status-max-images="50"
May 25 11:17:38.697372 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.693662    2560 flags.go:64] FLAG: --node-status-update-frequency="10s"
May 25 11:17:38.697372 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.693665    2560 flags.go:64] FLAG: --oom-score-adj="-999"
May 25 11:17:38.697372 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.693668    2560 flags.go:64] FLAG: --pod-cidr=""
May 25 11:17:38.697372 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.693671    2560 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:afe88671bf163c42b467d159dc6f0b784dbe28b1ed44e171edc9c06fd99cc6fc"
May 25 11:17:38.697372 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.693676    2560 flags.go:64] FLAG: --pod-manifest-path=""
May 25 11:17:38.697372 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.693679    2560 flags.go:64] FLAG: --pod-max-pids="-1"
May 25 11:17:38.697372 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.693682    2560 flags.go:64] FLAG: --pods-per-core="0"
May 25 11:17:38.697372 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.693685    2560 flags.go:64] FLAG: --port="10250"
May 25 11:17:38.697372 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.693688    2560 flags.go:64] FLAG: --protect-kernel-defaults="false"
May 25 11:17:38.697372 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.693691    2560 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0ad2314f2b2e83b85"
May 25 11:17:38.697372 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.693694    2560 flags.go:64] FLAG: --qos-reserved=""
May 25 11:17:38.697372 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.693697    2560 flags.go:64] FLAG: --read-only-port="10255"
May 25 11:17:38.697986 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.693700    2560 flags.go:64] FLAG: --register-node="true"
May 25 11:17:38.697986 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.693703    2560 flags.go:64] FLAG: --register-schedulable="true"
May 25 11:17:38.697986 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.693707    2560 flags.go:64] FLAG: --register-with-taints=""
May 25 11:17:38.697986 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.693710    2560 flags.go:64] FLAG: --registry-burst="10"
May 25 11:17:38.697986 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.693713    2560 flags.go:64] FLAG: --registry-qps="5"
May 25 11:17:38.697986 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.693716    2560 flags.go:64] FLAG: --reserved-cpus=""
May 25 11:17:38.697986 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.693719    2560 flags.go:64] FLAG: --reserved-memory=""
May 25 11:17:38.697986 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.693723    2560 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
May 25 11:17:38.697986 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.693726    2560 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
May 25 11:17:38.697986 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.693729    2560 flags.go:64] FLAG: --rotate-certificates="false"
May 25 11:17:38.697986 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.693732    2560 flags.go:64] FLAG: --rotate-server-certificates="false"
May 25 11:17:38.697986 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.693736    2560 flags.go:64] FLAG: --runonce="false"
May 25 11:17:38.697986 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.693739    2560 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
May 25 11:17:38.697986 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.693742    2560 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
May 25 11:17:38.697986 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.693745    2560 flags.go:64] FLAG: --seccomp-default="false"
May 25 11:17:38.697986 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.693748    2560 flags.go:64] FLAG: --serialize-image-pulls="true"
May 25 11:17:38.697986 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.693751    2560 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
May 25 11:17:38.697986 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.693754    2560 flags.go:64] FLAG: --storage-driver-db="cadvisor"
May 25 11:17:38.697986 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.693757    2560 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
May 25 11:17:38.697986 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.693760    2560 flags.go:64] FLAG: --storage-driver-password="root"
May 25 11:17:38.697986 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.693762    2560 flags.go:64] FLAG: --storage-driver-secure="false"
May 25 11:17:38.697986 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.693766    2560 flags.go:64] FLAG: --storage-driver-table="stats"
May 25 11:17:38.697986 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.693769    2560 flags.go:64] FLAG: --storage-driver-user="root"
May 25 11:17:38.697986 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.693771    2560 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
May 25 11:17:38.697986 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.693774    2560 flags.go:64] FLAG: --sync-frequency="1m0s"
May 25 11:17:38.697986 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.693777    2560 flags.go:64] FLAG: --system-cgroups=""
May 25 11:17:38.698639 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.693780    2560 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
May 25 11:17:38.698639 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.693786    2560 flags.go:64] FLAG: --system-reserved-cgroup=""
May 25 11:17:38.698639 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.693788    2560 flags.go:64] FLAG: --tls-cert-file=""
May 25 11:17:38.698639 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.693791    2560 flags.go:64] FLAG: --tls-cipher-suites="[]"
May 25 11:17:38.698639 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.693796    2560 flags.go:64] FLAG: --tls-min-version=""
May 25 11:17:38.698639 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.693799    2560 flags.go:64] FLAG: --tls-private-key-file=""
May 25 11:17:38.698639 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.693802    2560 flags.go:64] FLAG: --topology-manager-policy="none"
May 25 11:17:38.698639 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.693805    2560 flags.go:64] FLAG: --topology-manager-policy-options=""
May 25 11:17:38.698639 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.693808    2560 flags.go:64] FLAG: --topology-manager-scope="container"
May 25 11:17:38.698639 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.693811    2560 flags.go:64] FLAG: --v="2"
May 25 11:17:38.698639 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.693815    2560 flags.go:64] FLAG: --version="false"
May 25 11:17:38.698639 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.693819    2560 flags.go:64] FLAG: --vmodule=""
May 25 11:17:38.698639 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.693824    2560 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
May 25 11:17:38.698639 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.693827    2560 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
May 25 11:17:38.698639 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.693929    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 25 11:17:38.698639 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.693933    2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 25 11:17:38.698639 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.693937    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 25 11:17:38.698639 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.693940    2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 25 11:17:38.698639 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.693944    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 25 11:17:38.698639 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.693947    2560 feature_gate.go:328] unrecognized feature gate: DualReplica
May 25 11:17:38.698639 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.693950    2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 25 11:17:38.698639 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.693953    2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 25 11:17:38.698639 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.693955    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 25 11:17:38.699191 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.693958    2560 feature_gate.go:328] unrecognized feature gate: NewOLM
May 25 11:17:38.699191 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.693960    2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 25 11:17:38.699191 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.693963    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 25 11:17:38.699191 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.693966    2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 25 11:17:38.699191 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.693969    2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 25 11:17:38.699191 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.693971    2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 25 11:17:38.699191 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.693974    2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 25 11:17:38.699191 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.693976    2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 25 11:17:38.699191 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.693979    2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 25 11:17:38.699191 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.693982    2560 feature_gate.go:328] unrecognized feature gate: Example
May 25 11:17:38.699191 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.693985    2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 25 11:17:38.699191 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.693987    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 25 11:17:38.699191 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.693990    2560 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 25 11:17:38.699191 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.693994    2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 25 11:17:38.699191 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.693996    2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 25 11:17:38.699191 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.693999    2560 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 25 11:17:38.699191 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.694002    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 25 11:17:38.699191 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.694005    2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 25 11:17:38.699191 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.694008    2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 25 11:17:38.699191 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.694010    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 25 11:17:38.699714 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.694013    2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 25 11:17:38.699714 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.694016    2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 25 11:17:38.699714 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.694018    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 25 11:17:38.699714 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.694021    2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 25 11:17:38.699714 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.694024    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 25 11:17:38.699714 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.694027    2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 25 11:17:38.699714 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.694029    2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 25 11:17:38.699714 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.694032    2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 25 11:17:38.699714 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.694036    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 25 11:17:38.699714 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.694038    2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 25 11:17:38.699714 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.694041    2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 25 11:17:38.699714 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.694044    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 25 11:17:38.699714 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.694047    2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 25 11:17:38.699714 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.694050    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 25 11:17:38.699714 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.694052    2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 25 11:17:38.699714 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.694056    2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 25 11:17:38.699714 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.694060    2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 25 11:17:38.699714 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.694063    2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 25 11:17:38.699714 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.694066    2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 25 11:17:38.700203 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.694068    2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 25 11:17:38.700203 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.694071    2560 feature_gate.go:328] unrecognized feature gate: Example2
May 25 11:17:38.700203 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.694074    2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 25 11:17:38.700203 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.694076    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 25 11:17:38.700203 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.694079    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 25 11:17:38.700203 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.694081    2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 25 11:17:38.700203 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.694084    2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 25 11:17:38.700203 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.694087    2560 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 25 11:17:38.700203 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.694089    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 25 11:17:38.700203 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.694092    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 25 11:17:38.700203 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.694094    2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 25 11:17:38.700203 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.694097    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 25 11:17:38.700203 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.694099    2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 25 11:17:38.700203 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.694102    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 25 11:17:38.700203 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.694105    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 25 11:17:38.700203 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.694110    2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 25 11:17:38.700203 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.694113    2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 25 11:17:38.700203 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.694115    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 25 11:17:38.700203 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.694121    2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 25 11:17:38.700203 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.694124    2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 25 11:17:38.700760 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.694127    2560 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 25 11:17:38.700760 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.694129    2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 25 11:17:38.700760 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.694133    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 25 11:17:38.700760 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.694135    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 25 11:17:38.700760 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.694138    2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 25 11:17:38.700760 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.694140    2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 25 11:17:38.700760 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.694143    2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 25 11:17:38.700760 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.694146    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 25 11:17:38.700760 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.694149    2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 25 11:17:38.700760 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.694151    2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 25 11:17:38.700760 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.694154    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 25 11:17:38.700760 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.694156    2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 25 11:17:38.700760 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.694159    2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 25 11:17:38.700760 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.694161    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 25 11:17:38.700760 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.694165    2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 25 11:17:38.700760 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.694169    2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 25 11:17:38.700760 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.694172    2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 25 11:17:38.700760 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.694175    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 25 11:17:38.701252 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.695220    2560 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]}
May 25 11:17:38.701850 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.701734    2560 server.go:530] "Kubelet version" kubeletVersion="v1.33.11"
May 25 11:17:38.701850 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.701849    2560 server.go:532] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
May 25 11:17:38.701944 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.701900    2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 25 11:17:38.701944 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.701906    2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 25 11:17:38.701944 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.701910    2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 25 11:17:38.701944 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.701914    2560 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 25 11:17:38.701944 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.701917    2560 feature_gate.go:328] unrecognized feature gate: Example2
May 25 11:17:38.701944 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.701920    2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 25 11:17:38.701944 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.701923    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 25 11:17:38.701944 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.701926    2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 25 11:17:38.701944 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.701929    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 25 11:17:38.701944 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.701932    2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 25 11:17:38.701944 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.701934    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 25 11:17:38.701944 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.701937    2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 25 11:17:38.701944 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.701939    2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 25 11:17:38.701944 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.701942    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 25 11:17:38.701944 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.701945    2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 25 11:17:38.701944 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.701948    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 25 11:17:38.701944 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.701951    2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 25 11:17:38.701944 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.701954    2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 25 11:17:38.702729 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.701957    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 25 11:17:38.702729 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.701960    2560 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 25 11:17:38.702729 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.701962    2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 25 11:17:38.702729 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.701965    2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 25 11:17:38.702729 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.701967    2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 25 11:17:38.702729 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.701970    2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 25 11:17:38.702729 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.701973    2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 25 11:17:38.702729 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.701975    2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 25 11:17:38.702729 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.701978    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 25 11:17:38.702729 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.701980    2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 25 11:17:38.702729 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.701983    2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 25 11:17:38.702729 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.701985    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 25 11:17:38.702729 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.701988    2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 25 11:17:38.702729 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.701991    2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 25 11:17:38.702729 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.701994    2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 25 11:17:38.702729 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.701997    2560 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 25 11:17:38.702729 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.701999    2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 25 11:17:38.702729 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.702002    2560 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 25 11:17:38.702729 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.702004    2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 25 11:17:38.702729 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.702007    2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 25 11:17:38.703249 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.702009    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 25 11:17:38.703249 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.702012    2560 feature_gate.go:328] unrecognized feature gate: Example
May 25 11:17:38.703249 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.702014    2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 25 11:17:38.703249 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.702017    2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 25 11:17:38.703249 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.702019    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 25 11:17:38.703249 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.702022    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 25 11:17:38.703249 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.702024    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 25 11:17:38.703249 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.702027    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 25 11:17:38.703249 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.702030    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 25 11:17:38.703249 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.702033    2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 25 11:17:38.703249 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.702035    2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 25 11:17:38.703249 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.702038    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 25 11:17:38.703249 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.702041    2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 25 11:17:38.703249 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.702044    2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 25 11:17:38.703249 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.702047    2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 25 11:17:38.703249 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.702049    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 25 11:17:38.703249 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.702052    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 25 11:17:38.703249 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.702054    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 25 11:17:38.703249 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.702057    2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 25 11:17:38.703249 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.702059    2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 25 11:17:38.703736 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.702062    2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 25 11:17:38.703736 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.702065    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 25 11:17:38.703736 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.702067    2560 feature_gate.go:328] unrecognized feature gate: NewOLM
May 25 11:17:38.703736 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.702071    2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 25 11:17:38.703736 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.702073    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 25 11:17:38.703736 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.702075    2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 25 11:17:38.703736 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.702079    2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 25 11:17:38.703736 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.702082    2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 25 11:17:38.703736 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.702087    2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 25 11:17:38.703736 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.702089    2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 25 11:17:38.703736 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.702092    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 25 11:17:38.703736 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.702095    2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 25 11:17:38.703736 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.702097    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 25 11:17:38.703736 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.702100    2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 25 11:17:38.703736 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.702102    2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 25 11:17:38.703736 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.702107    2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 25 11:17:38.703736 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.702110    2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 25 11:17:38.703736 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.702113    2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 25 11:17:38.704202 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.702116    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 25 11:17:38.704202 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.702119    2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 25 11:17:38.704202 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.702122    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 25 11:17:38.704202 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.702125    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 25 11:17:38.704202 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.702128    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 25 11:17:38.704202 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.702131    2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 25 11:17:38.704202 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.702134    2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 25 11:17:38.704202 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.702138    2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 25 11:17:38.704202 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.702141    2560 feature_gate.go:328] unrecognized feature gate: DualReplica
May 25 11:17:38.704202 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.702143    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 25 11:17:38.704202 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.702149    2560 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]}
May 25 11:17:38.704202 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.702263    2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 25 11:17:38.704202 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.702268    2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 25 11:17:38.704202 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.702272    2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 25 11:17:38.704202 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.702275    2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 25 11:17:38.704202 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.702278    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 25 11:17:38.704630 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.702281    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 25 11:17:38.704630 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.702284    2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 25 11:17:38.704630 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.702287    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 25 11:17:38.704630 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.702289    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 25 11:17:38.704630 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.702292    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 25 11:17:38.704630 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.702295    2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 25 11:17:38.704630 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.702298    2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 25 11:17:38.704630 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.702300    2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 25 11:17:38.704630 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.702303    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 25 11:17:38.704630 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.702305    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 25 11:17:38.704630 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.702308    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 25 11:17:38.704630 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.702310    2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 25 11:17:38.704630 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.702313    2560 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 25 11:17:38.704630 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.702315    2560 feature_gate.go:328] unrecognized feature gate: DualReplica
May 25 11:17:38.704630 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.702318    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 25 11:17:38.704630 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.702320    2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 25 11:17:38.704630 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.702323    2560 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 25 11:17:38.704630 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.702325    2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 25 11:17:38.704630 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.702328    2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 25 11:17:38.704630 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.702330    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 25 11:17:38.705129 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.702333    2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 25 11:17:38.705129 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.702335    2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 25 11:17:38.705129 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.702337    2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 25 11:17:38.705129 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.702341    2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 25 11:17:38.705129 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.702344    2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 25 11:17:38.705129 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.702346    2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 25 11:17:38.705129 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.702349    2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 25 11:17:38.705129 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.702352    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 25 11:17:38.705129 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.702354    2560 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 25 11:17:38.705129 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.702357    2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 25 11:17:38.705129 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.702359    2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 25 11:17:38.705129 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.702363    2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 25 11:17:38.705129 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.702366    2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 25 11:17:38.705129 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.702369    2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 25 11:17:38.705129 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.702372    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 25 11:17:38.705129 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.702374    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 25 11:17:38.705129 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.702377    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 25 11:17:38.705129 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.702379    2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 25 11:17:38.705129 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.702382    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 25 11:17:38.705129 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.702385    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 25 11:17:38.705664 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.702387    2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 25 11:17:38.705664 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.702389    2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 25 11:17:38.705664 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.702392    2560 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 25 11:17:38.705664 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.702394    2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 25 11:17:38.705664 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.702397    2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 25 11:17:38.705664 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.702399    2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 25 11:17:38.705664 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.702402    2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 25 11:17:38.705664 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.702404    2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 25 11:17:38.705664 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.702407    2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 25 11:17:38.705664 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.702409    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 25 11:17:38.705664 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.702412    2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 25 11:17:38.705664 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.702414    2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 25 11:17:38.705664 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.702417    2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 25 11:17:38.705664 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.702419    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 25 11:17:38.705664 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.702422    2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 25 11:17:38.705664 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.702425    2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 25 11:17:38.705664 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.702427    2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 25 11:17:38.705664 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.702430    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 25 11:17:38.705664 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.702433    2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 25 11:17:38.705664 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.702435    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 25 11:17:38.706147 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.702438    2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 25 11:17:38.706147 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.702441    2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 25 11:17:38.706147 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.702444    2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 25 11:17:38.706147 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.702446    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 25 11:17:38.706147 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.702449    2560 feature_gate.go:328] unrecognized feature gate: Example
May 25 11:17:38.706147 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.702452    2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 25 11:17:38.706147 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.702455    2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 25 11:17:38.706147 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.702457    2560 feature_gate.go:328] unrecognized feature gate: NewOLM
May 25 11:17:38.706147 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.702460    2560 feature_gate.go:328] unrecognized feature gate: Example2
May 25 11:17:38.706147 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.702462    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 25 11:17:38.706147 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.702465    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 25 11:17:38.706147 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.702468    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 25 11:17:38.706147 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.702471    2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 25 11:17:38.706147 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.702473    2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 25 11:17:38.706147 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.702476    2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 25 11:17:38.706147 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.702478    2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 25 11:17:38.706147 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.702481    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 25 11:17:38.706147 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.702483    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 25 11:17:38.706147 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.702486    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 25 11:17:38.706680 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.702490    2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 25 11:17:38.706680 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:38.702493    2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 25 11:17:38.706680 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.702498    2560 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]}
May 25 11:17:38.706680 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.703263    2560 server.go:962] "Client rotation is on, will bootstrap in background"
May 25 11:17:38.707526 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.707512    2560 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
May 25 11:17:38.708531 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.708520    2560 server.go:1019] "Starting client certificate rotation"
May 25 11:17:38.708646 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.708627    2560 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
May 25 11:17:38.708696 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.708682    2560 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
May 25 11:17:38.736630 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.736608    2560 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
May 25 11:17:38.738821 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.738802    2560 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
May 25 11:17:38.757135 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.757115    2560 log.go:25] "Validated CRI v1 runtime API"
May 25 11:17:38.762368 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.762353    2560 log.go:25] "Validated CRI v1 image API"
May 25 11:17:38.764887 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.764870    2560 server.go:1452] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
May 25 11:17:38.767323 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.767300    2560 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 86f14fe2-6032-4f9a-a0f7-28d5f9aee13c:/dev/nvme0n1p4 96f5dc8c-9fec-4a07-a199-ccbfcaa3de71:/dev/nvme0n1p3]
May 25 11:17:38.767418 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.767321    2560 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
May 25 11:17:38.769656 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.769641    2560 reflector.go:430] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
May 25 11:17:38.771634 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.771531    2560 manager.go:217] Machine: {Timestamp:2026-05-25 11:17:38.771034235 +0000 UTC m=+0.452012199 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100483 MemoryCapacity:33164500992 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec249e4e53688cad332d4a022041b5f3 SystemUUID:ec249e4e-5368-8cad-332d-4a022041b5f3 BootID:cd563fe2-0fe6-44eb-bc01-76a015adfcbf Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582250496 Type:vfs Inodes:4048401 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6103040 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:5d:05:57:d9:3b Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:5d:05:57:d9:3b Speed:0 Mtu:9001} {Name:ovs-system MacAddress:06:44:6a:74:97:cf Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164500992 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
May 25 11:17:38.771716 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.771657    2560 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
May 25 11:17:38.771791 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.771779    2560 manager.go:233] Version: {KernelVersion:5.14.0-570.114.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260512-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
May 25 11:17:38.773103 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.773080    2560 container_manager_linux.go:270] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
May 25 11:17:38.773258 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.773104    2560 container_manager_linux.go:275] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-146.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
May 25 11:17:38.773315 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.773269    2560 topology_manager.go:138] "Creating topology manager with none policy"
May 25 11:17:38.773315 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.773277    2560 container_manager_linux.go:306] "Creating device plugin manager"
May 25 11:17:38.773315 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.773284    2560 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
May 25 11:17:38.774232 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.774222    2560 server.go:72] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
May 25 11:17:38.776010 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.776000    2560 state_mem.go:36] "Initialized new in-memory state store"
May 25 11:17:38.776128 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.776119    2560 server.go:1267] "Using root directory" path="/var/lib/kubelet"
May 25 11:17:38.779326 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.779314    2560 kubelet.go:491] "Attempting to sync node with API server"
May 25 11:17:38.779378 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.779329    2560 kubelet.go:386] "Adding static pod path" path="/etc/kubernetes/manifests"
May 25 11:17:38.779378 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.779342    2560 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
May 25 11:17:38.779378 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.779352    2560 kubelet.go:397] "Adding apiserver pod source"
May 25 11:17:38.779378 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.779361    2560 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
May 25 11:17:38.781590 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.781573    2560 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
May 25 11:17:38.781692 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.781603    2560 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
May 25 11:17:38.785752 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.785734    2560 kuberuntime_manager.go:279] "Container runtime initialized" containerRuntime="cri-o" version="1.33.11-2.rhaos4.20.gitb2a8320.el9" apiVersion="v1"
May 25 11:17:38.787893 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.787873    2560 kubelet.go:953] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
May 25 11:17:38.789408 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.789396    2560 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
May 25 11:17:38.789463 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.789414    2560 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
May 25 11:17:38.789463 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.789421    2560 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
May 25 11:17:38.789463 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.789426    2560 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
May 25 11:17:38.789463 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.789434    2560 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
May 25 11:17:38.789463 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.789442    2560 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/secret"
May 25 11:17:38.789463 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.789451    2560 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
May 25 11:17:38.789463 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.789459    2560 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
May 25 11:17:38.789463 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.789466    2560 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/fc"
May 25 11:17:38.789667 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.789472    2560 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
May 25 11:17:38.789667 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.789481    2560 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/projected"
May 25 11:17:38.789946 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.789937    2560 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
May 25 11:17:38.791021 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.791009    2560 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/csi"
May 25 11:17:38.791021 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.791020    2560 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/image"
May 25 11:17:38.793383 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:17:38.793358    2560 reflector.go:200] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-128-146.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
May 25 11:17:38.793453 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:17:38.793424    2560 reflector.go:200] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
May 25 11:17:38.794145 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.794132    2560 watchdog_linux.go:99] "Systemd watchdog is not enabled"
May 25 11:17:38.794180 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.794170    2560 server.go:1295] "Started kubelet"
May 25 11:17:38.794326 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.794265    2560 ratelimit.go:55] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
May 25 11:17:38.794417 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.794344    2560 server_v1.go:47] "podresources" method="list" useActivePods=true
May 25 11:17:38.794417 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.794332    2560 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
May 25 11:17:38.794960 ip-10-0-128-146 systemd[1]: Started Kubernetes Kubelet.
May 25 11:17:38.795847 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.795826    2560 server.go:255] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
May 25 11:17:38.796289 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.796276    2560 server.go:317] "Adding debug handlers to kubelet server"
May 25 11:17:38.801524 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.801505    2560 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
May 25 11:17:38.802226 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.802196    2560 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
May 25 11:17:38.804983 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.804964    2560 volume_manager.go:295] "The desired_state_of_world populator starts"
May 25 11:17:38.804983 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.804983    2560 volume_manager.go:297] "Starting Kubelet Volume Manager"
May 25 11:17:38.805346 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.805316    2560 desired_state_of_world_populator.go:150] "Desired state populator starts to run"
May 25 11:17:38.805426 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.805420    2560 reconstruct.go:97] "Volume reconstruction finished"
May 25 11:17:38.805487 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.805429    2560 reconciler.go:26] "Reconciler: start to sync state"
May 25 11:17:38.805487 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:17:38.805428    2560 kubelet.go:1618] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
May 25 11:17:38.805853 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.805833    2560 factory.go:55] Registering systemd factory
May 25 11:17:38.805926 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.805866    2560 factory.go:223] Registration of the systemd container factory successfully
May 25 11:17:38.806406 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.806389    2560 factory.go:153] Registering CRI-O factory
May 25 11:17:38.806406 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.806406    2560 factory.go:223] Registration of the crio container factory successfully
May 25 11:17:38.806532 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.806450    2560 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
May 25 11:17:38.806532 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.806465    2560 factory.go:103] Registering Raw factory
May 25 11:17:38.806532 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.806501    2560 manager.go:1196] Started watching for new ooms in manager
May 25 11:17:38.807011 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:17:38.806956    2560 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-128-146.ec2.internal\" not found"
May 25 11:17:38.807660 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.807284    2560 manager.go:319] Starting recovery of all containers
May 25 11:17:38.808671 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:17:38.808644    2560 reflector.go:200] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
May 25 11:17:38.808775 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.808759    2560 csi_plugin.go:988] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-128-146.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
May 25 11:17:38.812597 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:17:38.808878    2560 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-146.ec2.internal.18b2cade47656a60  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-146.ec2.internal,UID:ip-10-0-128-146.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-128-146.ec2.internal,},FirstTimestamp:2026-05-25 11:17:38.7941464 +0000 UTC m=+0.475124364,LastTimestamp:2026-05-25 11:17:38.7941464 +0000 UTC m=+0.475124364,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-146.ec2.internal,}"
May 25 11:17:38.812712 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:17:38.812690    2560 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-128-146.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
May 25 11:17:38.820430 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.820415    2560 manager.go:324] Recovery completed
May 25 11:17:38.824772 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.824760    2560 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 25 11:17:38.827962 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.827945    2560 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-s2r7q"
May 25 11:17:38.829453 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.829437    2560 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-128-146.ec2.internal" event="NodeHasSufficientMemory"
May 25 11:17:38.829504 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.829464    2560 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-128-146.ec2.internal" event="NodeHasNoDiskPressure"
May 25 11:17:38.829504 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.829474    2560 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-128-146.ec2.internal" event="NodeHasSufficientPID"
May 25 11:17:38.829922 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.829907    2560 cpu_manager.go:222] "Starting CPU manager" policy="none"
May 25 11:17:38.829922 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.829919    2560 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
May 25 11:17:38.830001 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.829936    2560 state_mem.go:36] "Initialized new in-memory state store"
May 25 11:17:38.831509 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:17:38.831444    2560 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-146.ec2.internal.18b2cade498022dc  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-146.ec2.internal,UID:ip-10-0-128-146.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-128-146.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-128-146.ec2.internal,},FirstTimestamp:2026-05-25 11:17:38.829451996 +0000 UTC m=+0.510429959,LastTimestamp:2026-05-25 11:17:38.829451996 +0000 UTC m=+0.510429959,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-146.ec2.internal,}"
May 25 11:17:38.833306 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.833293    2560 policy_none.go:49] "None policy: Start"
May 25 11:17:38.833372 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.833310    2560 memory_manager.go:186] "Starting memorymanager" policy="None"
May 25 11:17:38.833372 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.833322    2560 state_mem.go:35] "Initializing new in-memory state store"
May 25 11:17:38.838944 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.838922    2560 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-s2r7q"
May 25 11:17:38.885561 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.873763    2560 manager.go:341] "Starting Device Plugin manager"
May 25 11:17:38.885561 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:17:38.873797    2560 manager.go:517] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
May 25 11:17:38.885561 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.873807    2560 server.go:85] "Starting device plugin registration server"
May 25 11:17:38.885561 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.874005    2560 eviction_manager.go:189] "Eviction manager: starting control loop"
May 25 11:17:38.885561 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.874016    2560 container_log_manager.go:189] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
May 25 11:17:38.885561 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.874106    2560 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
May 25 11:17:38.885561 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.874177    2560 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
May 25 11:17:38.885561 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.874185    2560 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
May 25 11:17:38.885561 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:17:38.874723    2560 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
May 25 11:17:38.885561 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:17:38.874784    2560 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-128-146.ec2.internal\" not found"
May 25 11:17:38.928422 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.928381    2560 kubelet_network_linux.go:49] "Initialized iptables rules." protocol="IPv4"
May 25 11:17:38.929544 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.929530    2560 kubelet_network_linux.go:49] "Initialized iptables rules." protocol="IPv6"
May 25 11:17:38.929629 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.929560    2560 status_manager.go:230] "Starting to sync pod status with apiserver"
May 25 11:17:38.929629 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.929582    2560 watchdog_linux.go:127] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
May 25 11:17:38.929629 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.929592    2560 kubelet.go:2451] "Starting kubelet main sync loop"
May 25 11:17:38.929756 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:17:38.929699    2560 kubelet.go:2475] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
May 25 11:17:38.932221 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.932191    2560 reflector.go:430] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
May 25 11:17:38.975140 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.975079    2560 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 25 11:17:38.975979 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.975956    2560 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-128-146.ec2.internal" event="NodeHasSufficientMemory"
May 25 11:17:38.976085 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.975988    2560 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-128-146.ec2.internal" event="NodeHasNoDiskPressure"
May 25 11:17:38.976085 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.975999    2560 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-128-146.ec2.internal" event="NodeHasSufficientPID"
May 25 11:17:38.976085 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.976022    2560 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-128-146.ec2.internal"
May 25 11:17:38.983299 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:38.983284    2560 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-128-146.ec2.internal"
May 25 11:17:38.983361 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:17:38.983307    2560 kubelet_node_status.go:597] "Error updating node status, will retry" err="error getting node \"ip-10-0-128-146.ec2.internal\": node \"ip-10-0-128-146.ec2.internal\" not found"
May 25 11:17:39.001130 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:17:39.001103    2560 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-128-146.ec2.internal\" not found"
May 25 11:17:39.029921 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:39.029893    2560 kubelet.go:2537] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-146.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-128-146.ec2.internal"]
May 25 11:17:39.030068 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:39.029958    2560 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 25 11:17:39.031868 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:39.031839    2560 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-128-146.ec2.internal" event="NodeHasSufficientMemory"
May 25 11:17:39.031971 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:39.031877    2560 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-128-146.ec2.internal" event="NodeHasNoDiskPressure"
May 25 11:17:39.031971 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:39.031891    2560 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-128-146.ec2.internal" event="NodeHasSufficientPID"
May 25 11:17:39.034110 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:39.034097    2560 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 25 11:17:39.034325 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:39.034304    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-146.ec2.internal"
May 25 11:17:39.034365 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:39.034357    2560 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 25 11:17:39.034805 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:39.034790    2560 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-128-146.ec2.internal" event="NodeHasSufficientMemory"
May 25 11:17:39.034882 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:39.034818    2560 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-128-146.ec2.internal" event="NodeHasNoDiskPressure"
May 25 11:17:39.034882 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:39.034829    2560 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-128-146.ec2.internal" event="NodeHasSufficientPID"
May 25 11:17:39.034960 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:39.034932    2560 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-128-146.ec2.internal" event="NodeHasSufficientMemory"
May 25 11:17:39.034960 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:39.034956    2560 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-128-146.ec2.internal" event="NodeHasNoDiskPressure"
May 25 11:17:39.035016 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:39.034965    2560 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-128-146.ec2.internal" event="NodeHasSufficientPID"
May 25 11:17:39.036875 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:39.036859    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-146.ec2.internal"
May 25 11:17:39.036951 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:39.036896    2560 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 25 11:17:39.037476 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:39.037463    2560 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-128-146.ec2.internal" event="NodeHasSufficientMemory"
May 25 11:17:39.037536 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:39.037486    2560 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-128-146.ec2.internal" event="NodeHasNoDiskPressure"
May 25 11:17:39.037536 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:39.037496    2560 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-128-146.ec2.internal" event="NodeHasSufficientPID"
May 25 11:17:39.062848 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:17:39.062826    2560 kubelet.go:3336] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-146.ec2.internal\" not found" node="ip-10-0-128-146.ec2.internal"
May 25 11:17:39.068563 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:17:39.068545    2560 kubelet.go:3336] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-146.ec2.internal\" not found" node="ip-10-0-128-146.ec2.internal"
May 25 11:17:39.101197 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:17:39.101176    2560 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-128-146.ec2.internal\" not found"
May 25 11:17:39.201777 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:17:39.201748    2560 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-128-146.ec2.internal\" not found"
May 25 11:17:39.207062 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:39.207041    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f7c7694d1cd47f61dc1e9c46a604067d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-146.ec2.internal\" (UID: \"f7c7694d1cd47f61dc1e9c46a604067d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-146.ec2.internal"
May 25 11:17:39.207133 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:39.207069    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f7c7694d1cd47f61dc1e9c46a604067d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-146.ec2.internal\" (UID: \"f7c7694d1cd47f61dc1e9c46a604067d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-146.ec2.internal"
May 25 11:17:39.207133 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:39.207117    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/205c636549bb34e2af97bc6edc40842f-config\") pod \"kube-apiserver-proxy-ip-10-0-128-146.ec2.internal\" (UID: \"205c636549bb34e2af97bc6edc40842f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-146.ec2.internal"
May 25 11:17:39.302495 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:17:39.302426    2560 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-128-146.ec2.internal\" not found"
May 25 11:17:39.307781 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:39.307765    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f7c7694d1cd47f61dc1e9c46a604067d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-146.ec2.internal\" (UID: \"f7c7694d1cd47f61dc1e9c46a604067d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-146.ec2.internal"
May 25 11:17:39.307837 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:39.307790    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f7c7694d1cd47f61dc1e9c46a604067d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-146.ec2.internal\" (UID: \"f7c7694d1cd47f61dc1e9c46a604067d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-146.ec2.internal"
May 25 11:17:39.307837 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:39.307808    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/205c636549bb34e2af97bc6edc40842f-config\") pod \"kube-apiserver-proxy-ip-10-0-128-146.ec2.internal\" (UID: \"205c636549bb34e2af97bc6edc40842f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-146.ec2.internal"
May 25 11:17:39.307912 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:39.307848    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/205c636549bb34e2af97bc6edc40842f-config\") pod \"kube-apiserver-proxy-ip-10-0-128-146.ec2.internal\" (UID: \"205c636549bb34e2af97bc6edc40842f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-146.ec2.internal"
May 25 11:17:39.307912 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:39.307882    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f7c7694d1cd47f61dc1e9c46a604067d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-146.ec2.internal\" (UID: \"f7c7694d1cd47f61dc1e9c46a604067d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-146.ec2.internal"
May 25 11:17:39.307981 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:39.307882    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f7c7694d1cd47f61dc1e9c46a604067d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-146.ec2.internal\" (UID: \"f7c7694d1cd47f61dc1e9c46a604067d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-146.ec2.internal"
May 25 11:17:39.365924 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:39.365894    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-146.ec2.internal"
May 25 11:17:39.371681 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:39.371657    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-146.ec2.internal"
May 25 11:17:39.403145 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:17:39.403113    2560 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-128-146.ec2.internal\" not found"
May 25 11:17:39.503674 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:17:39.503630    2560 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-128-146.ec2.internal\" not found"
May 25 11:17:39.604258 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:17:39.604166    2560 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-128-146.ec2.internal\" not found"
May 25 11:17:39.644026 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:39.643997    2560 reflector.go:430] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
May 25 11:17:39.687493 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:39.687464    2560 reflector.go:430] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
May 25 11:17:39.704596 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:17:39.704571    2560 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-128-146.ec2.internal\" not found"
May 25 11:17:39.707891 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:39.707879    2560 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
May 25 11:17:39.708010 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:39.707993    2560 reflector.go:556] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
May 25 11:17:39.708052 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:39.708023    2560 reflector.go:556] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
May 25 11:17:39.708094 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:39.708046    2560 reflector.go:556] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
May 25 11:17:39.802647 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:39.802612    2560 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
May 25 11:17:39.805617 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:17:39.805600    2560 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-128-146.ec2.internal\" not found"
May 25 11:17:39.824324 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:39.824301    2560 reflector.go:430] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
May 25 11:17:39.840799 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:39.840771    2560 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-05-24 11:12:38 +0000 UTC" deadline="2027-12-31 03:28:53.783165133 +0000 UTC"
May 25 11:17:39.840879 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:39.840799    2560 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14032h11m13.942368716s"
May 25 11:17:39.852694 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:39.852673    2560 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-p4m4f"
May 25 11:17:39.864891 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:39.864743    2560 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-p4m4f"
May 25 11:17:39.868080 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:39.868050    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod205c636549bb34e2af97bc6edc40842f.slice/crio-1d836ce64b19626fd6bec926e56812cad5c67c27ea9e622908b6746b2ce4f35a WatchSource:0}: Error finding container 1d836ce64b19626fd6bec926e56812cad5c67c27ea9e622908b6746b2ce4f35a: Status 404 returned error can't find the container with id 1d836ce64b19626fd6bec926e56812cad5c67c27ea9e622908b6746b2ce4f35a
May 25 11:17:39.868377 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:39.868351    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7c7694d1cd47f61dc1e9c46a604067d.slice/crio-aedc3275adbd65576ff1e8db78dde7800dc35ced0910a4e2344e73e52b596a34 WatchSource:0}: Error finding container aedc3275adbd65576ff1e8db78dde7800dc35ced0910a4e2344e73e52b596a34: Status 404 returned error can't find the container with id aedc3275adbd65576ff1e8db78dde7800dc35ced0910a4e2344e73e52b596a34
May 25 11:17:39.873436 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:39.873421    2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 25 11:17:39.905856 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:17:39.905816    2560 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-128-146.ec2.internal\" not found"
May 25 11:17:39.933238 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:39.933160    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-146.ec2.internal" event={"ID":"205c636549bb34e2af97bc6edc40842f","Type":"ContainerStarted","Data":"1d836ce64b19626fd6bec926e56812cad5c67c27ea9e622908b6746b2ce4f35a"}
May 25 11:17:39.934064 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:39.934042    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-146.ec2.internal" event={"ID":"f7c7694d1cd47f61dc1e9c46a604067d","Type":"ContainerStarted","Data":"aedc3275adbd65576ff1e8db78dde7800dc35ced0910a4e2344e73e52b596a34"}
May 25 11:17:40.006313 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:17:40.006262    2560 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-128-146.ec2.internal\" not found"
May 25 11:17:40.106742 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:17:40.106709    2560 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-128-146.ec2.internal\" not found"
May 25 11:17:40.137614 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.137545    2560 reflector.go:430] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
May 25 11:17:40.206914 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.206877    2560 kubelet.go:3340] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-146.ec2.internal"
May 25 11:17:40.220595 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.220564    2560 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
May 25 11:17:40.221680 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.221655    2560 kubelet.go:3340] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-146.ec2.internal"
May 25 11:17:40.230372 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.230350    2560 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
May 25 11:17:40.781344 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.781310    2560 apiserver.go:52] "Watching apiserver"
May 25 11:17:40.789277 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.789254    2560 reflector.go:430] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
May 25 11:17:40.790493 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.790468    2560 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-network-operator/iptables-alerter-kj6wk","kube-system/kube-apiserver-proxy-ip-10-0-128-146.ec2.internal","openshift-cluster-node-tuning-operator/tuned-tgqrm","openshift-multus/network-metrics-daemon-hxdhk","openshift-ovn-kubernetes/ovnkube-node-m55v2","kube-system/konnectivity-agent-fbljm","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lwkh6","openshift-image-registry/node-ca-8vzrv","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-146.ec2.internal","openshift-multus/multus-5qqv9","openshift-multus/multus-additional-cni-plugins-vdfgw","openshift-network-diagnostics/network-check-target-zfljp"]
May 25 11:17:40.795168 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.795146    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-kj6wk"
May 25 11:17:40.797156 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.797133    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-tgqrm"
May 25 11:17:40.797283 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.797264    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hxdhk"
May 25 11:17:40.797373 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:17:40.797343    2560 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hxdhk" podUID="b0dc07c1-2dce-4b08-9fee-41a5937a7bae"
May 25 11:17:40.798274 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.798253    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
May 25 11:17:40.798365 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.798297    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
May 25 11:17:40.798488 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.798462    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-n5r4b\""
May 25 11:17:40.798594 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.798575    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
May 25 11:17:40.799507 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.799487    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-m55v2"
May 25 11:17:40.800224 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.800185    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-bvsdf\""
May 25 11:17:40.800313 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.800264    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
May 25 11:17:40.800358 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.800331    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
May 25 11:17:40.801735 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.801717    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-fbljm"
May 25 11:17:40.806226 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.805491    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lwkh6"
May 25 11:17:40.806226 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.805979    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-5rv8b\""
May 25 11:17:40.806226 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.805985    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
May 25 11:17:40.806226 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.806009    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
May 25 11:17:40.806226 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.806015    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
May 25 11:17:40.806226 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.806106    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
May 25 11:17:40.806226 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.806154    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
May 25 11:17:40.806226 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.806181    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
May 25 11:17:40.806624 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.806365    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-hhwzg\""
May 25 11:17:40.807077 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.807052    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
May 25 11:17:40.807165 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.807079    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
May 25 11:17:40.807753 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.807737    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-8vzrv"
May 25 11:17:40.810158 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.810138    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-5qqv9"
May 25 11:17:40.812452 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.812426    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-vdfgw"
May 25 11:17:40.814388 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.814369    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7c5c3c1e-f8a8-437b-972e-1c293222f327-sys-fs\") pod \"aws-ebs-csi-driver-node-lwkh6\" (UID: \"7c5c3c1e-f8a8-437b-972e-1c293222f327\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lwkh6"
May 25 11:17:40.814482 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.814393    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2d4d0de1-9c41-46ef-89e3-9a0615a665af-host\") pod \"node-ca-8vzrv\" (UID: \"2d4d0de1-9c41-46ef-89e3-9a0615a665af\") " pod="openshift-image-registry/node-ca-8vzrv"
May 25 11:17:40.814482 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.814415    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/448f7dca-880c-4e81-8adc-d29e391dfefd-etc-kubernetes\") pod \"tuned-tgqrm\" (UID: \"448f7dca-880c-4e81-8adc-d29e391dfefd\") " pod="openshift-cluster-node-tuning-operator/tuned-tgqrm"
May 25 11:17:40.814482 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.814437    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/448f7dca-880c-4e81-8adc-d29e391dfefd-host\") pod \"tuned-tgqrm\" (UID: \"448f7dca-880c-4e81-8adc-d29e391dfefd\") " pod="openshift-cluster-node-tuning-operator/tuned-tgqrm"
May 25 11:17:40.814644 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.814500    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/448f7dca-880c-4e81-8adc-d29e391dfefd-tmp\") pod \"tuned-tgqrm\" (UID: \"448f7dca-880c-4e81-8adc-d29e391dfefd\") " pod="openshift-cluster-node-tuning-operator/tuned-tgqrm"
May 25 11:17:40.814644 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.814552    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdts4\" (UniqueName: \"kubernetes.io/projected/b0dc07c1-2dce-4b08-9fee-41a5937a7bae-kube-api-access-kdts4\") pod \"network-metrics-daemon-hxdhk\" (UID: \"b0dc07c1-2dce-4b08-9fee-41a5937a7bae\") " pod="openshift-multus/network-metrics-daemon-hxdhk"
May 25 11:17:40.814644 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.814600    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b93f3e9c-f4f9-49dc-bfba-6d0f80c0971d-ovn-node-metrics-cert\") pod \"ovnkube-node-m55v2\" (UID: \"b93f3e9c-f4f9-49dc-bfba-6d0f80c0971d\") " pod="openshift-ovn-kubernetes/ovnkube-node-m55v2"
May 25 11:17:40.814644 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.814635    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5d252e49-a8a8-4794-befd-50a55e95b17b-agent-certs\") pod \"konnectivity-agent-fbljm\" (UID: \"5d252e49-a8a8-4794-befd-50a55e95b17b\") " pod="kube-system/konnectivity-agent-fbljm"
May 25 11:17:40.814834 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.814651    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zfljp"
May 25 11:17:40.814834 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.814662    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7c5c3c1e-f8a8-437b-972e-1c293222f327-socket-dir\") pod \"aws-ebs-csi-driver-node-lwkh6\" (UID: \"7c5c3c1e-f8a8-437b-972e-1c293222f327\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lwkh6"
May 25 11:17:40.814834 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.814688    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnzzp\" (UniqueName: \"kubernetes.io/projected/7c5c3c1e-f8a8-437b-972e-1c293222f327-kube-api-access-pnzzp\") pod \"aws-ebs-csi-driver-node-lwkh6\" (UID: \"7c5c3c1e-f8a8-437b-972e-1c293222f327\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lwkh6"
May 25 11:17:40.814834 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:17:40.814702    2560 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zfljp" podUID="0d6828f5-3d2a-417d-ab45-6750dd8cdb92"
May 25 11:17:40.814834 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.814715    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b93f3e9c-f4f9-49dc-bfba-6d0f80c0971d-host-run-ovn-kubernetes\") pod \"ovnkube-node-m55v2\" (UID: \"b93f3e9c-f4f9-49dc-bfba-6d0f80c0971d\") " pod="openshift-ovn-kubernetes/ovnkube-node-m55v2"
May 25 11:17:40.814834 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.814739    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b93f3e9c-f4f9-49dc-bfba-6d0f80c0971d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-m55v2\" (UID: \"b93f3e9c-f4f9-49dc-bfba-6d0f80c0971d\") " pod="openshift-ovn-kubernetes/ovnkube-node-m55v2"
May 25 11:17:40.814834 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.814765    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b93f3e9c-f4f9-49dc-bfba-6d0f80c0971d-run-systemd\") pod \"ovnkube-node-m55v2\" (UID: \"b93f3e9c-f4f9-49dc-bfba-6d0f80c0971d\") " pod="openshift-ovn-kubernetes/ovnkube-node-m55v2"
May 25 11:17:40.814834 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.814811    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/448f7dca-880c-4e81-8adc-d29e391dfefd-etc-modprobe-d\") pod \"tuned-tgqrm\" (UID: \"448f7dca-880c-4e81-8adc-d29e391dfefd\") " pod="openshift-cluster-node-tuning-operator/tuned-tgqrm"
May 25 11:17:40.815234 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.814847    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/448f7dca-880c-4e81-8adc-d29e391dfefd-etc-sysctl-d\") pod \"tuned-tgqrm\" (UID: \"448f7dca-880c-4e81-8adc-d29e391dfefd\") " pod="openshift-cluster-node-tuning-operator/tuned-tgqrm"
May 25 11:17:40.815234 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.814877    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xsv5\" (UniqueName: \"kubernetes.io/projected/b93f3e9c-f4f9-49dc-bfba-6d0f80c0971d-kube-api-access-5xsv5\") pod \"ovnkube-node-m55v2\" (UID: \"b93f3e9c-f4f9-49dc-bfba-6d0f80c0971d\") " pod="openshift-ovn-kubernetes/ovnkube-node-m55v2"
May 25 11:17:40.815234 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.814898    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7c5c3c1e-f8a8-437b-972e-1c293222f327-device-dir\") pod \"aws-ebs-csi-driver-node-lwkh6\" (UID: \"7c5c3c1e-f8a8-437b-972e-1c293222f327\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lwkh6"
May 25 11:17:40.815234 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.814925    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b93f3e9c-f4f9-49dc-bfba-6d0f80c0971d-var-lib-openvswitch\") pod \"ovnkube-node-m55v2\" (UID: \"b93f3e9c-f4f9-49dc-bfba-6d0f80c0971d\") " pod="openshift-ovn-kubernetes/ovnkube-node-m55v2"
May 25 11:17:40.815234 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.814947    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-td25w\" (UniqueName: \"kubernetes.io/projected/0a24f6a0-7ac5-4f1e-84f0-301e16006665-kube-api-access-td25w\") pod \"iptables-alerter-kj6wk\" (UID: \"0a24f6a0-7ac5-4f1e-84f0-301e16006665\") " pod="openshift-network-operator/iptables-alerter-kj6wk"
May 25 11:17:40.815234 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.814962    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/448f7dca-880c-4e81-8adc-d29e391dfefd-lib-modules\") pod \"tuned-tgqrm\" (UID: \"448f7dca-880c-4e81-8adc-d29e391dfefd\") " pod="openshift-cluster-node-tuning-operator/tuned-tgqrm"
May 25 11:17:40.815234 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.814989    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b93f3e9c-f4f9-49dc-bfba-6d0f80c0971d-host-cni-netd\") pod \"ovnkube-node-m55v2\" (UID: \"b93f3e9c-f4f9-49dc-bfba-6d0f80c0971d\") " pod="openshift-ovn-kubernetes/ovnkube-node-m55v2"
May 25 11:17:40.815234 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.815020    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7c5c3c1e-f8a8-437b-972e-1c293222f327-kubelet-dir\") pod \"aws-ebs-csi-driver-node-lwkh6\" (UID: \"7c5c3c1e-f8a8-437b-972e-1c293222f327\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lwkh6"
May 25 11:17:40.815234 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.815051    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7c5c3c1e-f8a8-437b-972e-1c293222f327-registration-dir\") pod \"aws-ebs-csi-driver-node-lwkh6\" (UID: \"7c5c3c1e-f8a8-437b-972e-1c293222f327\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lwkh6"
May 25 11:17:40.815234 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.815078    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2d4d0de1-9c41-46ef-89e3-9a0615a665af-serviceca\") pod \"node-ca-8vzrv\" (UID: \"2d4d0de1-9c41-46ef-89e3-9a0615a665af\") " pod="openshift-image-registry/node-ca-8vzrv"
May 25 11:17:40.815234 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.815096    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22dpr\" (UniqueName: \"kubernetes.io/projected/2d4d0de1-9c41-46ef-89e3-9a0615a665af-kube-api-access-22dpr\") pod \"node-ca-8vzrv\" (UID: \"2d4d0de1-9c41-46ef-89e3-9a0615a665af\") " pod="openshift-image-registry/node-ca-8vzrv"
May 25 11:17:40.815234 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.815117    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/448f7dca-880c-4e81-8adc-d29e391dfefd-etc-sysctl-conf\") pod \"tuned-tgqrm\" (UID: \"448f7dca-880c-4e81-8adc-d29e391dfefd\") " pod="openshift-cluster-node-tuning-operator/tuned-tgqrm"
May 25 11:17:40.815234 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.815140    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hh9c\" (UniqueName: \"kubernetes.io/projected/448f7dca-880c-4e81-8adc-d29e391dfefd-kube-api-access-9hh9c\") pod \"tuned-tgqrm\" (UID: \"448f7dca-880c-4e81-8adc-d29e391dfefd\") " pod="openshift-cluster-node-tuning-operator/tuned-tgqrm"
May 25 11:17:40.815234 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.815200    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b93f3e9c-f4f9-49dc-bfba-6d0f80c0971d-host-slash\") pod \"ovnkube-node-m55v2\" (UID: \"b93f3e9c-f4f9-49dc-bfba-6d0f80c0971d\") " pod="openshift-ovn-kubernetes/ovnkube-node-m55v2"
May 25 11:17:40.815932 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.815246    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b93f3e9c-f4f9-49dc-bfba-6d0f80c0971d-etc-openvswitch\") pod \"ovnkube-node-m55v2\" (UID: \"b93f3e9c-f4f9-49dc-bfba-6d0f80c0971d\") " pod="openshift-ovn-kubernetes/ovnkube-node-m55v2"
May 25 11:17:40.815932 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.815274    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b93f3e9c-f4f9-49dc-bfba-6d0f80c0971d-log-socket\") pod \"ovnkube-node-m55v2\" (UID: \"b93f3e9c-f4f9-49dc-bfba-6d0f80c0971d\") " pod="openshift-ovn-kubernetes/ovnkube-node-m55v2"
May 25 11:17:40.815932 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.815300    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b93f3e9c-f4f9-49dc-bfba-6d0f80c0971d-host-cni-bin\") pod \"ovnkube-node-m55v2\" (UID: \"b93f3e9c-f4f9-49dc-bfba-6d0f80c0971d\") " pod="openshift-ovn-kubernetes/ovnkube-node-m55v2"
May 25 11:17:40.815932 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.815324    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b93f3e9c-f4f9-49dc-bfba-6d0f80c0971d-env-overrides\") pod \"ovnkube-node-m55v2\" (UID: \"b93f3e9c-f4f9-49dc-bfba-6d0f80c0971d\") " pod="openshift-ovn-kubernetes/ovnkube-node-m55v2"
May 25 11:17:40.815932 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.815346    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0a24f6a0-7ac5-4f1e-84f0-301e16006665-host-slash\") pod \"iptables-alerter-kj6wk\" (UID: \"0a24f6a0-7ac5-4f1e-84f0-301e16006665\") " pod="openshift-network-operator/iptables-alerter-kj6wk"
May 25 11:17:40.815932 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.815370    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b93f3e9c-f4f9-49dc-bfba-6d0f80c0971d-systemd-units\") pod \"ovnkube-node-m55v2\" (UID: \"b93f3e9c-f4f9-49dc-bfba-6d0f80c0971d\") " pod="openshift-ovn-kubernetes/ovnkube-node-m55v2"
May 25 11:17:40.815932 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.815395    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b93f3e9c-f4f9-49dc-bfba-6d0f80c0971d-run-ovn\") pod \"ovnkube-node-m55v2\" (UID: \"b93f3e9c-f4f9-49dc-bfba-6d0f80c0971d\") " pod="openshift-ovn-kubernetes/ovnkube-node-m55v2"
May 25 11:17:40.815932 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.815419    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5d252e49-a8a8-4794-befd-50a55e95b17b-konnectivity-ca\") pod \"konnectivity-agent-fbljm\" (UID: \"5d252e49-a8a8-4794-befd-50a55e95b17b\") " pod="kube-system/konnectivity-agent-fbljm"
May 25 11:17:40.815932 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.815452    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7c5c3c1e-f8a8-437b-972e-1c293222f327-etc-selinux\") pod \"aws-ebs-csi-driver-node-lwkh6\" (UID: \"7c5c3c1e-f8a8-437b-972e-1c293222f327\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lwkh6"
May 25 11:17:40.815932 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.815479    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/448f7dca-880c-4e81-8adc-d29e391dfefd-etc-systemd\") pod \"tuned-tgqrm\" (UID: \"448f7dca-880c-4e81-8adc-d29e391dfefd\") " pod="openshift-cluster-node-tuning-operator/tuned-tgqrm"
May 25 11:17:40.815932 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.815501    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/448f7dca-880c-4e81-8adc-d29e391dfefd-run\") pod \"tuned-tgqrm\" (UID: \"448f7dca-880c-4e81-8adc-d29e391dfefd\") " pod="openshift-cluster-node-tuning-operator/tuned-tgqrm"
May 25 11:17:40.815932 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.815526    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0a24f6a0-7ac5-4f1e-84f0-301e16006665-iptables-alerter-script\") pod \"iptables-alerter-kj6wk\" (UID: \"0a24f6a0-7ac5-4f1e-84f0-301e16006665\") " pod="openshift-network-operator/iptables-alerter-kj6wk"
May 25 11:17:40.815932 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.815549    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/448f7dca-880c-4e81-8adc-d29e391dfefd-var-lib-kubelet\") pod \"tuned-tgqrm\" (UID: \"448f7dca-880c-4e81-8adc-d29e391dfefd\") " pod="openshift-cluster-node-tuning-operator/tuned-tgqrm"
May 25 11:17:40.815932 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.815592    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0dc07c1-2dce-4b08-9fee-41a5937a7bae-metrics-certs\") pod \"network-metrics-daemon-hxdhk\" (UID: \"b0dc07c1-2dce-4b08-9fee-41a5937a7bae\") " pod="openshift-multus/network-metrics-daemon-hxdhk"
May 25 11:17:40.815932 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.815625    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b93f3e9c-f4f9-49dc-bfba-6d0f80c0971d-host-kubelet\") pod \"ovnkube-node-m55v2\" (UID: \"b93f3e9c-f4f9-49dc-bfba-6d0f80c0971d\") " pod="openshift-ovn-kubernetes/ovnkube-node-m55v2"
May 25 11:17:40.815932 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.815674    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b93f3e9c-f4f9-49dc-bfba-6d0f80c0971d-ovnkube-config\") pod \"ovnkube-node-m55v2\" (UID: \"b93f3e9c-f4f9-49dc-bfba-6d0f80c0971d\") " pod="openshift-ovn-kubernetes/ovnkube-node-m55v2"
May 25 11:17:40.816693 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.815700    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b93f3e9c-f4f9-49dc-bfba-6d0f80c0971d-ovnkube-script-lib\") pod \"ovnkube-node-m55v2\" (UID: \"b93f3e9c-f4f9-49dc-bfba-6d0f80c0971d\") " pod="openshift-ovn-kubernetes/ovnkube-node-m55v2"
May 25 11:17:40.816693 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.815721    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/448f7dca-880c-4e81-8adc-d29e391dfefd-etc-sysconfig\") pod \"tuned-tgqrm\" (UID: \"448f7dca-880c-4e81-8adc-d29e391dfefd\") " pod="openshift-cluster-node-tuning-operator/tuned-tgqrm"
May 25 11:17:40.816693 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.815771    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/448f7dca-880c-4e81-8adc-d29e391dfefd-sys\") pod \"tuned-tgqrm\" (UID: \"448f7dca-880c-4e81-8adc-d29e391dfefd\") " pod="openshift-cluster-node-tuning-operator/tuned-tgqrm"
May 25 11:17:40.816693 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.815785    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/448f7dca-880c-4e81-8adc-d29e391dfefd-etc-tuned\") pod \"tuned-tgqrm\" (UID: \"448f7dca-880c-4e81-8adc-d29e391dfefd\") " pod="openshift-cluster-node-tuning-operator/tuned-tgqrm"
May 25 11:17:40.816693 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.815807    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b93f3e9c-f4f9-49dc-bfba-6d0f80c0971d-host-run-netns\") pod \"ovnkube-node-m55v2\" (UID: \"b93f3e9c-f4f9-49dc-bfba-6d0f80c0971d\") " pod="openshift-ovn-kubernetes/ovnkube-node-m55v2"
May 25 11:17:40.816693 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.815830    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b93f3e9c-f4f9-49dc-bfba-6d0f80c0971d-run-openvswitch\") pod \"ovnkube-node-m55v2\" (UID: \"b93f3e9c-f4f9-49dc-bfba-6d0f80c0971d\") " pod="openshift-ovn-kubernetes/ovnkube-node-m55v2"
May 25 11:17:40.816693 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.815852    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b93f3e9c-f4f9-49dc-bfba-6d0f80c0971d-node-log\") pod \"ovnkube-node-m55v2\" (UID: \"b93f3e9c-f4f9-49dc-bfba-6d0f80c0971d\") " pod="openshift-ovn-kubernetes/ovnkube-node-m55v2"
May 25 11:17:40.816693 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.816325    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
May 25 11:17:40.816693 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.816362    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
May 25 11:17:40.816693 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.816398    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
May 25 11:17:40.816693 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.816580    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
May 25 11:17:40.816693 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.816601    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
May 25 11:17:40.816693 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.816614    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
May 25 11:17:40.816693 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.816630    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
May 25 11:17:40.817392 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.817373    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
May 25 11:17:40.817575 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.817561    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
May 25 11:17:40.817575 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.817569    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-kmbkd\""
May 25 11:17:40.817750 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.817709    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-hw9xl\""
May 25 11:17:40.817750 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.817744    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-8jcj7\""
May 25 11:17:40.818045 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.817825    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
May 25 11:17:40.818045 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.817863    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
May 25 11:17:40.818164 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.818091    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-gszzw\""
May 25 11:17:40.818164 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.818105    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
May 25 11:17:40.842353 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.842333    2560 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-nq67w"]
May 25 11:17:40.845045 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.845027    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nq67w"
May 25 11:17:40.845131 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:17:40.845100    2560 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nq67w" podUID="c0068087-c30b-4ff5-a794-b84738aaa168"
May 25 11:17:40.866115 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.866088    2560 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-05-24 11:12:39 +0000 UTC" deadline="2027-10-23 13:04:24.406289011 +0000 UTC"
May 25 11:17:40.866204 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.866115    2560 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12385h46m43.540180506s"
May 25 11:17:40.906928 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.906904    2560 desired_state_of_world_populator.go:158] "Finished populating initial desired state of world"
May 25 11:17:40.916442 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.916416    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b93f3e9c-f4f9-49dc-bfba-6d0f80c0971d-run-ovn\") pod \"ovnkube-node-m55v2\" (UID: \"b93f3e9c-f4f9-49dc-bfba-6d0f80c0971d\") " pod="openshift-ovn-kubernetes/ovnkube-node-m55v2"
May 25 11:17:40.916575 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.916450    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9bfeee33-bbdf-4407-9f32-5dfe1a29d075-system-cni-dir\") pod \"multus-5qqv9\" (UID: \"9bfeee33-bbdf-4407-9f32-5dfe1a29d075\") " pod="openshift-multus/multus-5qqv9"
May 25 11:17:40.916575 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.916475    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b93f3e9c-f4f9-49dc-bfba-6d0f80c0971d-run-ovn\") pod \"ovnkube-node-m55v2\" (UID: \"b93f3e9c-f4f9-49dc-bfba-6d0f80c0971d\") " pod="openshift-ovn-kubernetes/ovnkube-node-m55v2"
May 25 11:17:40.916575 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.916478    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9bfeee33-bbdf-4407-9f32-5dfe1a29d075-host-var-lib-cni-multus\") pod \"multus-5qqv9\" (UID: \"9bfeee33-bbdf-4407-9f32-5dfe1a29d075\") " pod="openshift-multus/multus-5qqv9"
May 25 11:17:40.916575 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.916522    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9bfeee33-bbdf-4407-9f32-5dfe1a29d075-hostroot\") pod \"multus-5qqv9\" (UID: \"9bfeee33-bbdf-4407-9f32-5dfe1a29d075\") " pod="openshift-multus/multus-5qqv9"
May 25 11:17:40.916575 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.916550    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9bfeee33-bbdf-4407-9f32-5dfe1a29d075-multus-daemon-config\") pod \"multus-5qqv9\" (UID: \"9bfeee33-bbdf-4407-9f32-5dfe1a29d075\") " pod="openshift-multus/multus-5qqv9"
May 25 11:17:40.916808 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.916580    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0a24f6a0-7ac5-4f1e-84f0-301e16006665-iptables-alerter-script\") pod \"iptables-alerter-kj6wk\" (UID: \"0a24f6a0-7ac5-4f1e-84f0-301e16006665\") " pod="openshift-network-operator/iptables-alerter-kj6wk"
May 25 11:17:40.916808 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.916603    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0dc07c1-2dce-4b08-9fee-41a5937a7bae-metrics-certs\") pod \"network-metrics-daemon-hxdhk\" (UID: \"b0dc07c1-2dce-4b08-9fee-41a5937a7bae\") " pod="openshift-multus/network-metrics-daemon-hxdhk"
May 25 11:17:40.916808 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.916641    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b93f3e9c-f4f9-49dc-bfba-6d0f80c0971d-host-kubelet\") pod \"ovnkube-node-m55v2\" (UID: \"b93f3e9c-f4f9-49dc-bfba-6d0f80c0971d\") " pod="openshift-ovn-kubernetes/ovnkube-node-m55v2"
May 25 11:17:40.916808 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.916674    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b93f3e9c-f4f9-49dc-bfba-6d0f80c0971d-host-kubelet\") pod \"ovnkube-node-m55v2\" (UID: \"b93f3e9c-f4f9-49dc-bfba-6d0f80c0971d\") " pod="openshift-ovn-kubernetes/ovnkube-node-m55v2"
May 25 11:17:40.916808 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.916735    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b93f3e9c-f4f9-49dc-bfba-6d0f80c0971d-ovnkube-config\") pod \"ovnkube-node-m55v2\" (UID: \"b93f3e9c-f4f9-49dc-bfba-6d0f80c0971d\") " pod="openshift-ovn-kubernetes/ovnkube-node-m55v2"
May 25 11:17:40.916808 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:17:40.916783    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 11:17:40.916808 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.916804    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9bfeee33-bbdf-4407-9f32-5dfe1a29d075-multus-socket-dir-parent\") pod \"multus-5qqv9\" (UID: \"9bfeee33-bbdf-4407-9f32-5dfe1a29d075\") " pod="openshift-multus/multus-5qqv9"
May 25 11:17:40.917033 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:17:40.916851    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0dc07c1-2dce-4b08-9fee-41a5937a7bae-metrics-certs podName:b0dc07c1-2dce-4b08-9fee-41a5937a7bae nodeName:}" failed. No retries permitted until 2026-05-25 11:17:41.416821765 +0000 UTC m=+3.097799736 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b0dc07c1-2dce-4b08-9fee-41a5937a7bae-metrics-certs") pod "network-metrics-daemon-hxdhk" (UID: "b0dc07c1-2dce-4b08-9fee-41a5937a7bae") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 11:17:40.917033 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.916949    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/372c0836-0d39-4561-b0fa-7171eecad7ca-cni-binary-copy\") pod \"multus-additional-cni-plugins-vdfgw\" (UID: \"372c0836-0d39-4561-b0fa-7171eecad7ca\") " pod="openshift-multus/multus-additional-cni-plugins-vdfgw"
May 25 11:17:40.917033 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.916982    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/372c0836-0d39-4561-b0fa-7171eecad7ca-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vdfgw\" (UID: \"372c0836-0d39-4561-b0fa-7171eecad7ca\") " pod="openshift-multus/multus-additional-cni-plugins-vdfgw"
May 25 11:17:40.917033 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.917012    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/448f7dca-880c-4e81-8adc-d29e391dfefd-etc-tuned\") pod \"tuned-tgqrm\" (UID: \"448f7dca-880c-4e81-8adc-d29e391dfefd\") " pod="openshift-cluster-node-tuning-operator/tuned-tgqrm"
May 25 11:17:40.917182 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.917038    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b93f3e9c-f4f9-49dc-bfba-6d0f80c0971d-host-run-netns\") pod \"ovnkube-node-m55v2\" (UID: \"b93f3e9c-f4f9-49dc-bfba-6d0f80c0971d\") " pod="openshift-ovn-kubernetes/ovnkube-node-m55v2"
May 25 11:17:40.917182 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.917080    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b93f3e9c-f4f9-49dc-bfba-6d0f80c0971d-node-log\") pod \"ovnkube-node-m55v2\" (UID: \"b93f3e9c-f4f9-49dc-bfba-6d0f80c0971d\") " pod="openshift-ovn-kubernetes/ovnkube-node-m55v2"
May 25 11:17:40.917182 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.917107    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97fvl\" (UniqueName: \"kubernetes.io/projected/0d6828f5-3d2a-417d-ab45-6750dd8cdb92-kube-api-access-97fvl\") pod \"network-check-target-zfljp\" (UID: \"0d6828f5-3d2a-417d-ab45-6750dd8cdb92\") " pod="openshift-network-diagnostics/network-check-target-zfljp"
May 25 11:17:40.917182 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.917131    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/9bfeee33-bbdf-4407-9f32-5dfe1a29d075-os-release\") pod \"multus-5qqv9\" (UID: \"9bfeee33-bbdf-4407-9f32-5dfe1a29d075\") " pod="openshift-multus/multus-5qqv9"
May 25 11:17:40.917182 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.917170    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/372c0836-0d39-4561-b0fa-7171eecad7ca-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vdfgw\" (UID: \"372c0836-0d39-4561-b0fa-7171eecad7ca\") " pod="openshift-multus/multus-additional-cni-plugins-vdfgw"
May 25 11:17:40.917359 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.917184    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0a24f6a0-7ac5-4f1e-84f0-301e16006665-iptables-alerter-script\") pod \"iptables-alerter-kj6wk\" (UID: \"0a24f6a0-7ac5-4f1e-84f0-301e16006665\") " pod="openshift-network-operator/iptables-alerter-kj6wk"
May 25 11:17:40.917359 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.917198    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-kdts4\" (UniqueName: \"kubernetes.io/projected/b0dc07c1-2dce-4b08-9fee-41a5937a7bae-kube-api-access-kdts4\") pod \"network-metrics-daemon-hxdhk\" (UID: \"b0dc07c1-2dce-4b08-9fee-41a5937a7bae\") " pod="openshift-multus/network-metrics-daemon-hxdhk"
May 25 11:17:40.917359 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.917241    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b93f3e9c-f4f9-49dc-bfba-6d0f80c0971d-ovn-node-metrics-cert\") pod \"ovnkube-node-m55v2\" (UID: \"b93f3e9c-f4f9-49dc-bfba-6d0f80c0971d\") " pod="openshift-ovn-kubernetes/ovnkube-node-m55v2"
May 25 11:17:40.917359 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.917267    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5d252e49-a8a8-4794-befd-50a55e95b17b-agent-certs\") pod \"konnectivity-agent-fbljm\" (UID: \"5d252e49-a8a8-4794-befd-50a55e95b17b\") " pod="kube-system/konnectivity-agent-fbljm"
May 25 11:17:40.917359 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.917276    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b93f3e9c-f4f9-49dc-bfba-6d0f80c0971d-ovnkube-config\") pod \"ovnkube-node-m55v2\" (UID: \"b93f3e9c-f4f9-49dc-bfba-6d0f80c0971d\") " pod="openshift-ovn-kubernetes/ovnkube-node-m55v2"
May 25 11:17:40.917359 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.917291    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7c5c3c1e-f8a8-437b-972e-1c293222f327-socket-dir\") pod \"aws-ebs-csi-driver-node-lwkh6\" (UID: \"7c5c3c1e-f8a8-437b-972e-1c293222f327\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lwkh6"
May 25 11:17:40.917359 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.917298    2560 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
May 25 11:17:40.917359 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.917319    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-pnzzp\" (UniqueName: \"kubernetes.io/projected/7c5c3c1e-f8a8-437b-972e-1c293222f327-kube-api-access-pnzzp\") pod \"aws-ebs-csi-driver-node-lwkh6\" (UID: \"7c5c3c1e-f8a8-437b-972e-1c293222f327\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lwkh6"
May 25 11:17:40.917359 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.917334    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b93f3e9c-f4f9-49dc-bfba-6d0f80c0971d-host-run-netns\") pod \"ovnkube-node-m55v2\" (UID: \"b93f3e9c-f4f9-49dc-bfba-6d0f80c0971d\") " pod="openshift-ovn-kubernetes/ovnkube-node-m55v2"
May 25 11:17:40.917359 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.917342    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9bfeee33-bbdf-4407-9f32-5dfe1a29d075-host-run-netns\") pod \"multus-5qqv9\" (UID: \"9bfeee33-bbdf-4407-9f32-5dfe1a29d075\") " pod="openshift-multus/multus-5qqv9"
May 25 11:17:40.917359 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.917362    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b93f3e9c-f4f9-49dc-bfba-6d0f80c0971d-node-log\") pod \"ovnkube-node-m55v2\" (UID: \"b93f3e9c-f4f9-49dc-bfba-6d0f80c0971d\") " pod="openshift-ovn-kubernetes/ovnkube-node-m55v2"
May 25 11:17:40.917719 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.917368    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b93f3e9c-f4f9-49dc-bfba-6d0f80c0971d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-m55v2\" (UID: \"b93f3e9c-f4f9-49dc-bfba-6d0f80c0971d\") " pod="openshift-ovn-kubernetes/ovnkube-node-m55v2"
May 25 11:17:40.917719 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.917405    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c0068087-c30b-4ff5-a794-b84738aaa168-kubelet-config\") pod \"global-pull-secret-syncer-nq67w\" (UID: \"c0068087-c30b-4ff5-a794-b84738aaa168\") " pod="kube-system/global-pull-secret-syncer-nq67w"
May 25 11:17:40.917719 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.917432    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/448f7dca-880c-4e81-8adc-d29e391dfefd-etc-modprobe-d\") pod \"tuned-tgqrm\" (UID: \"448f7dca-880c-4e81-8adc-d29e391dfefd\") " pod="openshift-cluster-node-tuning-operator/tuned-tgqrm"
May 25 11:17:40.917719 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.917455    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7c5c3c1e-f8a8-437b-972e-1c293222f327-device-dir\") pod \"aws-ebs-csi-driver-node-lwkh6\" (UID: \"7c5c3c1e-f8a8-437b-972e-1c293222f327\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lwkh6"
May 25 11:17:40.917719 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.917481    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9bfeee33-bbdf-4407-9f32-5dfe1a29d075-cni-binary-copy\") pod \"multus-5qqv9\" (UID: \"9bfeee33-bbdf-4407-9f32-5dfe1a29d075\") " pod="openshift-multus/multus-5qqv9"
May 25 11:17:40.917719 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.917508    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-td25w\" (UniqueName: \"kubernetes.io/projected/0a24f6a0-7ac5-4f1e-84f0-301e16006665-kube-api-access-td25w\") pod \"iptables-alerter-kj6wk\" (UID: \"0a24f6a0-7ac5-4f1e-84f0-301e16006665\") " pod="openshift-network-operator/iptables-alerter-kj6wk"
May 25 11:17:40.917719 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.917515    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b93f3e9c-f4f9-49dc-bfba-6d0f80c0971d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-m55v2\" (UID: \"b93f3e9c-f4f9-49dc-bfba-6d0f80c0971d\") " pod="openshift-ovn-kubernetes/ovnkube-node-m55v2"
May 25 11:17:40.917719 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.917533    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/448f7dca-880c-4e81-8adc-d29e391dfefd-lib-modules\") pod \"tuned-tgqrm\" (UID: \"448f7dca-880c-4e81-8adc-d29e391dfefd\") " pod="openshift-cluster-node-tuning-operator/tuned-tgqrm"
May 25 11:17:40.917719 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.917556    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b93f3e9c-f4f9-49dc-bfba-6d0f80c0971d-host-cni-netd\") pod \"ovnkube-node-m55v2\" (UID: \"b93f3e9c-f4f9-49dc-bfba-6d0f80c0971d\") " pod="openshift-ovn-kubernetes/ovnkube-node-m55v2"
May 25 11:17:40.917719 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.917579    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7c5c3c1e-f8a8-437b-972e-1c293222f327-kubelet-dir\") pod \"aws-ebs-csi-driver-node-lwkh6\" (UID: \"7c5c3c1e-f8a8-437b-972e-1c293222f327\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lwkh6"
May 25 11:17:40.917719 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.917605    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2d4d0de1-9c41-46ef-89e3-9a0615a665af-serviceca\") pod \"node-ca-8vzrv\" (UID: \"2d4d0de1-9c41-46ef-89e3-9a0615a665af\") " pod="openshift-image-registry/node-ca-8vzrv"
May 25 11:17:40.917719 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.917622    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7c5c3c1e-f8a8-437b-972e-1c293222f327-socket-dir\") pod \"aws-ebs-csi-driver-node-lwkh6\" (UID: \"7c5c3c1e-f8a8-437b-972e-1c293222f327\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lwkh6"
May 25 11:17:40.917719 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.917634    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c0068087-c30b-4ff5-a794-b84738aaa168-original-pull-secret\") pod \"global-pull-secret-syncer-nq67w\" (UID: \"c0068087-c30b-4ff5-a794-b84738aaa168\") " pod="kube-system/global-pull-secret-syncer-nq67w"
May 25 11:17:40.917719 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.917661    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/448f7dca-880c-4e81-8adc-d29e391dfefd-etc-sysctl-conf\") pod \"tuned-tgqrm\" (UID: \"448f7dca-880c-4e81-8adc-d29e391dfefd\") " pod="openshift-cluster-node-tuning-operator/tuned-tgqrm"
May 25 11:17:40.917719 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.917695    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b93f3e9c-f4f9-49dc-bfba-6d0f80c0971d-etc-openvswitch\") pod \"ovnkube-node-m55v2\" (UID: \"b93f3e9c-f4f9-49dc-bfba-6d0f80c0971d\") " pod="openshift-ovn-kubernetes/ovnkube-node-m55v2"
May 25 11:17:40.918424 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.917736    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b93f3e9c-f4f9-49dc-bfba-6d0f80c0971d-log-socket\") pod \"ovnkube-node-m55v2\" (UID: \"b93f3e9c-f4f9-49dc-bfba-6d0f80c0971d\") " pod="openshift-ovn-kubernetes/ovnkube-node-m55v2"
May 25 11:17:40.918424 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.917761    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7c5c3c1e-f8a8-437b-972e-1c293222f327-device-dir\") pod \"aws-ebs-csi-driver-node-lwkh6\" (UID: \"7c5c3c1e-f8a8-437b-972e-1c293222f327\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lwkh6"
May 25 11:17:40.918424 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.917763    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b93f3e9c-f4f9-49dc-bfba-6d0f80c0971d-host-cni-bin\") pod \"ovnkube-node-m55v2\" (UID: \"b93f3e9c-f4f9-49dc-bfba-6d0f80c0971d\") " pod="openshift-ovn-kubernetes/ovnkube-node-m55v2"
May 25 11:17:40.918424 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.917798    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b93f3e9c-f4f9-49dc-bfba-6d0f80c0971d-env-overrides\") pod \"ovnkube-node-m55v2\" (UID: \"b93f3e9c-f4f9-49dc-bfba-6d0f80c0971d\") " pod="openshift-ovn-kubernetes/ovnkube-node-m55v2"
May 25 11:17:40.918424 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.917799    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b93f3e9c-f4f9-49dc-bfba-6d0f80c0971d-host-cni-bin\") pod \"ovnkube-node-m55v2\" (UID: \"b93f3e9c-f4f9-49dc-bfba-6d0f80c0971d\") " pod="openshift-ovn-kubernetes/ovnkube-node-m55v2"
May 25 11:17:40.918424 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.917831    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b93f3e9c-f4f9-49dc-bfba-6d0f80c0971d-host-cni-netd\") pod \"ovnkube-node-m55v2\" (UID: \"b93f3e9c-f4f9-49dc-bfba-6d0f80c0971d\") " pod="openshift-ovn-kubernetes/ovnkube-node-m55v2"
May 25 11:17:40.918424 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.917883    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7c5c3c1e-f8a8-437b-972e-1c293222f327-kubelet-dir\") pod \"aws-ebs-csi-driver-node-lwkh6\" (UID: \"7c5c3c1e-f8a8-437b-972e-1c293222f327\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lwkh6"
May 25 11:17:40.918424 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.917899    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/448f7dca-880c-4e81-8adc-d29e391dfefd-etc-modprobe-d\") pod \"tuned-tgqrm\" (UID: \"448f7dca-880c-4e81-8adc-d29e391dfefd\") " pod="openshift-cluster-node-tuning-operator/tuned-tgqrm"
May 25 11:17:40.918424 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.917977    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b93f3e9c-f4f9-49dc-bfba-6d0f80c0971d-log-socket\") pod \"ovnkube-node-m55v2\" (UID: \"b93f3e9c-f4f9-49dc-bfba-6d0f80c0971d\") " pod="openshift-ovn-kubernetes/ovnkube-node-m55v2"
May 25 11:17:40.918424 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.917989    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/448f7dca-880c-4e81-8adc-d29e391dfefd-lib-modules\") pod \"tuned-tgqrm\" (UID: \"448f7dca-880c-4e81-8adc-d29e391dfefd\") " pod="openshift-cluster-node-tuning-operator/tuned-tgqrm"
May 25 11:17:40.918424 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.918032    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0a24f6a0-7ac5-4f1e-84f0-301e16006665-host-slash\") pod \"iptables-alerter-kj6wk\" (UID: \"0a24f6a0-7ac5-4f1e-84f0-301e16006665\") " pod="openshift-network-operator/iptables-alerter-kj6wk"
May 25 11:17:40.918424 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.918059    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b93f3e9c-f4f9-49dc-bfba-6d0f80c0971d-systemd-units\") pod \"ovnkube-node-m55v2\" (UID: \"b93f3e9c-f4f9-49dc-bfba-6d0f80c0971d\") " pod="openshift-ovn-kubernetes/ovnkube-node-m55v2"
May 25 11:17:40.918424 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.918083    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7c5c3c1e-f8a8-437b-972e-1c293222f327-etc-selinux\") pod \"aws-ebs-csi-driver-node-lwkh6\" (UID: \"7c5c3c1e-f8a8-437b-972e-1c293222f327\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lwkh6"
May 25 11:17:40.918424 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.918098    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/448f7dca-880c-4e81-8adc-d29e391dfefd-etc-sysctl-conf\") pod \"tuned-tgqrm\" (UID: \"448f7dca-880c-4e81-8adc-d29e391dfefd\") " pod="openshift-cluster-node-tuning-operator/tuned-tgqrm"
May 25 11:17:40.918424 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.918110    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9bfeee33-bbdf-4407-9f32-5dfe1a29d075-cnibin\") pod \"multus-5qqv9\" (UID: \"9bfeee33-bbdf-4407-9f32-5dfe1a29d075\") " pod="openshift-multus/multus-5qqv9"
May 25 11:17:40.918424 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.918135    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9bfeee33-bbdf-4407-9f32-5dfe1a29d075-host-run-k8s-cni-cncf-io\") pod \"multus-5qqv9\" (UID: \"9bfeee33-bbdf-4407-9f32-5dfe1a29d075\") " pod="openshift-multus/multus-5qqv9"
May 25 11:17:40.918424 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.918163    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9bfeee33-bbdf-4407-9f32-5dfe1a29d075-host-run-multus-certs\") pod \"multus-5qqv9\" (UID: \"9bfeee33-bbdf-4407-9f32-5dfe1a29d075\") " pod="openshift-multus/multus-5qqv9"
May 25 11:17:40.919166 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.918168    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b93f3e9c-f4f9-49dc-bfba-6d0f80c0971d-systemd-units\") pod \"ovnkube-node-m55v2\" (UID: \"b93f3e9c-f4f9-49dc-bfba-6d0f80c0971d\") " pod="openshift-ovn-kubernetes/ovnkube-node-m55v2"
May 25 11:17:40.919166 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.918192    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/448f7dca-880c-4e81-8adc-d29e391dfefd-etc-systemd\") pod \"tuned-tgqrm\" (UID: \"448f7dca-880c-4e81-8adc-d29e391dfefd\") " pod="openshift-cluster-node-tuning-operator/tuned-tgqrm"
May 25 11:17:40.919166 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.918225    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0a24f6a0-7ac5-4f1e-84f0-301e16006665-host-slash\") pod \"iptables-alerter-kj6wk\" (UID: \"0a24f6a0-7ac5-4f1e-84f0-301e16006665\") " pod="openshift-network-operator/iptables-alerter-kj6wk"
May 25 11:17:40.919166 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.918240    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/448f7dca-880c-4e81-8adc-d29e391dfefd-run\") pod \"tuned-tgqrm\" (UID: \"448f7dca-880c-4e81-8adc-d29e391dfefd\") " pod="openshift-cluster-node-tuning-operator/tuned-tgqrm"
May 25 11:17:40.919166 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.918265    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/448f7dca-880c-4e81-8adc-d29e391dfefd-var-lib-kubelet\") pod \"tuned-tgqrm\" (UID: \"448f7dca-880c-4e81-8adc-d29e391dfefd\") " pod="openshift-cluster-node-tuning-operator/tuned-tgqrm"
May 25 11:17:40.919166 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.918283    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7c5c3c1e-f8a8-437b-972e-1c293222f327-etc-selinux\") pod \"aws-ebs-csi-driver-node-lwkh6\" (UID: \"7c5c3c1e-f8a8-437b-972e-1c293222f327\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lwkh6"
May 25 11:17:40.919166 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.918138    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b93f3e9c-f4f9-49dc-bfba-6d0f80c0971d-etc-openvswitch\") pod \"ovnkube-node-m55v2\" (UID: \"b93f3e9c-f4f9-49dc-bfba-6d0f80c0971d\") " pod="openshift-ovn-kubernetes/ovnkube-node-m55v2"
May 25 11:17:40.919166 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.918302    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b93f3e9c-f4f9-49dc-bfba-6d0f80c0971d-ovnkube-script-lib\") pod \"ovnkube-node-m55v2\" (UID: \"b93f3e9c-f4f9-49dc-bfba-6d0f80c0971d\") " pod="openshift-ovn-kubernetes/ovnkube-node-m55v2"
May 25 11:17:40.919166 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.918304    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b93f3e9c-f4f9-49dc-bfba-6d0f80c0971d-env-overrides\") pod \"ovnkube-node-m55v2\" (UID: \"b93f3e9c-f4f9-49dc-bfba-6d0f80c0971d\") " pod="openshift-ovn-kubernetes/ovnkube-node-m55v2"
May 25 11:17:40.919166 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.918329    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7c5c3c1e-f8a8-437b-972e-1c293222f327-sys-fs\") pod \"aws-ebs-csi-driver-node-lwkh6\" (UID: \"7c5c3c1e-f8a8-437b-972e-1c293222f327\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lwkh6"
May 25 11:17:40.919166 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.918331    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/448f7dca-880c-4e81-8adc-d29e391dfefd-etc-systemd\") pod \"tuned-tgqrm\" (UID: \"448f7dca-880c-4e81-8adc-d29e391dfefd\") " pod="openshift-cluster-node-tuning-operator/tuned-tgqrm"
May 25 11:17:40.919166 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.918275    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2d4d0de1-9c41-46ef-89e3-9a0615a665af-serviceca\") pod \"node-ca-8vzrv\" (UID: \"2d4d0de1-9c41-46ef-89e3-9a0615a665af\") " pod="openshift-image-registry/node-ca-8vzrv"
May 25 11:17:40.919166 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.918351    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/448f7dca-880c-4e81-8adc-d29e391dfefd-run\") pod \"tuned-tgqrm\" (UID: \"448f7dca-880c-4e81-8adc-d29e391dfefd\") " pod="openshift-cluster-node-tuning-operator/tuned-tgqrm"
May 25 11:17:40.919166 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.918367    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/448f7dca-880c-4e81-8adc-d29e391dfefd-var-lib-kubelet\") pod \"tuned-tgqrm\" (UID: \"448f7dca-880c-4e81-8adc-d29e391dfefd\") " pod="openshift-cluster-node-tuning-operator/tuned-tgqrm"
May 25 11:17:40.919166 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.918369    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2d4d0de1-9c41-46ef-89e3-9a0615a665af-host\") pod \"node-ca-8vzrv\" (UID: \"2d4d0de1-9c41-46ef-89e3-9a0615a665af\") " pod="openshift-image-registry/node-ca-8vzrv"
May 25 11:17:40.919166 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.918397    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7c5c3c1e-f8a8-437b-972e-1c293222f327-sys-fs\") pod \"aws-ebs-csi-driver-node-lwkh6\" (UID: \"7c5c3c1e-f8a8-437b-972e-1c293222f327\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lwkh6"
May 25 11:17:40.919166 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.918401    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2d4d0de1-9c41-46ef-89e3-9a0615a665af-host\") pod \"node-ca-8vzrv\" (UID: \"2d4d0de1-9c41-46ef-89e3-9a0615a665af\") " pod="openshift-image-registry/node-ca-8vzrv"
May 25 11:17:40.919166 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.918408    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9bfeee33-bbdf-4407-9f32-5dfe1a29d075-multus-conf-dir\") pod \"multus-5qqv9\" (UID: \"9bfeee33-bbdf-4407-9f32-5dfe1a29d075\") " pod="openshift-multus/multus-5qqv9"
May 25 11:17:40.919990 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.918434    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9bfeee33-bbdf-4407-9f32-5dfe1a29d075-etc-kubernetes\") pod \"multus-5qqv9\" (UID: \"9bfeee33-bbdf-4407-9f32-5dfe1a29d075\") " pod="openshift-multus/multus-5qqv9"
May 25 11:17:40.919990 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.918459    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/448f7dca-880c-4e81-8adc-d29e391dfefd-etc-sysconfig\") pod \"tuned-tgqrm\" (UID: \"448f7dca-880c-4e81-8adc-d29e391dfefd\") " pod="openshift-cluster-node-tuning-operator/tuned-tgqrm"
May 25 11:17:40.919990 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.918503    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/448f7dca-880c-4e81-8adc-d29e391dfefd-etc-sysconfig\") pod \"tuned-tgqrm\" (UID: \"448f7dca-880c-4e81-8adc-d29e391dfefd\") " pod="openshift-cluster-node-tuning-operator/tuned-tgqrm"
May 25 11:17:40.919990 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.918549    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/448f7dca-880c-4e81-8adc-d29e391dfefd-sys\") pod \"tuned-tgqrm\" (UID: \"448f7dca-880c-4e81-8adc-d29e391dfefd\") " pod="openshift-cluster-node-tuning-operator/tuned-tgqrm"
May 25 11:17:40.919990 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.918582    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b93f3e9c-f4f9-49dc-bfba-6d0f80c0971d-run-openvswitch\") pod \"ovnkube-node-m55v2\" (UID: \"b93f3e9c-f4f9-49dc-bfba-6d0f80c0971d\") " pod="openshift-ovn-kubernetes/ovnkube-node-m55v2"
May 25 11:17:40.919990 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.918615    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9p7jf\" (UniqueName: \"kubernetes.io/projected/372c0836-0d39-4561-b0fa-7171eecad7ca-kube-api-access-9p7jf\") pod \"multus-additional-cni-plugins-vdfgw\" (UID: \"372c0836-0d39-4561-b0fa-7171eecad7ca\") " pod="openshift-multus/multus-additional-cni-plugins-vdfgw"
May 25 11:17:40.919990 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.918645    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/448f7dca-880c-4e81-8adc-d29e391dfefd-etc-kubernetes\") pod \"tuned-tgqrm\" (UID: \"448f7dca-880c-4e81-8adc-d29e391dfefd\") " pod="openshift-cluster-node-tuning-operator/tuned-tgqrm"
May 25 11:17:40.919990 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.918669    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/448f7dca-880c-4e81-8adc-d29e391dfefd-host\") pod \"tuned-tgqrm\" (UID: \"448f7dca-880c-4e81-8adc-d29e391dfefd\") " pod="openshift-cluster-node-tuning-operator/tuned-tgqrm"
May 25 11:17:40.919990 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.918693    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/448f7dca-880c-4e81-8adc-d29e391dfefd-tmp\") pod \"tuned-tgqrm\" (UID: \"448f7dca-880c-4e81-8adc-d29e391dfefd\") " pod="openshift-cluster-node-tuning-operator/tuned-tgqrm"
May 25 11:17:40.919990 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.918713    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b93f3e9c-f4f9-49dc-bfba-6d0f80c0971d-run-openvswitch\") pod \"ovnkube-node-m55v2\" (UID: \"b93f3e9c-f4f9-49dc-bfba-6d0f80c0971d\") " pod="openshift-ovn-kubernetes/ovnkube-node-m55v2"
May 25 11:17:40.919990 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.918719    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9bfeee33-bbdf-4407-9f32-5dfe1a29d075-multus-cni-dir\") pod \"multus-5qqv9\" (UID: \"9bfeee33-bbdf-4407-9f32-5dfe1a29d075\") " pod="openshift-multus/multus-5qqv9"
May 25 11:17:40.919990 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.918754    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/372c0836-0d39-4561-b0fa-7171eecad7ca-cnibin\") pod \"multus-additional-cni-plugins-vdfgw\" (UID: \"372c0836-0d39-4561-b0fa-7171eecad7ca\") " pod="openshift-multus/multus-additional-cni-plugins-vdfgw"
May 25 11:17:40.919990 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.918782    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/372c0836-0d39-4561-b0fa-7171eecad7ca-os-release\") pod \"multus-additional-cni-plugins-vdfgw\" (UID: \"372c0836-0d39-4561-b0fa-7171eecad7ca\") " pod="openshift-multus/multus-additional-cni-plugins-vdfgw"
May 25 11:17:40.919990 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.918788    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/448f7dca-880c-4e81-8adc-d29e391dfefd-etc-kubernetes\") pod \"tuned-tgqrm\" (UID: \"448f7dca-880c-4e81-8adc-d29e391dfefd\") " pod="openshift-cluster-node-tuning-operator/tuned-tgqrm"
May 25 11:17:40.919990 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.918789    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b93f3e9c-f4f9-49dc-bfba-6d0f80c0971d-ovnkube-script-lib\") pod \"ovnkube-node-m55v2\" (UID: \"b93f3e9c-f4f9-49dc-bfba-6d0f80c0971d\") " pod="openshift-ovn-kubernetes/ovnkube-node-m55v2"
May 25 11:17:40.919990 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.918646    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/448f7dca-880c-4e81-8adc-d29e391dfefd-sys\") pod \"tuned-tgqrm\" (UID: \"448f7dca-880c-4e81-8adc-d29e391dfefd\") " pod="openshift-cluster-node-tuning-operator/tuned-tgqrm"
May 25 11:17:40.919990 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.918811    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b93f3e9c-f4f9-49dc-bfba-6d0f80c0971d-host-run-ovn-kubernetes\") pod \"ovnkube-node-m55v2\" (UID: \"b93f3e9c-f4f9-49dc-bfba-6d0f80c0971d\") " pod="openshift-ovn-kubernetes/ovnkube-node-m55v2"
May 25 11:17:40.921110 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.918839    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9bfeee33-bbdf-4407-9f32-5dfe1a29d075-host-var-lib-kubelet\") pod \"multus-5qqv9\" (UID: \"9bfeee33-bbdf-4407-9f32-5dfe1a29d075\") " pod="openshift-multus/multus-5qqv9"
May 25 11:17:40.921110 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.918854    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/448f7dca-880c-4e81-8adc-d29e391dfefd-host\") pod \"tuned-tgqrm\" (UID: \"448f7dca-880c-4e81-8adc-d29e391dfefd\") " pod="openshift-cluster-node-tuning-operator/tuned-tgqrm"
May 25 11:17:40.921110 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.918865    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xjfr\" (UniqueName: \"kubernetes.io/projected/9bfeee33-bbdf-4407-9f32-5dfe1a29d075-kube-api-access-6xjfr\") pod \"multus-5qqv9\" (UID: \"9bfeee33-bbdf-4407-9f32-5dfe1a29d075\") " pod="openshift-multus/multus-5qqv9"
May 25 11:17:40.921110 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.918891    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/372c0836-0d39-4561-b0fa-7171eecad7ca-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vdfgw\" (UID: \"372c0836-0d39-4561-b0fa-7171eecad7ca\") " pod="openshift-multus/multus-additional-cni-plugins-vdfgw"
May 25 11:17:40.921110 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.918919    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b93f3e9c-f4f9-49dc-bfba-6d0f80c0971d-run-systemd\") pod \"ovnkube-node-m55v2\" (UID: \"b93f3e9c-f4f9-49dc-bfba-6d0f80c0971d\") " pod="openshift-ovn-kubernetes/ovnkube-node-m55v2"
May 25 11:17:40.921110 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.918921    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b93f3e9c-f4f9-49dc-bfba-6d0f80c0971d-host-run-ovn-kubernetes\") pod \"ovnkube-node-m55v2\" (UID: \"b93f3e9c-f4f9-49dc-bfba-6d0f80c0971d\") " pod="openshift-ovn-kubernetes/ovnkube-node-m55v2"
May 25 11:17:40.921110 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.918950    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b93f3e9c-f4f9-49dc-bfba-6d0f80c0971d-run-systemd\") pod \"ovnkube-node-m55v2\" (UID: \"b93f3e9c-f4f9-49dc-bfba-6d0f80c0971d\") " pod="openshift-ovn-kubernetes/ovnkube-node-m55v2"
May 25 11:17:40.921110 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.918951    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/448f7dca-880c-4e81-8adc-d29e391dfefd-etc-sysctl-d\") pod \"tuned-tgqrm\" (UID: \"448f7dca-880c-4e81-8adc-d29e391dfefd\") " pod="openshift-cluster-node-tuning-operator/tuned-tgqrm"
May 25 11:17:40.921110 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.918975    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-5xsv5\" (UniqueName: \"kubernetes.io/projected/b93f3e9c-f4f9-49dc-bfba-6d0f80c0971d-kube-api-access-5xsv5\") pod \"ovnkube-node-m55v2\" (UID: \"b93f3e9c-f4f9-49dc-bfba-6d0f80c0971d\") " pod="openshift-ovn-kubernetes/ovnkube-node-m55v2"
May 25 11:17:40.921110 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.918999    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b93f3e9c-f4f9-49dc-bfba-6d0f80c0971d-var-lib-openvswitch\") pod \"ovnkube-node-m55v2\" (UID: \"b93f3e9c-f4f9-49dc-bfba-6d0f80c0971d\") " pod="openshift-ovn-kubernetes/ovnkube-node-m55v2"
May 25 11:17:40.921110 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.919018    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7c5c3c1e-f8a8-437b-972e-1c293222f327-registration-dir\") pod \"aws-ebs-csi-driver-node-lwkh6\" (UID: \"7c5c3c1e-f8a8-437b-972e-1c293222f327\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lwkh6"
May 25 11:17:40.921110 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.919030    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/448f7dca-880c-4e81-8adc-d29e391dfefd-etc-sysctl-d\") pod \"tuned-tgqrm\" (UID: \"448f7dca-880c-4e81-8adc-d29e391dfefd\") " pod="openshift-cluster-node-tuning-operator/tuned-tgqrm"
May 25 11:17:40.921110 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.919040    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-22dpr\" (UniqueName: \"kubernetes.io/projected/2d4d0de1-9c41-46ef-89e3-9a0615a665af-kube-api-access-22dpr\") pod \"node-ca-8vzrv\" (UID: \"2d4d0de1-9c41-46ef-89e3-9a0615a665af\") " pod="openshift-image-registry/node-ca-8vzrv"
May 25 11:17:40.921110 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.919057    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b93f3e9c-f4f9-49dc-bfba-6d0f80c0971d-var-lib-openvswitch\") pod \"ovnkube-node-m55v2\" (UID: \"b93f3e9c-f4f9-49dc-bfba-6d0f80c0971d\") " pod="openshift-ovn-kubernetes/ovnkube-node-m55v2"
May 25 11:17:40.921110 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.919065    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9bfeee33-bbdf-4407-9f32-5dfe1a29d075-host-var-lib-cni-bin\") pod \"multus-5qqv9\" (UID: \"9bfeee33-bbdf-4407-9f32-5dfe1a29d075\") " pod="openshift-multus/multus-5qqv9"
May 25 11:17:40.921110 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.919123    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7c5c3c1e-f8a8-437b-972e-1c293222f327-registration-dir\") pod \"aws-ebs-csi-driver-node-lwkh6\" (UID: \"7c5c3c1e-f8a8-437b-972e-1c293222f327\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lwkh6"
May 25 11:17:40.921110 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.919288    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/372c0836-0d39-4561-b0fa-7171eecad7ca-system-cni-dir\") pod \"multus-additional-cni-plugins-vdfgw\" (UID: \"372c0836-0d39-4561-b0fa-7171eecad7ca\") " pod="openshift-multus/multus-additional-cni-plugins-vdfgw"
May 25 11:17:40.921940 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.919324    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5d252e49-a8a8-4794-befd-50a55e95b17b-konnectivity-ca\") pod \"konnectivity-agent-fbljm\" (UID: \"5d252e49-a8a8-4794-befd-50a55e95b17b\") " pod="kube-system/konnectivity-agent-fbljm"
May 25 11:17:40.921940 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.919349    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-9hh9c\" (UniqueName: \"kubernetes.io/projected/448f7dca-880c-4e81-8adc-d29e391dfefd-kube-api-access-9hh9c\") pod \"tuned-tgqrm\" (UID: \"448f7dca-880c-4e81-8adc-d29e391dfefd\") " pod="openshift-cluster-node-tuning-operator/tuned-tgqrm"
May 25 11:17:40.921940 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.919380    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b93f3e9c-f4f9-49dc-bfba-6d0f80c0971d-host-slash\") pod \"ovnkube-node-m55v2\" (UID: \"b93f3e9c-f4f9-49dc-bfba-6d0f80c0971d\") " pod="openshift-ovn-kubernetes/ovnkube-node-m55v2"
May 25 11:17:40.921940 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.919406    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c0068087-c30b-4ff5-a794-b84738aaa168-dbus\") pod \"global-pull-secret-syncer-nq67w\" (UID: \"c0068087-c30b-4ff5-a794-b84738aaa168\") " pod="kube-system/global-pull-secret-syncer-nq67w"
May 25 11:17:40.921940 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.919657    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b93f3e9c-f4f9-49dc-bfba-6d0f80c0971d-host-slash\") pod \"ovnkube-node-m55v2\" (UID: \"b93f3e9c-f4f9-49dc-bfba-6d0f80c0971d\") " pod="openshift-ovn-kubernetes/ovnkube-node-m55v2"
May 25 11:17:40.921940 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.919852    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5d252e49-a8a8-4794-befd-50a55e95b17b-konnectivity-ca\") pod \"konnectivity-agent-fbljm\" (UID: \"5d252e49-a8a8-4794-befd-50a55e95b17b\") " pod="kube-system/konnectivity-agent-fbljm"
May 25 11:17:40.921940 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.920894    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/448f7dca-880c-4e81-8adc-d29e391dfefd-etc-tuned\") pod \"tuned-tgqrm\" (UID: \"448f7dca-880c-4e81-8adc-d29e391dfefd\") " pod="openshift-cluster-node-tuning-operator/tuned-tgqrm"
May 25 11:17:40.921940 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.920928    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b93f3e9c-f4f9-49dc-bfba-6d0f80c0971d-ovn-node-metrics-cert\") pod \"ovnkube-node-m55v2\" (UID: \"b93f3e9c-f4f9-49dc-bfba-6d0f80c0971d\") " pod="openshift-ovn-kubernetes/ovnkube-node-m55v2"
May 25 11:17:40.921940 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.920963    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5d252e49-a8a8-4794-befd-50a55e95b17b-agent-certs\") pod \"konnectivity-agent-fbljm\" (UID: \"5d252e49-a8a8-4794-befd-50a55e95b17b\") " pod="kube-system/konnectivity-agent-fbljm"
May 25 11:17:40.921940 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.921087    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/448f7dca-880c-4e81-8adc-d29e391dfefd-tmp\") pod \"tuned-tgqrm\" (UID: \"448f7dca-880c-4e81-8adc-d29e391dfefd\") " pod="openshift-cluster-node-tuning-operator/tuned-tgqrm"
May 25 11:17:40.943027 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.943008    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-22dpr\" (UniqueName: \"kubernetes.io/projected/2d4d0de1-9c41-46ef-89e3-9a0615a665af-kube-api-access-22dpr\") pod \"node-ca-8vzrv\" (UID: \"2d4d0de1-9c41-46ef-89e3-9a0615a665af\") " pod="openshift-image-registry/node-ca-8vzrv"
May 25 11:17:40.943116 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.943050    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hh9c\" (UniqueName: \"kubernetes.io/projected/448f7dca-880c-4e81-8adc-d29e391dfefd-kube-api-access-9hh9c\") pod \"tuned-tgqrm\" (UID: \"448f7dca-880c-4e81-8adc-d29e391dfefd\") " pod="openshift-cluster-node-tuning-operator/tuned-tgqrm"
May 25 11:17:40.943322 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.943258    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-td25w\" (UniqueName: \"kubernetes.io/projected/0a24f6a0-7ac5-4f1e-84f0-301e16006665-kube-api-access-td25w\") pod \"iptables-alerter-kj6wk\" (UID: \"0a24f6a0-7ac5-4f1e-84f0-301e16006665\") " pod="openshift-network-operator/iptables-alerter-kj6wk"
May 25 11:17:40.945173 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.945128    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnzzp\" (UniqueName: \"kubernetes.io/projected/7c5c3c1e-f8a8-437b-972e-1c293222f327-kube-api-access-pnzzp\") pod \"aws-ebs-csi-driver-node-lwkh6\" (UID: \"7c5c3c1e-f8a8-437b-972e-1c293222f327\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lwkh6"
May 25 11:17:40.945644 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.945623    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdts4\" (UniqueName: \"kubernetes.io/projected/b0dc07c1-2dce-4b08-9fee-41a5937a7bae-kube-api-access-kdts4\") pod \"network-metrics-daemon-hxdhk\" (UID: \"b0dc07c1-2dce-4b08-9fee-41a5937a7bae\") " pod="openshift-multus/network-metrics-daemon-hxdhk"
May 25 11:17:40.945825 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:40.945805    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xsv5\" (UniqueName: \"kubernetes.io/projected/b93f3e9c-f4f9-49dc-bfba-6d0f80c0971d-kube-api-access-5xsv5\") pod \"ovnkube-node-m55v2\" (UID: \"b93f3e9c-f4f9-49dc-bfba-6d0f80c0971d\") " pod="openshift-ovn-kubernetes/ovnkube-node-m55v2"
May 25 11:17:41.019869 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:41.019834    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-9p7jf\" (UniqueName: \"kubernetes.io/projected/372c0836-0d39-4561-b0fa-7171eecad7ca-kube-api-access-9p7jf\") pod \"multus-additional-cni-plugins-vdfgw\" (UID: \"372c0836-0d39-4561-b0fa-7171eecad7ca\") " pod="openshift-multus/multus-additional-cni-plugins-vdfgw"
May 25 11:17:41.020024 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:41.019877    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9bfeee33-bbdf-4407-9f32-5dfe1a29d075-multus-cni-dir\") pod \"multus-5qqv9\" (UID: \"9bfeee33-bbdf-4407-9f32-5dfe1a29d075\") " pod="openshift-multus/multus-5qqv9"
May 25 11:17:41.020024 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:41.019905    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/372c0836-0d39-4561-b0fa-7171eecad7ca-cnibin\") pod \"multus-additional-cni-plugins-vdfgw\" (UID: \"372c0836-0d39-4561-b0fa-7171eecad7ca\") " pod="openshift-multus/multus-additional-cni-plugins-vdfgw"
May 25 11:17:41.020024 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:41.019943    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/372c0836-0d39-4561-b0fa-7171eecad7ca-os-release\") pod \"multus-additional-cni-plugins-vdfgw\" (UID: \"372c0836-0d39-4561-b0fa-7171eecad7ca\") " pod="openshift-multus/multus-additional-cni-plugins-vdfgw"
May 25 11:17:41.020024 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:41.019972    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9bfeee33-bbdf-4407-9f32-5dfe1a29d075-host-var-lib-kubelet\") pod \"multus-5qqv9\" (UID: \"9bfeee33-bbdf-4407-9f32-5dfe1a29d075\") " pod="openshift-multus/multus-5qqv9"
May 25 11:17:41.020024 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:41.019996    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-6xjfr\" (UniqueName: \"kubernetes.io/projected/9bfeee33-bbdf-4407-9f32-5dfe1a29d075-kube-api-access-6xjfr\") pod \"multus-5qqv9\" (UID: \"9bfeee33-bbdf-4407-9f32-5dfe1a29d075\") " pod="openshift-multus/multus-5qqv9"
May 25 11:17:41.020308 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:41.020031    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/372c0836-0d39-4561-b0fa-7171eecad7ca-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vdfgw\" (UID: \"372c0836-0d39-4561-b0fa-7171eecad7ca\") " pod="openshift-multus/multus-additional-cni-plugins-vdfgw"
May 25 11:17:41.020308 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:41.020060    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9bfeee33-bbdf-4407-9f32-5dfe1a29d075-host-var-lib-cni-bin\") pod \"multus-5qqv9\" (UID: \"9bfeee33-bbdf-4407-9f32-5dfe1a29d075\") " pod="openshift-multus/multus-5qqv9"
May 25 11:17:41.020308 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:41.020091    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/372c0836-0d39-4561-b0fa-7171eecad7ca-system-cni-dir\") pod \"multus-additional-cni-plugins-vdfgw\" (UID: \"372c0836-0d39-4561-b0fa-7171eecad7ca\") " pod="openshift-multus/multus-additional-cni-plugins-vdfgw"
May 25 11:17:41.020308 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:41.020129    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c0068087-c30b-4ff5-a794-b84738aaa168-dbus\") pod \"global-pull-secret-syncer-nq67w\" (UID: \"c0068087-c30b-4ff5-a794-b84738aaa168\") " pod="kube-system/global-pull-secret-syncer-nq67w"
May 25 11:17:41.020308 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:41.020157    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9bfeee33-bbdf-4407-9f32-5dfe1a29d075-system-cni-dir\") pod \"multus-5qqv9\" (UID: \"9bfeee33-bbdf-4407-9f32-5dfe1a29d075\") " pod="openshift-multus/multus-5qqv9"
May 25 11:17:41.020308 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:41.020173    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9bfeee33-bbdf-4407-9f32-5dfe1a29d075-host-var-lib-kubelet\") pod \"multus-5qqv9\" (UID: \"9bfeee33-bbdf-4407-9f32-5dfe1a29d075\") " pod="openshift-multus/multus-5qqv9"
May 25 11:17:41.020308 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:41.020180    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9bfeee33-bbdf-4407-9f32-5dfe1a29d075-host-var-lib-cni-multus\") pod \"multus-5qqv9\" (UID: \"9bfeee33-bbdf-4407-9f32-5dfe1a29d075\") " pod="openshift-multus/multus-5qqv9"
May 25 11:17:41.020308 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:41.020243    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9bfeee33-bbdf-4407-9f32-5dfe1a29d075-host-var-lib-cni-multus\") pod \"multus-5qqv9\" (UID: \"9bfeee33-bbdf-4407-9f32-5dfe1a29d075\") " pod="openshift-multus/multus-5qqv9"
May 25 11:17:41.020308 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:41.020250    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9bfeee33-bbdf-4407-9f32-5dfe1a29d075-hostroot\") pod \"multus-5qqv9\" (UID: \"9bfeee33-bbdf-4407-9f32-5dfe1a29d075\") " pod="openshift-multus/multus-5qqv9"
May 25 11:17:41.020308 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:41.020284    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9bfeee33-bbdf-4407-9f32-5dfe1a29d075-multus-daemon-config\") pod \"multus-5qqv9\" (UID: \"9bfeee33-bbdf-4407-9f32-5dfe1a29d075\") " pod="openshift-multus/multus-5qqv9"
May 25 11:17:41.020308 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:41.020289    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9bfeee33-bbdf-4407-9f32-5dfe1a29d075-hostroot\") pod \"multus-5qqv9\" (UID: \"9bfeee33-bbdf-4407-9f32-5dfe1a29d075\") " pod="openshift-multus/multus-5qqv9"
May 25 11:17:41.020849 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:41.020327    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9bfeee33-bbdf-4407-9f32-5dfe1a29d075-multus-socket-dir-parent\") pod \"multus-5qqv9\" (UID: \"9bfeee33-bbdf-4407-9f32-5dfe1a29d075\") " pod="openshift-multus/multus-5qqv9"
May 25 11:17:41.020849 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:41.020351    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/372c0836-0d39-4561-b0fa-7171eecad7ca-cni-binary-copy\") pod \"multus-additional-cni-plugins-vdfgw\" (UID: \"372c0836-0d39-4561-b0fa-7171eecad7ca\") " pod="openshift-multus/multus-additional-cni-plugins-vdfgw"
May 25 11:17:41.020849 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:41.020354    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9bfeee33-bbdf-4407-9f32-5dfe1a29d075-multus-cni-dir\") pod \"multus-5qqv9\" (UID: \"9bfeee33-bbdf-4407-9f32-5dfe1a29d075\") " pod="openshift-multus/multus-5qqv9"
May 25 11:17:41.020849 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:41.020378    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/372c0836-0d39-4561-b0fa-7171eecad7ca-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vdfgw\" (UID: \"372c0836-0d39-4561-b0fa-7171eecad7ca\") " pod="openshift-multus/multus-additional-cni-plugins-vdfgw"
May 25 11:17:41.020849 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:41.020389    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/372c0836-0d39-4561-b0fa-7171eecad7ca-cnibin\") pod \"multus-additional-cni-plugins-vdfgw\" (UID: \"372c0836-0d39-4561-b0fa-7171eecad7ca\") " pod="openshift-multus/multus-additional-cni-plugins-vdfgw"
May 25 11:17:41.020849 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:41.020408    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-97fvl\" (UniqueName: \"kubernetes.io/projected/0d6828f5-3d2a-417d-ab45-6750dd8cdb92-kube-api-access-97fvl\") pod \"network-check-target-zfljp\" (UID: \"0d6828f5-3d2a-417d-ab45-6750dd8cdb92\") " pod="openshift-network-diagnostics/network-check-target-zfljp"
May 25 11:17:41.020849 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:41.020419    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/372c0836-0d39-4561-b0fa-7171eecad7ca-os-release\") pod \"multus-additional-cni-plugins-vdfgw\" (UID: \"372c0836-0d39-4561-b0fa-7171eecad7ca\") " pod="openshift-multus/multus-additional-cni-plugins-vdfgw"
May 25 11:17:41.020849 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:41.020428    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c0068087-c30b-4ff5-a794-b84738aaa168-dbus\") pod \"global-pull-secret-syncer-nq67w\" (UID: \"c0068087-c30b-4ff5-a794-b84738aaa168\") " pod="kube-system/global-pull-secret-syncer-nq67w"
May 25 11:17:41.020849 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:41.020446    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9bfeee33-bbdf-4407-9f32-5dfe1a29d075-system-cni-dir\") pod \"multus-5qqv9\" (UID: \"9bfeee33-bbdf-4407-9f32-5dfe1a29d075\") " pod="openshift-multus/multus-5qqv9"
May 25 11:17:41.020849 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:41.020461    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9bfeee33-bbdf-4407-9f32-5dfe1a29d075-multus-socket-dir-parent\") pod \"multus-5qqv9\" (UID: \"9bfeee33-bbdf-4407-9f32-5dfe1a29d075\") " pod="openshift-multus/multus-5qqv9"
May 25 11:17:41.020849 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:41.020494    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/9bfeee33-bbdf-4407-9f32-5dfe1a29d075-os-release\") pod \"multus-5qqv9\" (UID: \"9bfeee33-bbdf-4407-9f32-5dfe1a29d075\") " pod="openshift-multus/multus-5qqv9"
May 25 11:17:41.020849 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:41.020526    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/372c0836-0d39-4561-b0fa-7171eecad7ca-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vdfgw\" (UID: \"372c0836-0d39-4561-b0fa-7171eecad7ca\") " pod="openshift-multus/multus-additional-cni-plugins-vdfgw"
May 25 11:17:41.020849 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:41.020559    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9bfeee33-bbdf-4407-9f32-5dfe1a29d075-host-run-netns\") pod \"multus-5qqv9\" (UID: \"9bfeee33-bbdf-4407-9f32-5dfe1a29d075\") " pod="openshift-multus/multus-5qqv9"
May 25 11:17:41.020849 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:41.020566    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/372c0836-0d39-4561-b0fa-7171eecad7ca-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vdfgw\" (UID: \"372c0836-0d39-4561-b0fa-7171eecad7ca\") " pod="openshift-multus/multus-additional-cni-plugins-vdfgw"
May 25 11:17:41.020849 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:41.020582    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c0068087-c30b-4ff5-a794-b84738aaa168-kubelet-config\") pod \"global-pull-secret-syncer-nq67w\" (UID: \"c0068087-c30b-4ff5-a794-b84738aaa168\") " pod="kube-system/global-pull-secret-syncer-nq67w"
May 25 11:17:41.020849 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:41.020611    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9bfeee33-bbdf-4407-9f32-5dfe1a29d075-cni-binary-copy\") pod \"multus-5qqv9\" (UID: \"9bfeee33-bbdf-4407-9f32-5dfe1a29d075\") " pod="openshift-multus/multus-5qqv9"
May 25 11:17:41.020849 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:41.020617    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9bfeee33-bbdf-4407-9f32-5dfe1a29d075-host-var-lib-cni-bin\") pod \"multus-5qqv9\" (UID: \"9bfeee33-bbdf-4407-9f32-5dfe1a29d075\") " pod="openshift-multus/multus-5qqv9"
May 25 11:17:41.021568 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:41.020640    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c0068087-c30b-4ff5-a794-b84738aaa168-original-pull-secret\") pod \"global-pull-secret-syncer-nq67w\" (UID: \"c0068087-c30b-4ff5-a794-b84738aaa168\") " pod="kube-system/global-pull-secret-syncer-nq67w"
May 25 11:17:41.021568 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:41.020657    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/372c0836-0d39-4561-b0fa-7171eecad7ca-system-cni-dir\") pod \"multus-additional-cni-plugins-vdfgw\" (UID: \"372c0836-0d39-4561-b0fa-7171eecad7ca\") " pod="openshift-multus/multus-additional-cni-plugins-vdfgw"
May 25 11:17:41.021568 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:41.020680    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9bfeee33-bbdf-4407-9f32-5dfe1a29d075-cnibin\") pod \"multus-5qqv9\" (UID: \"9bfeee33-bbdf-4407-9f32-5dfe1a29d075\") " pod="openshift-multus/multus-5qqv9"
May 25 11:17:41.021568 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:41.020696    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9bfeee33-bbdf-4407-9f32-5dfe1a29d075-host-run-netns\") pod \"multus-5qqv9\" (UID: \"9bfeee33-bbdf-4407-9f32-5dfe1a29d075\") " pod="openshift-multus/multus-5qqv9"
May 25 11:17:41.021568 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:41.020704    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9bfeee33-bbdf-4407-9f32-5dfe1a29d075-host-run-k8s-cni-cncf-io\") pod \"multus-5qqv9\" (UID: \"9bfeee33-bbdf-4407-9f32-5dfe1a29d075\") " pod="openshift-multus/multus-5qqv9"
May 25 11:17:41.021568 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:41.020727    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9bfeee33-bbdf-4407-9f32-5dfe1a29d075-host-run-multus-certs\") pod \"multus-5qqv9\" (UID: \"9bfeee33-bbdf-4407-9f32-5dfe1a29d075\") " pod="openshift-multus/multus-5qqv9"
May 25 11:17:41.021568 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:41.020760    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9bfeee33-bbdf-4407-9f32-5dfe1a29d075-multus-conf-dir\") pod \"multus-5qqv9\" (UID: \"9bfeee33-bbdf-4407-9f32-5dfe1a29d075\") " pod="openshift-multus/multus-5qqv9"
May 25 11:17:41.021568 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:17:41.020778    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
May 25 11:17:41.021568 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:41.020785    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9bfeee33-bbdf-4407-9f32-5dfe1a29d075-etc-kubernetes\") pod \"multus-5qqv9\" (UID: \"9bfeee33-bbdf-4407-9f32-5dfe1a29d075\") " pod="openshift-multus/multus-5qqv9"
May 25 11:17:41.021568 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:17:41.020838    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0068087-c30b-4ff5-a794-b84738aaa168-original-pull-secret podName:c0068087-c30b-4ff5-a794-b84738aaa168 nodeName:}" failed. No retries permitted until 2026-05-25 11:17:41.520821184 +0000 UTC m=+3.201799140 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c0068087-c30b-4ff5-a794-b84738aaa168-original-pull-secret") pod "global-pull-secret-syncer-nq67w" (UID: "c0068087-c30b-4ff5-a794-b84738aaa168") : object "kube-system"/"original-pull-secret" not registered
May 25 11:17:41.021568 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:41.020851    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9bfeee33-bbdf-4407-9f32-5dfe1a29d075-etc-kubernetes\") pod \"multus-5qqv9\" (UID: \"9bfeee33-bbdf-4407-9f32-5dfe1a29d075\") " pod="openshift-multus/multus-5qqv9"
May 25 11:17:41.021568 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:41.020906    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9bfeee33-bbdf-4407-9f32-5dfe1a29d075-cnibin\") pod \"multus-5qqv9\" (UID: \"9bfeee33-bbdf-4407-9f32-5dfe1a29d075\") " pod="openshift-multus/multus-5qqv9"
May 25 11:17:41.021568 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:41.020950    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9bfeee33-bbdf-4407-9f32-5dfe1a29d075-host-run-k8s-cni-cncf-io\") pod \"multus-5qqv9\" (UID: \"9bfeee33-bbdf-4407-9f32-5dfe1a29d075\") " pod="openshift-multus/multus-5qqv9"
May 25 11:17:41.021568 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:41.020957    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c0068087-c30b-4ff5-a794-b84738aaa168-kubelet-config\") pod \"global-pull-secret-syncer-nq67w\" (UID: \"c0068087-c30b-4ff5-a794-b84738aaa168\") " pod="kube-system/global-pull-secret-syncer-nq67w"
May 25 11:17:41.021568 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:41.020992    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9bfeee33-bbdf-4407-9f32-5dfe1a29d075-host-run-multus-certs\") pod \"multus-5qqv9\" (UID: \"9bfeee33-bbdf-4407-9f32-5dfe1a29d075\") " pod="openshift-multus/multus-5qqv9"
May 25 11:17:41.021568 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:41.021005    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/372c0836-0d39-4561-b0fa-7171eecad7ca-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vdfgw\" (UID: \"372c0836-0d39-4561-b0fa-7171eecad7ca\") " pod="openshift-multus/multus-additional-cni-plugins-vdfgw"
May 25 11:17:41.021568 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:41.021046    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9bfeee33-bbdf-4407-9f32-5dfe1a29d075-multus-daemon-config\") pod \"multus-5qqv9\" (UID: \"9bfeee33-bbdf-4407-9f32-5dfe1a29d075\") " pod="openshift-multus/multus-5qqv9"
May 25 11:17:41.022181 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:41.021094    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9bfeee33-bbdf-4407-9f32-5dfe1a29d075-multus-conf-dir\") pod \"multus-5qqv9\" (UID: \"9bfeee33-bbdf-4407-9f32-5dfe1a29d075\") " pod="openshift-multus/multus-5qqv9"
May 25 11:17:41.022181 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:41.021094    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/9bfeee33-bbdf-4407-9f32-5dfe1a29d075-os-release\") pod \"multus-5qqv9\" (UID: \"9bfeee33-bbdf-4407-9f32-5dfe1a29d075\") " pod="openshift-multus/multus-5qqv9"
May 25 11:17:41.022181 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:41.021115    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/372c0836-0d39-4561-b0fa-7171eecad7ca-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vdfgw\" (UID: \"372c0836-0d39-4561-b0fa-7171eecad7ca\") " pod="openshift-multus/multus-additional-cni-plugins-vdfgw"
May 25 11:17:41.022181 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:41.021372    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9bfeee33-bbdf-4407-9f32-5dfe1a29d075-cni-binary-copy\") pod \"multus-5qqv9\" (UID: \"9bfeee33-bbdf-4407-9f32-5dfe1a29d075\") " pod="openshift-multus/multus-5qqv9"
May 25 11:17:41.022181 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:41.021465    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/372c0836-0d39-4561-b0fa-7171eecad7ca-cni-binary-copy\") pod \"multus-additional-cni-plugins-vdfgw\" (UID: \"372c0836-0d39-4561-b0fa-7171eecad7ca\") " pod="openshift-multus/multus-additional-cni-plugins-vdfgw"
May 25 11:17:41.033679 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:41.033631    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9p7jf\" (UniqueName: \"kubernetes.io/projected/372c0836-0d39-4561-b0fa-7171eecad7ca-kube-api-access-9p7jf\") pod \"multus-additional-cni-plugins-vdfgw\" (UID: \"372c0836-0d39-4561-b0fa-7171eecad7ca\") " pod="openshift-multus/multus-additional-cni-plugins-vdfgw"
May 25 11:17:41.039293 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:17:41.039277    2560 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 25 11:17:41.039293 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:17:41.039295    2560 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 25 11:17:41.039415 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:17:41.039305    2560 projected.go:194] Error preparing data for projected volume kube-api-access-97fvl for pod openshift-network-diagnostics/network-check-target-zfljp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 11:17:41.039415 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:17:41.039350    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0d6828f5-3d2a-417d-ab45-6750dd8cdb92-kube-api-access-97fvl podName:0d6828f5-3d2a-417d-ab45-6750dd8cdb92 nodeName:}" failed. No retries permitted until 2026-05-25 11:17:41.539337911 +0000 UTC m=+3.220315876 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-97fvl" (UniqueName: "kubernetes.io/projected/0d6828f5-3d2a-417d-ab45-6750dd8cdb92-kube-api-access-97fvl") pod "network-check-target-zfljp" (UID: "0d6828f5-3d2a-417d-ab45-6750dd8cdb92") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 11:17:41.041617 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:41.041590    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xjfr\" (UniqueName: \"kubernetes.io/projected/9bfeee33-bbdf-4407-9f32-5dfe1a29d075-kube-api-access-6xjfr\") pod \"multus-5qqv9\" (UID: \"9bfeee33-bbdf-4407-9f32-5dfe1a29d075\") " pod="openshift-multus/multus-5qqv9"
May 25 11:17:41.093492 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:41.093458    2560 reflector.go:430] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
May 25 11:17:41.107639 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:41.107607    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-kj6wk"
May 25 11:17:41.116020 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:41.115998    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-tgqrm"
May 25 11:17:41.125374 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:41.125355    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-m55v2"
May 25 11:17:41.129010 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:41.128987    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-fbljm"
May 25 11:17:41.135547 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:41.135529    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lwkh6"
May 25 11:17:41.142050 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:41.142033    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-8vzrv"
May 25 11:17:41.148608 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:41.148592    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-5qqv9"
May 25 11:17:41.153149 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:41.153131    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-vdfgw"
May 25 11:17:41.410085 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:41.410044    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb93f3e9c_f4f9_49dc_bfba_6d0f80c0971d.slice/crio-278eb894691f04e8c94d382c0f3b11c27e51a9f98b11ef3e78ee77cf89373e54 WatchSource:0}: Error finding container 278eb894691f04e8c94d382c0f3b11c27e51a9f98b11ef3e78ee77cf89373e54: Status 404 returned error can't find the container with id 278eb894691f04e8c94d382c0f3b11c27e51a9f98b11ef3e78ee77cf89373e54
May 25 11:17:41.411590 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:41.411508    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d252e49_a8a8_4794_befd_50a55e95b17b.slice/crio-ba9b00341382dadc68f8125e85bd3c01bfbaee5cbc4b004923b6db873a7603cf WatchSource:0}: Error finding container ba9b00341382dadc68f8125e85bd3c01bfbaee5cbc4b004923b6db873a7603cf: Status 404 returned error can't find the container with id ba9b00341382dadc68f8125e85bd3c01bfbaee5cbc4b004923b6db873a7603cf
May 25 11:17:41.412434 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:41.412307    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c5c3c1e_f8a8_437b_972e_1c293222f327.slice/crio-82093ef5e53cb12a421c4f52fff68dd1d45064c9f8b29aa2684c96f3d349deeb WatchSource:0}: Error finding container 82093ef5e53cb12a421c4f52fff68dd1d45064c9f8b29aa2684c96f3d349deeb: Status 404 returned error can't find the container with id 82093ef5e53cb12a421c4f52fff68dd1d45064c9f8b29aa2684c96f3d349deeb
May 25 11:17:41.413587 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:41.413420    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod372c0836_0d39_4561_b0fa_7171eecad7ca.slice/crio-f72054270ebe1133d3bbb5c3629957697029f1ea2c084f3bcf51df9990cd9224 WatchSource:0}: Error finding container f72054270ebe1133d3bbb5c3629957697029f1ea2c084f3bcf51df9990cd9224: Status 404 returned error can't find the container with id f72054270ebe1133d3bbb5c3629957697029f1ea2c084f3bcf51df9990cd9224
May 25 11:17:41.416913 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:41.416872    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bfeee33_bbdf_4407_9f32_5dfe1a29d075.slice/crio-f6dfdc566189fec8bf337e2d40ef647f9d5db81a82a004c0814c132f1c574dca WatchSource:0}: Error finding container f6dfdc566189fec8bf337e2d40ef647f9d5db81a82a004c0814c132f1c574dca: Status 404 returned error can't find the container with id f6dfdc566189fec8bf337e2d40ef647f9d5db81a82a004c0814c132f1c574dca
May 25 11:17:41.418080 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:41.418008    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod448f7dca_880c_4e81_8adc_d29e391dfefd.slice/crio-bf7d334cf2c58a9cb477b998a6c5b361fc03ec2e186c08e5e59ea6ee25df4f61 WatchSource:0}: Error finding container bf7d334cf2c58a9cb477b998a6c5b361fc03ec2e186c08e5e59ea6ee25df4f61: Status 404 returned error can't find the container with id bf7d334cf2c58a9cb477b998a6c5b361fc03ec2e186c08e5e59ea6ee25df4f61
May 25 11:17:41.419911 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:41.419884    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a24f6a0_7ac5_4f1e_84f0_301e16006665.slice/crio-160092e6d31159f29f57239471286136278b928ce7e2a45829bcbb38cd618947 WatchSource:0}: Error finding container 160092e6d31159f29f57239471286136278b928ce7e2a45829bcbb38cd618947: Status 404 returned error can't find the container with id 160092e6d31159f29f57239471286136278b928ce7e2a45829bcbb38cd618947
May 25 11:17:41.421907 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:17:41.421877    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d4d0de1_9c41_46ef_89e3_9a0615a665af.slice/crio-333042d24f2da1282cb6ecdc51262c44e41a2310db7e5d0ff4e9b5820c908e71 WatchSource:0}: Error finding container 333042d24f2da1282cb6ecdc51262c44e41a2310db7e5d0ff4e9b5820c908e71: Status 404 returned error can't find the container with id 333042d24f2da1282cb6ecdc51262c44e41a2310db7e5d0ff4e9b5820c908e71
May 25 11:17:41.422809 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:41.422766    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0dc07c1-2dce-4b08-9fee-41a5937a7bae-metrics-certs\") pod \"network-metrics-daemon-hxdhk\" (UID: \"b0dc07c1-2dce-4b08-9fee-41a5937a7bae\") " pod="openshift-multus/network-metrics-daemon-hxdhk"
May 25 11:17:41.423086 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:17:41.422966    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 11:17:41.423086 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:17:41.423034    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0dc07c1-2dce-4b08-9fee-41a5937a7bae-metrics-certs podName:b0dc07c1-2dce-4b08-9fee-41a5937a7bae nodeName:}" failed. No retries permitted until 2026-05-25 11:17:42.423013346 +0000 UTC m=+4.103991301 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b0dc07c1-2dce-4b08-9fee-41a5937a7bae-metrics-certs") pod "network-metrics-daemon-hxdhk" (UID: "b0dc07c1-2dce-4b08-9fee-41a5937a7bae") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 11:17:41.523343 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:41.523308    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c0068087-c30b-4ff5-a794-b84738aaa168-original-pull-secret\") pod \"global-pull-secret-syncer-nq67w\" (UID: \"c0068087-c30b-4ff5-a794-b84738aaa168\") " pod="kube-system/global-pull-secret-syncer-nq67w"
May 25 11:17:41.523487 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:17:41.523455    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
May 25 11:17:41.523558 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:17:41.523527    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0068087-c30b-4ff5-a794-b84738aaa168-original-pull-secret podName:c0068087-c30b-4ff5-a794-b84738aaa168 nodeName:}" failed. No retries permitted until 2026-05-25 11:17:42.523506941 +0000 UTC m=+4.204484895 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c0068087-c30b-4ff5-a794-b84738aaa168-original-pull-secret") pod "global-pull-secret-syncer-nq67w" (UID: "c0068087-c30b-4ff5-a794-b84738aaa168") : object "kube-system"/"original-pull-secret" not registered
May 25 11:17:41.623898 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:41.623863    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-97fvl\" (UniqueName: \"kubernetes.io/projected/0d6828f5-3d2a-417d-ab45-6750dd8cdb92-kube-api-access-97fvl\") pod \"network-check-target-zfljp\" (UID: \"0d6828f5-3d2a-417d-ab45-6750dd8cdb92\") " pod="openshift-network-diagnostics/network-check-target-zfljp"
May 25 11:17:41.624109 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:17:41.624006    2560 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 25 11:17:41.624109 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:17:41.624027    2560 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 25 11:17:41.624109 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:17:41.624040    2560 projected.go:194] Error preparing data for projected volume kube-api-access-97fvl for pod openshift-network-diagnostics/network-check-target-zfljp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 11:17:41.624303 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:17:41.624113    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0d6828f5-3d2a-417d-ab45-6750dd8cdb92-kube-api-access-97fvl podName:0d6828f5-3d2a-417d-ab45-6750dd8cdb92 nodeName:}" failed. No retries permitted until 2026-05-25 11:17:42.624094559 +0000 UTC m=+4.305072516 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-97fvl" (UniqueName: "kubernetes.io/projected/0d6828f5-3d2a-417d-ab45-6750dd8cdb92-kube-api-access-97fvl") pod "network-check-target-zfljp" (UID: "0d6828f5-3d2a-417d-ab45-6750dd8cdb92") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 11:17:41.866289 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:41.866242    2560 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-05-24 11:12:39 +0000 UTC" deadline="2028-02-21 22:15:33.025415678 +0000 UTC"
May 25 11:17:41.866289 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:41.866286    2560 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15298h57m51.159133047s"
May 25 11:17:41.931409 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:41.930769    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hxdhk"
May 25 11:17:41.931409 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:17:41.930915    2560 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hxdhk" podUID="b0dc07c1-2dce-4b08-9fee-41a5937a7bae"
May 25 11:17:41.957059 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:41.957018    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-146.ec2.internal" event={"ID":"205c636549bb34e2af97bc6edc40842f","Type":"ContainerStarted","Data":"b905e3729656efcf092095c0e0ecaeae1afb31c27fe40e57b3c9e87a6ecfd2df"}
May 25 11:17:41.958650 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:41.958622    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-tgqrm" event={"ID":"448f7dca-880c-4e81-8adc-d29e391dfefd","Type":"ContainerStarted","Data":"bf7d334cf2c58a9cb477b998a6c5b361fc03ec2e186c08e5e59ea6ee25df4f61"}
May 25 11:17:41.960535 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:41.960485    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-8vzrv" event={"ID":"2d4d0de1-9c41-46ef-89e3-9a0615a665af","Type":"ContainerStarted","Data":"333042d24f2da1282cb6ecdc51262c44e41a2310db7e5d0ff4e9b5820c908e71"}
May 25 11:17:41.963145 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:41.962992    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-5qqv9" event={"ID":"9bfeee33-bbdf-4407-9f32-5dfe1a29d075","Type":"ContainerStarted","Data":"f6dfdc566189fec8bf337e2d40ef647f9d5db81a82a004c0814c132f1c574dca"}
May 25 11:17:41.978539 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:41.978476    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lwkh6" event={"ID":"7c5c3c1e-f8a8-437b-972e-1c293222f327","Type":"ContainerStarted","Data":"82093ef5e53cb12a421c4f52fff68dd1d45064c9f8b29aa2684c96f3d349deeb"}
May 25 11:17:41.981641 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:41.981577    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vdfgw" event={"ID":"372c0836-0d39-4561-b0fa-7171eecad7ca","Type":"ContainerStarted","Data":"f72054270ebe1133d3bbb5c3629957697029f1ea2c084f3bcf51df9990cd9224"}
May 25 11:17:41.983765 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:41.983741    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-fbljm" event={"ID":"5d252e49-a8a8-4794-befd-50a55e95b17b","Type":"ContainerStarted","Data":"ba9b00341382dadc68f8125e85bd3c01bfbaee5cbc4b004923b6db873a7603cf"}
May 25 11:17:41.985970 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:41.985914    2560 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-146.ec2.internal" podStartSLOduration=1.985898331 podStartE2EDuration="1.985898331s" podCreationTimestamp="2026-05-25 11:17:40 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 11:17:41.984117988 +0000 UTC m=+3.665095971" watchObservedRunningTime="2026-05-25 11:17:41.985898331 +0000 UTC m=+3.666876306"
May 25 11:17:41.991045 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:41.990970    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m55v2" event={"ID":"b93f3e9c-f4f9-49dc-bfba-6d0f80c0971d","Type":"ContainerStarted","Data":"278eb894691f04e8c94d382c0f3b11c27e51a9f98b11ef3e78ee77cf89373e54"}
May 25 11:17:41.995525 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:41.995498    2560 generic.go:358] "Generic (PLEG): container finished" podID="f7c7694d1cd47f61dc1e9c46a604067d" containerID="b5d98f3485edfe5b637c86acdecb878f46ef1f27cd489f518139a0002700f1f8" exitCode=0
May 25 11:17:41.995925 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:41.995765    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-146.ec2.internal" event={"ID":"f7c7694d1cd47f61dc1e9c46a604067d","Type":"ContainerDied","Data":"b5d98f3485edfe5b637c86acdecb878f46ef1f27cd489f518139a0002700f1f8"}
May 25 11:17:42.000160 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:42.000115    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-kj6wk" event={"ID":"0a24f6a0-7ac5-4f1e-84f0-301e16006665","Type":"ContainerStarted","Data":"160092e6d31159f29f57239471286136278b928ce7e2a45829bcbb38cd618947"}
May 25 11:17:42.381040 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:42.380826    2560 reflector.go:430] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
May 25 11:17:42.407977 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:42.407941    2560 reflector.go:430] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
May 25 11:17:42.429846 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:42.429812    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0dc07c1-2dce-4b08-9fee-41a5937a7bae-metrics-certs\") pod \"network-metrics-daemon-hxdhk\" (UID: \"b0dc07c1-2dce-4b08-9fee-41a5937a7bae\") " pod="openshift-multus/network-metrics-daemon-hxdhk"
May 25 11:17:42.430020 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:17:42.429982    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 11:17:42.430083 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:17:42.430044    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0dc07c1-2dce-4b08-9fee-41a5937a7bae-metrics-certs podName:b0dc07c1-2dce-4b08-9fee-41a5937a7bae nodeName:}" failed. No retries permitted until 2026-05-25 11:17:44.430025793 +0000 UTC m=+6.111003743 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b0dc07c1-2dce-4b08-9fee-41a5937a7bae-metrics-certs") pod "network-metrics-daemon-hxdhk" (UID: "b0dc07c1-2dce-4b08-9fee-41a5937a7bae") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 11:17:42.531317 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:42.531236    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c0068087-c30b-4ff5-a794-b84738aaa168-original-pull-secret\") pod \"global-pull-secret-syncer-nq67w\" (UID: \"c0068087-c30b-4ff5-a794-b84738aaa168\") " pod="kube-system/global-pull-secret-syncer-nq67w"
May 25 11:17:42.531481 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:17:42.531389    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
May 25 11:17:42.531481 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:17:42.531451    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0068087-c30b-4ff5-a794-b84738aaa168-original-pull-secret podName:c0068087-c30b-4ff5-a794-b84738aaa168 nodeName:}" failed. No retries permitted until 2026-05-25 11:17:44.531432662 +0000 UTC m=+6.212410637 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c0068087-c30b-4ff5-a794-b84738aaa168-original-pull-secret") pod "global-pull-secret-syncer-nq67w" (UID: "c0068087-c30b-4ff5-a794-b84738aaa168") : object "kube-system"/"original-pull-secret" not registered
May 25 11:17:42.631948 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:42.631910    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-97fvl\" (UniqueName: \"kubernetes.io/projected/0d6828f5-3d2a-417d-ab45-6750dd8cdb92-kube-api-access-97fvl\") pod \"network-check-target-zfljp\" (UID: \"0d6828f5-3d2a-417d-ab45-6750dd8cdb92\") " pod="openshift-network-diagnostics/network-check-target-zfljp"
May 25 11:17:42.632149 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:17:42.632129    2560 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 25 11:17:42.632273 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:17:42.632151    2560 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 25 11:17:42.632273 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:17:42.632164    2560 projected.go:194] Error preparing data for projected volume kube-api-access-97fvl for pod openshift-network-diagnostics/network-check-target-zfljp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 11:17:42.632273 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:17:42.632234    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0d6828f5-3d2a-417d-ab45-6750dd8cdb92-kube-api-access-97fvl podName:0d6828f5-3d2a-417d-ab45-6750dd8cdb92 nodeName:}" failed. No retries permitted until 2026-05-25 11:17:44.632203076 +0000 UTC m=+6.313181032 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-97fvl" (UniqueName: "kubernetes.io/projected/0d6828f5-3d2a-417d-ab45-6750dd8cdb92-kube-api-access-97fvl") pod "network-check-target-zfljp" (UID: "0d6828f5-3d2a-417d-ab45-6750dd8cdb92") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 11:17:42.931101 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:42.930370    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nq67w"
May 25 11:17:42.931101 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:17:42.930497    2560 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nq67w" podUID="c0068087-c30b-4ff5-a794-b84738aaa168"
May 25 11:17:42.931101 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:42.930919    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zfljp"
May 25 11:17:42.931101 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:17:42.931008    2560 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zfljp" podUID="0d6828f5-3d2a-417d-ab45-6750dd8cdb92"
May 25 11:17:43.014357 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:43.014299    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-146.ec2.internal" event={"ID":"f7c7694d1cd47f61dc1e9c46a604067d","Type":"ContainerStarted","Data":"a38ed12af7e5b3308190a2a9e3bc5fcd8fe3bf3f4463f94bd071d59999f879a7"}
May 25 11:17:43.029775 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:43.029716    2560 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-146.ec2.internal" podStartSLOduration=3.029699733 podStartE2EDuration="3.029699733s" podCreationTimestamp="2026-05-25 11:17:40 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 11:17:43.029048927 +0000 UTC m=+4.710026901" watchObservedRunningTime="2026-05-25 11:17:43.029699733 +0000 UTC m=+4.710677708"
May 25 11:17:43.930267 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:43.930232    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hxdhk"
May 25 11:17:43.930453 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:17:43.930378    2560 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hxdhk" podUID="b0dc07c1-2dce-4b08-9fee-41a5937a7bae"
May 25 11:17:44.448420 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:44.448380    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0dc07c1-2dce-4b08-9fee-41a5937a7bae-metrics-certs\") pod \"network-metrics-daemon-hxdhk\" (UID: \"b0dc07c1-2dce-4b08-9fee-41a5937a7bae\") " pod="openshift-multus/network-metrics-daemon-hxdhk"
May 25 11:17:44.448853 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:17:44.448549    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 11:17:44.448853 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:17:44.448632    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0dc07c1-2dce-4b08-9fee-41a5937a7bae-metrics-certs podName:b0dc07c1-2dce-4b08-9fee-41a5937a7bae nodeName:}" failed. No retries permitted until 2026-05-25 11:17:48.448609291 +0000 UTC m=+10.129587257 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b0dc07c1-2dce-4b08-9fee-41a5937a7bae-metrics-certs") pod "network-metrics-daemon-hxdhk" (UID: "b0dc07c1-2dce-4b08-9fee-41a5937a7bae") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 11:17:44.549751 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:44.549663    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c0068087-c30b-4ff5-a794-b84738aaa168-original-pull-secret\") pod \"global-pull-secret-syncer-nq67w\" (UID: \"c0068087-c30b-4ff5-a794-b84738aaa168\") " pod="kube-system/global-pull-secret-syncer-nq67w"
May 25 11:17:44.549931 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:17:44.549816    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
May 25 11:17:44.550287 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:17:44.549891    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0068087-c30b-4ff5-a794-b84738aaa168-original-pull-secret podName:c0068087-c30b-4ff5-a794-b84738aaa168 nodeName:}" failed. No retries permitted until 2026-05-25 11:17:48.549871889 +0000 UTC m=+10.230849864 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c0068087-c30b-4ff5-a794-b84738aaa168-original-pull-secret") pod "global-pull-secret-syncer-nq67w" (UID: "c0068087-c30b-4ff5-a794-b84738aaa168") : object "kube-system"/"original-pull-secret" not registered
May 25 11:17:44.650387 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:44.650341    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-97fvl\" (UniqueName: \"kubernetes.io/projected/0d6828f5-3d2a-417d-ab45-6750dd8cdb92-kube-api-access-97fvl\") pod \"network-check-target-zfljp\" (UID: \"0d6828f5-3d2a-417d-ab45-6750dd8cdb92\") " pod="openshift-network-diagnostics/network-check-target-zfljp"
May 25 11:17:44.650572 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:17:44.650554    2560 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 25 11:17:44.650644 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:17:44.650577    2560 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 25 11:17:44.650644 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:17:44.650590    2560 projected.go:194] Error preparing data for projected volume kube-api-access-97fvl for pod openshift-network-diagnostics/network-check-target-zfljp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 11:17:44.650747 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:17:44.650652    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0d6828f5-3d2a-417d-ab45-6750dd8cdb92-kube-api-access-97fvl podName:0d6828f5-3d2a-417d-ab45-6750dd8cdb92 nodeName:}" failed. No retries permitted until 2026-05-25 11:17:48.650635107 +0000 UTC m=+10.331613064 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-97fvl" (UniqueName: "kubernetes.io/projected/0d6828f5-3d2a-417d-ab45-6750dd8cdb92-kube-api-access-97fvl") pod "network-check-target-zfljp" (UID: "0d6828f5-3d2a-417d-ab45-6750dd8cdb92") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 11:17:44.930167 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:44.930088    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zfljp"
May 25 11:17:44.930326 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:17:44.930239    2560 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zfljp" podUID="0d6828f5-3d2a-417d-ab45-6750dd8cdb92"
May 25 11:17:44.930619 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:44.930088    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nq67w"
May 25 11:17:44.930739 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:17:44.930715    2560 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nq67w" podUID="c0068087-c30b-4ff5-a794-b84738aaa168"
May 25 11:17:45.930632 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:45.930586    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hxdhk"
May 25 11:17:45.931066 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:17:45.930756    2560 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hxdhk" podUID="b0dc07c1-2dce-4b08-9fee-41a5937a7bae"
May 25 11:17:46.929997 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:46.929967    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zfljp"
May 25 11:17:46.930234 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:17:46.930100    2560 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zfljp" podUID="0d6828f5-3d2a-417d-ab45-6750dd8cdb92"
May 25 11:17:46.930234 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:46.930168    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nq67w"
May 25 11:17:46.930401 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:17:46.930329    2560 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nq67w" podUID="c0068087-c30b-4ff5-a794-b84738aaa168"
May 25 11:17:47.930256 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:47.930153    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hxdhk"
May 25 11:17:47.930727 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:17:47.930333    2560 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hxdhk" podUID="b0dc07c1-2dce-4b08-9fee-41a5937a7bae"
May 25 11:17:48.483816 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:48.483498    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0dc07c1-2dce-4b08-9fee-41a5937a7bae-metrics-certs\") pod \"network-metrics-daemon-hxdhk\" (UID: \"b0dc07c1-2dce-4b08-9fee-41a5937a7bae\") " pod="openshift-multus/network-metrics-daemon-hxdhk"
May 25 11:17:48.483816 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:17:48.483656    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 11:17:48.483816 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:17:48.483720    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0dc07c1-2dce-4b08-9fee-41a5937a7bae-metrics-certs podName:b0dc07c1-2dce-4b08-9fee-41a5937a7bae nodeName:}" failed. No retries permitted until 2026-05-25 11:17:56.483701024 +0000 UTC m=+18.164678989 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b0dc07c1-2dce-4b08-9fee-41a5937a7bae-metrics-certs") pod "network-metrics-daemon-hxdhk" (UID: "b0dc07c1-2dce-4b08-9fee-41a5937a7bae") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 11:17:48.584495 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:48.584454    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c0068087-c30b-4ff5-a794-b84738aaa168-original-pull-secret\") pod \"global-pull-secret-syncer-nq67w\" (UID: \"c0068087-c30b-4ff5-a794-b84738aaa168\") " pod="kube-system/global-pull-secret-syncer-nq67w"
May 25 11:17:48.584664 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:17:48.584624    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
May 25 11:17:48.584743 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:17:48.584694    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0068087-c30b-4ff5-a794-b84738aaa168-original-pull-secret podName:c0068087-c30b-4ff5-a794-b84738aaa168 nodeName:}" failed. No retries permitted until 2026-05-25 11:17:56.58467588 +0000 UTC m=+18.265653836 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c0068087-c30b-4ff5-a794-b84738aaa168-original-pull-secret") pod "global-pull-secret-syncer-nq67w" (UID: "c0068087-c30b-4ff5-a794-b84738aaa168") : object "kube-system"/"original-pull-secret" not registered
May 25 11:17:48.686016 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:48.685465    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-97fvl\" (UniqueName: \"kubernetes.io/projected/0d6828f5-3d2a-417d-ab45-6750dd8cdb92-kube-api-access-97fvl\") pod \"network-check-target-zfljp\" (UID: \"0d6828f5-3d2a-417d-ab45-6750dd8cdb92\") " pod="openshift-network-diagnostics/network-check-target-zfljp"
May 25 11:17:48.686016 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:17:48.685649    2560 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 25 11:17:48.686016 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:17:48.685669    2560 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 25 11:17:48.686016 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:17:48.685681    2560 projected.go:194] Error preparing data for projected volume kube-api-access-97fvl for pod openshift-network-diagnostics/network-check-target-zfljp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 11:17:48.686016 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:17:48.685748    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0d6828f5-3d2a-417d-ab45-6750dd8cdb92-kube-api-access-97fvl podName:0d6828f5-3d2a-417d-ab45-6750dd8cdb92 nodeName:}" failed. No retries permitted until 2026-05-25 11:17:56.685729775 +0000 UTC m=+18.366707728 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-97fvl" (UniqueName: "kubernetes.io/projected/0d6828f5-3d2a-417d-ab45-6750dd8cdb92-kube-api-access-97fvl") pod "network-check-target-zfljp" (UID: "0d6828f5-3d2a-417d-ab45-6750dd8cdb92") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 11:17:48.931811 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:48.931777    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zfljp"
May 25 11:17:48.932269 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:17:48.931891    2560 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zfljp" podUID="0d6828f5-3d2a-417d-ab45-6750dd8cdb92"
May 25 11:17:48.932269 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:48.932179    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nq67w"
May 25 11:17:48.932396 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:17:48.932298    2560 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nq67w" podUID="c0068087-c30b-4ff5-a794-b84738aaa168"
May 25 11:17:49.929843 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:49.929809    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hxdhk"
May 25 11:17:49.930086 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:17:49.929957    2560 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hxdhk" podUID="b0dc07c1-2dce-4b08-9fee-41a5937a7bae"
May 25 11:17:50.930257 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:50.930220    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nq67w"
May 25 11:17:50.930623 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:50.930222    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zfljp"
May 25 11:17:50.930623 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:17:50.930327    2560 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nq67w" podUID="c0068087-c30b-4ff5-a794-b84738aaa168"
May 25 11:17:50.930623 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:17:50.930422    2560 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zfljp" podUID="0d6828f5-3d2a-417d-ab45-6750dd8cdb92"
May 25 11:17:51.930553 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:51.930521    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hxdhk"
May 25 11:17:51.930947 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:17:51.930636    2560 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hxdhk" podUID="b0dc07c1-2dce-4b08-9fee-41a5937a7bae"
May 25 11:17:52.930549 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:52.930517    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nq67w"
May 25 11:17:52.930704 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:52.930516    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zfljp"
May 25 11:17:52.930704 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:17:52.930606    2560 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nq67w" podUID="c0068087-c30b-4ff5-a794-b84738aaa168"
May 25 11:17:52.931130 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:17:52.930735    2560 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zfljp" podUID="0d6828f5-3d2a-417d-ab45-6750dd8cdb92"
May 25 11:17:53.930015 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:53.929975    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hxdhk"
May 25 11:17:53.930191 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:17:53.930126    2560 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hxdhk" podUID="b0dc07c1-2dce-4b08-9fee-41a5937a7bae"
May 25 11:17:54.930705 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:54.930495    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zfljp"
May 25 11:17:54.931144 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:54.930528    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nq67w"
May 25 11:17:54.931144 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:17:54.930808    2560 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zfljp" podUID="0d6828f5-3d2a-417d-ab45-6750dd8cdb92"
May 25 11:17:54.931144 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:17:54.930884    2560 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nq67w" podUID="c0068087-c30b-4ff5-a794-b84738aaa168"
May 25 11:17:55.930795 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:55.930755    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hxdhk"
May 25 11:17:55.931344 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:17:55.930903    2560 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hxdhk" podUID="b0dc07c1-2dce-4b08-9fee-41a5937a7bae"
May 25 11:17:56.543156 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:56.543113    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0dc07c1-2dce-4b08-9fee-41a5937a7bae-metrics-certs\") pod \"network-metrics-daemon-hxdhk\" (UID: \"b0dc07c1-2dce-4b08-9fee-41a5937a7bae\") " pod="openshift-multus/network-metrics-daemon-hxdhk"
May 25 11:17:56.543343 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:17:56.543318    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 11:17:56.543399 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:17:56.543387    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0dc07c1-2dce-4b08-9fee-41a5937a7bae-metrics-certs podName:b0dc07c1-2dce-4b08-9fee-41a5937a7bae nodeName:}" failed. No retries permitted until 2026-05-25 11:18:12.543371778 +0000 UTC m=+34.224349733 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b0dc07c1-2dce-4b08-9fee-41a5937a7bae-metrics-certs") pod "network-metrics-daemon-hxdhk" (UID: "b0dc07c1-2dce-4b08-9fee-41a5937a7bae") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 11:17:56.644189 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:56.644151    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c0068087-c30b-4ff5-a794-b84738aaa168-original-pull-secret\") pod \"global-pull-secret-syncer-nq67w\" (UID: \"c0068087-c30b-4ff5-a794-b84738aaa168\") " pod="kube-system/global-pull-secret-syncer-nq67w"
May 25 11:17:56.644358 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:17:56.644315    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
May 25 11:17:56.644414 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:17:56.644393    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0068087-c30b-4ff5-a794-b84738aaa168-original-pull-secret podName:c0068087-c30b-4ff5-a794-b84738aaa168 nodeName:}" failed. No retries permitted until 2026-05-25 11:18:12.644376346 +0000 UTC m=+34.325354320 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c0068087-c30b-4ff5-a794-b84738aaa168-original-pull-secret") pod "global-pull-secret-syncer-nq67w" (UID: "c0068087-c30b-4ff5-a794-b84738aaa168") : object "kube-system"/"original-pull-secret" not registered
May 25 11:17:56.744743 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:56.744702    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-97fvl\" (UniqueName: \"kubernetes.io/projected/0d6828f5-3d2a-417d-ab45-6750dd8cdb92-kube-api-access-97fvl\") pod \"network-check-target-zfljp\" (UID: \"0d6828f5-3d2a-417d-ab45-6750dd8cdb92\") " pod="openshift-network-diagnostics/network-check-target-zfljp"
May 25 11:17:56.744920 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:17:56.744876    2560 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 25 11:17:56.744920 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:17:56.744904    2560 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 25 11:17:56.744920 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:17:56.744917    2560 projected.go:194] Error preparing data for projected volume kube-api-access-97fvl for pod openshift-network-diagnostics/network-check-target-zfljp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 11:17:56.745044 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:17:56.744977    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0d6828f5-3d2a-417d-ab45-6750dd8cdb92-kube-api-access-97fvl podName:0d6828f5-3d2a-417d-ab45-6750dd8cdb92 nodeName:}" failed. No retries permitted until 2026-05-25 11:18:12.744957867 +0000 UTC m=+34.425935830 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-97fvl" (UniqueName: "kubernetes.io/projected/0d6828f5-3d2a-417d-ab45-6750dd8cdb92-kube-api-access-97fvl") pod "network-check-target-zfljp" (UID: "0d6828f5-3d2a-417d-ab45-6750dd8cdb92") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 11:17:56.930089 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:56.930011    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zfljp"
May 25 11:17:56.930268 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:17:56.930136    2560 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zfljp" podUID="0d6828f5-3d2a-417d-ab45-6750dd8cdb92"
May 25 11:17:56.930268 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:56.930198    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nq67w"
May 25 11:17:56.930383 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:17:56.930314    2560 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nq67w" podUID="c0068087-c30b-4ff5-a794-b84738aaa168"
May 25 11:17:57.930182 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:57.930145    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hxdhk"
May 25 11:17:57.930637 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:17:57.930298    2560 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hxdhk" podUID="b0dc07c1-2dce-4b08-9fee-41a5937a7bae"
May 25 11:17:58.930519 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:58.930485    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zfljp"
May 25 11:17:58.930908 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:17:58.930559    2560 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zfljp" podUID="0d6828f5-3d2a-417d-ab45-6750dd8cdb92"
May 25 11:17:58.930908 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:58.930648    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nq67w"
May 25 11:17:58.930908 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:17:58.930754    2560 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nq67w" podUID="c0068087-c30b-4ff5-a794-b84738aaa168"
May 25 11:17:59.930850 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:17:59.930594    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hxdhk"
May 25 11:17:59.931607 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:17:59.930911    2560 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hxdhk" podUID="b0dc07c1-2dce-4b08-9fee-41a5937a7bae"
May 25 11:18:00.046862 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:00.046823    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-tgqrm" event={"ID":"448f7dca-880c-4e81-8adc-d29e391dfefd","Type":"ContainerStarted","Data":"19fa47210c015ef328c362aa8c7f85661bbcce18e4811a7ee40ffacf9b671823"}
May 25 11:18:00.048081 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:00.048034    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-8vzrv" event={"ID":"2d4d0de1-9c41-46ef-89e3-9a0615a665af","Type":"ContainerStarted","Data":"cbc231aa996142b1993bb5f50370f46044fc3bfa62802f5810350fa859749eda"}
May 25 11:18:00.049283 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:00.049259    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-5qqv9" event={"ID":"9bfeee33-bbdf-4407-9f32-5dfe1a29d075","Type":"ContainerStarted","Data":"7cb2625134abb444a592f11cd8a4e04a9ece89d7ea553d3ba5a0a70063e6ac68"}
May 25 11:18:00.050406 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:00.050384    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lwkh6" event={"ID":"7c5c3c1e-f8a8-437b-972e-1c293222f327","Type":"ContainerStarted","Data":"4256400647b6dd810af1265606171b3af7a9aac8198b62ac6ec56c6fed37aea2"}
May 25 11:18:00.051634 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:00.051614    2560 generic.go:358] "Generic (PLEG): container finished" podID="372c0836-0d39-4561-b0fa-7171eecad7ca" containerID="144b26c4bdb36ba4d4ecdc08132783691376bfcf80173ae25d16ff5abe9e087f" exitCode=0
May 25 11:18:00.051705 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:00.051668    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vdfgw" event={"ID":"372c0836-0d39-4561-b0fa-7171eecad7ca","Type":"ContainerDied","Data":"144b26c4bdb36ba4d4ecdc08132783691376bfcf80173ae25d16ff5abe9e087f"}
May 25 11:18:00.052851 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:00.052830    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-fbljm" event={"ID":"5d252e49-a8a8-4794-befd-50a55e95b17b","Type":"ContainerStarted","Data":"adc05e7a399df6b3a6ca8c9f4dc5b6a28afd7d61e0752db8a6ba9233f962164f"}
May 25 11:18:00.055149 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:00.055130    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m55v2_b93f3e9c-f4f9-49dc-bfba-6d0f80c0971d/ovn-acl-logging/0.log"
May 25 11:18:00.055442 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:00.055414    2560 generic.go:358] "Generic (PLEG): container finished" podID="b93f3e9c-f4f9-49dc-bfba-6d0f80c0971d" containerID="a434fb8506907a44dc4d4dbd46b0dc49c047a87c159802cbbe1e27ddfe9dc835" exitCode=1
May 25 11:18:00.055490 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:00.055458    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m55v2" event={"ID":"b93f3e9c-f4f9-49dc-bfba-6d0f80c0971d","Type":"ContainerStarted","Data":"c4217233831b1be5adf4f1e5c9bf25f4265e83dc5d9cf6c26b10fc7d0ec988a8"}
May 25 11:18:00.055490 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:00.055474    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m55v2" event={"ID":"b93f3e9c-f4f9-49dc-bfba-6d0f80c0971d","Type":"ContainerStarted","Data":"c5a9dae07da571b3ce7f5b2ff42732db4339777ef4d79906d529b603ff46a252"}
May 25 11:18:00.055490 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:00.055488    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m55v2" event={"ID":"b93f3e9c-f4f9-49dc-bfba-6d0f80c0971d","Type":"ContainerStarted","Data":"43e007ab7f1a00999190c30e9b5eec95240b853e30a9582d8e9f69d6954c71df"}
May 25 11:18:00.055590 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:00.055496    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m55v2" event={"ID":"b93f3e9c-f4f9-49dc-bfba-6d0f80c0971d","Type":"ContainerStarted","Data":"5c8cfe772b8f07f62eedac52a6667b891d190df51d4b60d537342e3467bcf3f9"}
May 25 11:18:00.055590 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:00.055504    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m55v2" event={"ID":"b93f3e9c-f4f9-49dc-bfba-6d0f80c0971d","Type":"ContainerDied","Data":"a434fb8506907a44dc4d4dbd46b0dc49c047a87c159802cbbe1e27ddfe9dc835"}
May 25 11:18:00.055590 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:00.055512    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m55v2" event={"ID":"b93f3e9c-f4f9-49dc-bfba-6d0f80c0971d","Type":"ContainerStarted","Data":"f385081b95a1006a8278da2dbda026113b02c162cca85f2ae3d62ed1eee156f1"}
May 25 11:18:00.067978 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:00.067945    2560 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-tgqrm" podStartSLOduration=3.469257167 podStartE2EDuration="21.067933859s" podCreationTimestamp="2026-05-25 11:17:39 +0000 UTC" firstStartedPulling="2026-05-25 11:17:41.422273029 +0000 UTC m=+3.103250981" lastFinishedPulling="2026-05-25 11:17:59.020949709 +0000 UTC m=+20.701927673" observedRunningTime="2026-05-25 11:18:00.067620262 +0000 UTC m=+21.748598235" watchObservedRunningTime="2026-05-25 11:18:00.067933859 +0000 UTC m=+21.748911831"
May 25 11:18:00.092620 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:00.092577    2560 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/konnectivity-agent-fbljm" podStartSLOduration=3.487140861 podStartE2EDuration="21.092562811s" podCreationTimestamp="2026-05-25 11:17:39 +0000 UTC" firstStartedPulling="2026-05-25 11:17:41.413865046 +0000 UTC m=+3.094843002" lastFinishedPulling="2026-05-25 11:17:59.019286988 +0000 UTC m=+20.700264952" observedRunningTime="2026-05-25 11:18:00.091281064 +0000 UTC m=+21.772259038" watchObservedRunningTime="2026-05-25 11:18:00.092562811 +0000 UTC m=+21.773540817"
May 25 11:18:00.110246 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:00.110196    2560 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-image-registry/node-ca-8vzrv" podStartSLOduration=8.245275811 podStartE2EDuration="21.110183087s" podCreationTimestamp="2026-05-25 11:17:39 +0000 UTC" firstStartedPulling="2026-05-25 11:17:41.423377706 +0000 UTC m=+3.104355671" lastFinishedPulling="2026-05-25 11:17:54.288284982 +0000 UTC m=+15.969262947" observedRunningTime="2026-05-25 11:18:00.109976032 +0000 UTC m=+21.790954004" watchObservedRunningTime="2026-05-25 11:18:00.110183087 +0000 UTC m=+21.791161060"
May 25 11:18:00.135074 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:00.135035    2560 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/multus-5qqv9" podStartSLOduration=3.519499863 podStartE2EDuration="21.135022903s" podCreationTimestamp="2026-05-25 11:17:39 +0000 UTC" firstStartedPulling="2026-05-25 11:17:41.4210711 +0000 UTC m=+3.102049066" lastFinishedPulling="2026-05-25 11:17:59.036594153 +0000 UTC m=+20.717572106" observedRunningTime="2026-05-25 11:18:00.134761308 +0000 UTC m=+21.815739281" watchObservedRunningTime="2026-05-25 11:18:00.135022903 +0000 UTC m=+21.816000875"
May 25 11:18:00.305121 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:00.305097    2560 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
May 25 11:18:00.884745 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:00.884333    2560 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-05-25T11:18:00.305115262Z","UUID":"d601fc00-7336-4a12-b36c-cfc0d64f9f57","Handler":null,"Name":"","Endpoint":""}
May 25 11:18:00.886375 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:00.886349    2560 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
May 25 11:18:00.886375 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:00.886381    2560 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
May 25 11:18:00.930367 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:00.930341    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nq67w"
May 25 11:18:00.930514 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:00.930374    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zfljp"
May 25 11:18:00.930514 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:18:00.930467    2560 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nq67w" podUID="c0068087-c30b-4ff5-a794-b84738aaa168"
May 25 11:18:00.930634 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:18:00.930547    2560 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zfljp" podUID="0d6828f5-3d2a-417d-ab45-6750dd8cdb92"
May 25 11:18:01.060648 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:01.060613    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lwkh6" event={"ID":"7c5c3c1e-f8a8-437b-972e-1c293222f327","Type":"ContainerStarted","Data":"5bbdf5600d1b924fea20db0201b64e7327fefaad27a611df2f5aed08cdb72794"}
May 25 11:18:01.062324 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:01.062295    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-kj6wk" event={"ID":"0a24f6a0-7ac5-4f1e-84f0-301e16006665","Type":"ContainerStarted","Data":"fd181373679a06d6d1bd49b97fde32912831a1c7243511fca82be08c4e3c340d"}
May 25 11:18:01.077617 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:01.077565    2560 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-kj6wk" podStartSLOduration=4.4804748 podStartE2EDuration="22.077550512s" podCreationTimestamp="2026-05-25 11:17:39 +0000 UTC" firstStartedPulling="2026-05-25 11:17:41.422596976 +0000 UTC m=+3.103574927" lastFinishedPulling="2026-05-25 11:17:59.019672674 +0000 UTC m=+20.700650639" observedRunningTime="2026-05-25 11:18:01.077085982 +0000 UTC m=+22.758063954" watchObservedRunningTime="2026-05-25 11:18:01.077550512 +0000 UTC m=+22.758528485"
May 25 11:18:01.930488 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:01.930362    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hxdhk"
May 25 11:18:01.930488 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:18:01.930476    2560 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hxdhk" podUID="b0dc07c1-2dce-4b08-9fee-41a5937a7bae"
May 25 11:18:02.019444 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:02.019410    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-fbljm"
May 25 11:18:02.067025 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:02.066989    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m55v2_b93f3e9c-f4f9-49dc-bfba-6d0f80c0971d/ovn-acl-logging/0.log"
May 25 11:18:02.067554 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:02.067416    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m55v2" event={"ID":"b93f3e9c-f4f9-49dc-bfba-6d0f80c0971d","Type":"ContainerStarted","Data":"a60a04b4622c40c93156ac91cee77440ca4bb3c45bb704dc761d84992f4852c5"}
May 25 11:18:02.069371 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:02.069347    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lwkh6" event={"ID":"7c5c3c1e-f8a8-437b-972e-1c293222f327","Type":"ContainerStarted","Data":"b67d42b953ca808d3b286d8b56a697d6f25ccee58a5cc738b8ede325353fe770"}
May 25 11:18:02.093758 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:02.093665    2560 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lwkh6" podStartSLOduration=3.3043036089999998 podStartE2EDuration="23.093648117s" podCreationTimestamp="2026-05-25 11:17:39 +0000 UTC" firstStartedPulling="2026-05-25 11:17:41.415348347 +0000 UTC m=+3.096326304" lastFinishedPulling="2026-05-25 11:18:01.20469286 +0000 UTC m=+22.885670812" observedRunningTime="2026-05-25 11:18:02.0933582 +0000 UTC m=+23.774336185" watchObservedRunningTime="2026-05-25 11:18:02.093648117 +0000 UTC m=+23.774626092"
May 25 11:18:02.929969 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:02.929929    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zfljp"
May 25 11:18:02.930140 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:02.929932    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nq67w"
May 25 11:18:02.930140 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:18:02.930049    2560 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zfljp" podUID="0d6828f5-3d2a-417d-ab45-6750dd8cdb92"
May 25 11:18:02.930287 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:18:02.930140    2560 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nq67w" podUID="c0068087-c30b-4ff5-a794-b84738aaa168"
May 25 11:18:03.930248 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:03.930198    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hxdhk"
May 25 11:18:03.930831 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:18:03.930344    2560 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hxdhk" podUID="b0dc07c1-2dce-4b08-9fee-41a5937a7bae"
May 25 11:18:04.586654 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:04.586381    2560 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-fbljm"
May 25 11:18:04.587032 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:04.587016    2560 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-fbljm"
May 25 11:18:04.930054 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:04.929972    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nq67w"
May 25 11:18:04.930247 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:04.929967    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zfljp"
May 25 11:18:04.930247 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:18:04.930096    2560 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nq67w" podUID="c0068087-c30b-4ff5-a794-b84738aaa168"
May 25 11:18:04.930247 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:18:04.930162    2560 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zfljp" podUID="0d6828f5-3d2a-417d-ab45-6750dd8cdb92"
May 25 11:18:05.076225 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:05.076168    2560 generic.go:358] "Generic (PLEG): container finished" podID="372c0836-0d39-4561-b0fa-7171eecad7ca" containerID="a83dd03e10adde90d5346dc63672ba453d3056fa3008d8bcaa4fa87eb36c43ba" exitCode=0
May 25 11:18:05.076363 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:05.076256    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vdfgw" event={"ID":"372c0836-0d39-4561-b0fa-7171eecad7ca","Type":"ContainerDied","Data":"a83dd03e10adde90d5346dc63672ba453d3056fa3008d8bcaa4fa87eb36c43ba"}
May 25 11:18:05.079152 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:05.079131    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m55v2_b93f3e9c-f4f9-49dc-bfba-6d0f80c0971d/ovn-acl-logging/0.log"
May 25 11:18:05.079575 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:05.079549    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m55v2" event={"ID":"b93f3e9c-f4f9-49dc-bfba-6d0f80c0971d","Type":"ContainerStarted","Data":"e90c730728782b924fe2d26f68e9bf7d8d7eacaf2c36a28c25f9356ba3ef94d2"}
May 25 11:18:05.080088 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:05.079894    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-m55v2"
May 25 11:18:05.080088 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:05.079921    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-m55v2"
May 25 11:18:05.080088 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:05.079945    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-m55v2"
May 25 11:18:05.080088 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:05.079978    2560 scope.go:117] "RemoveContainer" containerID="a434fb8506907a44dc4d4dbd46b0dc49c047a87c159802cbbe1e27ddfe9dc835"
May 25 11:18:05.080387 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:05.080244    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-fbljm"
May 25 11:18:05.096085 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:05.096064    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-m55v2"
May 25 11:18:05.096286 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:05.096271    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-m55v2"
May 25 11:18:05.930246 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:05.930157    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hxdhk"
May 25 11:18:05.930370 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:18:05.930271    2560 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hxdhk" podUID="b0dc07c1-2dce-4b08-9fee-41a5937a7bae"
May 25 11:18:06.011793 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:06.011759    2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-nq67w"]
May 25 11:18:06.011968 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:06.011897    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nq67w"
May 25 11:18:06.012038 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:18:06.012009    2560 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nq67w" podUID="c0068087-c30b-4ff5-a794-b84738aaa168"
May 25 11:18:06.015311 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:06.015287    2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-zfljp"]
May 25 11:18:06.015413 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:06.015400    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zfljp"
May 25 11:18:06.015530 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:18:06.015509    2560 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zfljp" podUID="0d6828f5-3d2a-417d-ab45-6750dd8cdb92"
May 25 11:18:06.018024 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:06.018002    2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-hxdhk"]
May 25 11:18:06.083779 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:06.083736    2560 generic.go:358] "Generic (PLEG): container finished" podID="372c0836-0d39-4561-b0fa-7171eecad7ca" containerID="558416da5d6a371e512be8f3088767273bce4c0d38640c76029355326f050e73" exitCode=0
May 25 11:18:06.083931 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:06.083794    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vdfgw" event={"ID":"372c0836-0d39-4561-b0fa-7171eecad7ca","Type":"ContainerDied","Data":"558416da5d6a371e512be8f3088767273bce4c0d38640c76029355326f050e73"}
May 25 11:18:06.087314 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:06.087297    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m55v2_b93f3e9c-f4f9-49dc-bfba-6d0f80c0971d/ovn-acl-logging/0.log"
May 25 11:18:06.087649 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:06.087627    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m55v2" event={"ID":"b93f3e9c-f4f9-49dc-bfba-6d0f80c0971d","Type":"ContainerStarted","Data":"4d51edcbf5fe29d01bbb06b00d56ef769508466c6d5bc134e38b4267d91a528c"}
May 25 11:18:06.087925 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:06.087910    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hxdhk"
May 25 11:18:06.088034 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:18:06.088018    2560 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hxdhk" podUID="b0dc07c1-2dce-4b08-9fee-41a5937a7bae"
May 25 11:18:06.134928 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:06.134878    2560 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-m55v2" podStartSLOduration=9.479276042 podStartE2EDuration="27.13486599s" podCreationTimestamp="2026-05-25 11:17:39 +0000 UTC" firstStartedPulling="2026-05-25 11:17:41.411827448 +0000 UTC m=+3.092805399" lastFinishedPulling="2026-05-25 11:17:59.067417382 +0000 UTC m=+20.748395347" observedRunningTime="2026-05-25 11:18:06.133334099 +0000 UTC m=+27.814312071" watchObservedRunningTime="2026-05-25 11:18:06.13486599 +0000 UTC m=+27.815843960"
May 25 11:18:07.091756 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:07.091576    2560 generic.go:358] "Generic (PLEG): container finished" podID="372c0836-0d39-4561-b0fa-7171eecad7ca" containerID="d86aecfe08d03354838412ba2c304e9b91e922fc4902bd715238ba412be907c0" exitCode=0
May 25 11:18:07.092168 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:07.091657    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vdfgw" event={"ID":"372c0836-0d39-4561-b0fa-7171eecad7ca","Type":"ContainerDied","Data":"d86aecfe08d03354838412ba2c304e9b91e922fc4902bd715238ba412be907c0"}
May 25 11:18:07.930581 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:07.930548    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hxdhk"
May 25 11:18:07.930763 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:07.930547    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nq67w"
May 25 11:18:07.930763 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:18:07.930694    2560 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hxdhk" podUID="b0dc07c1-2dce-4b08-9fee-41a5937a7bae"
May 25 11:18:07.930763 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:07.930555    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zfljp"
May 25 11:18:07.930921 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:18:07.930766    2560 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nq67w" podUID="c0068087-c30b-4ff5-a794-b84738aaa168"
May 25 11:18:07.930921 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:18:07.930812    2560 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zfljp" podUID="0d6828f5-3d2a-417d-ab45-6750dd8cdb92"
May 25 11:18:09.930329 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:09.930291    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nq67w"
May 25 11:18:09.931043 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:09.930298    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hxdhk"
May 25 11:18:09.931043 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:18:09.930424    2560 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nq67w" podUID="c0068087-c30b-4ff5-a794-b84738aaa168"
May 25 11:18:09.931043 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:09.930309    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zfljp"
May 25 11:18:09.931043 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:18:09.930514    2560 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hxdhk" podUID="b0dc07c1-2dce-4b08-9fee-41a5937a7bae"
May 25 11:18:09.931043 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:18:09.930599    2560 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zfljp" podUID="0d6828f5-3d2a-417d-ab45-6750dd8cdb92"
May 25 11:18:11.930270 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:11.930237    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nq67w"
May 25 11:18:11.930869 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:11.930292    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hxdhk"
May 25 11:18:11.930869 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:11.930422    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zfljp"
May 25 11:18:11.930869 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:18:11.930432    2560 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nq67w" podUID="c0068087-c30b-4ff5-a794-b84738aaa168"
May 25 11:18:11.930869 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:18:11.930543    2560 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zfljp" podUID="0d6828f5-3d2a-417d-ab45-6750dd8cdb92"
May 25 11:18:11.930869 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:18:11.930648    2560 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hxdhk" podUID="b0dc07c1-2dce-4b08-9fee-41a5937a7bae"
May 25 11:18:12.113725 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.113696    2560 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-128-146.ec2.internal" event="NodeReady"
May 25 11:18:12.113914 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.113867    2560 kubelet_node_status.go:550] "Fast updating node status as it just became ready"
May 25 11:18:12.153322 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.153275    2560 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5f8d6ccbdc-8pc8t"]
May 25 11:18:12.199828 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.199763    2560 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-65cf6cdd84-r95n8"]
May 25 11:18:12.215005 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.214976    2560 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-78bdfb888c-rqcll"]
May 25 11:18:12.215597 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.215569    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5f8d6ccbdc-8pc8t"
May 25 11:18:12.218414 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.218394    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
May 25 11:18:12.218533 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.218395    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
May 25 11:18:12.218602 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.218588    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-lh8tr\""
May 25 11:18:12.218929 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.218912    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
May 25 11:18:12.225422 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.225405    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
May 25 11:18:12.228922 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.228902    2560 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6c75959455-n8lh8"]
May 25 11:18:12.229030 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.229014    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-65cf6cdd84-r95n8"
May 25 11:18:12.229126 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.229109    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-78bdfb888c-rqcll"
May 25 11:18:12.231692 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.231675    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-69bxb\""
May 25 11:18:12.231801 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.231749    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
May 25 11:18:12.232494 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.232475    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
May 25 11:18:12.232711 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.232696    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
May 25 11:18:12.232805 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.232778    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
May 25 11:18:12.232918 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.232904    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
May 25 11:18:12.233022 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.233007    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
May 25 11:18:12.233090 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.233014    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-bzkz7\""
May 25 11:18:12.233134 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.233091    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
May 25 11:18:12.233134 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.233117    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
May 25 11:18:12.233232 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.233134    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
May 25 11:18:12.234126 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.234106    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
May 25 11:18:12.238626 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.238588    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
May 25 11:18:12.247064 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.247048    2560 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-l2xsm"]
May 25 11:18:12.247202 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.247186    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-n8lh8"
May 25 11:18:12.250381 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.250362    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
May 25 11:18:12.250677 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.250659    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
May 25 11:18:12.252052 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.252028    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
May 25 11:18:12.252146 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.252064    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-vvc8x\""
May 25 11:18:12.252146 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.252065    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
May 25 11:18:12.255892 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.255867    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/01fa335c-4082-46b1-b059-8ae4c68ef3d3-image-registry-private-configuration\") pod \"image-registry-5f8d6ccbdc-8pc8t\" (UID: \"01fa335c-4082-46b1-b059-8ae4c68ef3d3\") " pod="openshift-image-registry/image-registry-5f8d6ccbdc-8pc8t"
May 25 11:18:12.255984 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.255909    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/01fa335c-4082-46b1-b059-8ae4c68ef3d3-registry-tls\") pod \"image-registry-5f8d6ccbdc-8pc8t\" (UID: \"01fa335c-4082-46b1-b059-8ae4c68ef3d3\") " pod="openshift-image-registry/image-registry-5f8d6ccbdc-8pc8t"
May 25 11:18:12.255984 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.255936    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/01fa335c-4082-46b1-b059-8ae4c68ef3d3-trusted-ca\") pod \"image-registry-5f8d6ccbdc-8pc8t\" (UID: \"01fa335c-4082-46b1-b059-8ae4c68ef3d3\") " pod="openshift-image-registry/image-registry-5f8d6ccbdc-8pc8t"
May 25 11:18:12.256105 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.255981    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/01fa335c-4082-46b1-b059-8ae4c68ef3d3-ca-trust-extracted\") pod \"image-registry-5f8d6ccbdc-8pc8t\" (UID: \"01fa335c-4082-46b1-b059-8ae4c68ef3d3\") " pod="openshift-image-registry/image-registry-5f8d6ccbdc-8pc8t"
May 25 11:18:12.256105 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.256021    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/01fa335c-4082-46b1-b059-8ae4c68ef3d3-bound-sa-token\") pod \"image-registry-5f8d6ccbdc-8pc8t\" (UID: \"01fa335c-4082-46b1-b059-8ae4c68ef3d3\") " pod="openshift-image-registry/image-registry-5f8d6ccbdc-8pc8t"
May 25 11:18:12.256105 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.256063    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/01fa335c-4082-46b1-b059-8ae4c68ef3d3-installation-pull-secrets\") pod \"image-registry-5f8d6ccbdc-8pc8t\" (UID: \"01fa335c-4082-46b1-b059-8ae4c68ef3d3\") " pod="openshift-image-registry/image-registry-5f8d6ccbdc-8pc8t"
May 25 11:18:12.256275 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.256109    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/01fa335c-4082-46b1-b059-8ae4c68ef3d3-registry-certificates\") pod \"image-registry-5f8d6ccbdc-8pc8t\" (UID: \"01fa335c-4082-46b1-b059-8ae4c68ef3d3\") " pod="openshift-image-registry/image-registry-5f8d6ccbdc-8pc8t"
May 25 11:18:12.256275 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.256127    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbpn8\" (UniqueName: \"kubernetes.io/projected/01fa335c-4082-46b1-b059-8ae4c68ef3d3-kube-api-access-jbpn8\") pod \"image-registry-5f8d6ccbdc-8pc8t\" (UID: \"01fa335c-4082-46b1-b059-8ae4c68ef3d3\") " pod="openshift-image-registry/image-registry-5f8d6ccbdc-8pc8t"
May 25 11:18:12.267941 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.267919    2560 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-fcqbz"]
May 25 11:18:12.268094 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.268072    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-l2xsm"
May 25 11:18:12.272844 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.272757    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
May 25 11:18:12.273157 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.273138    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-zqsvz\""
May 25 11:18:12.273275 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.273193    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
May 25 11:18:12.288964 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.288941    2560 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-697587599-l976t"]
May 25 11:18:12.289071 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.288990    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-fcqbz"
May 25 11:18:12.293195 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.293164    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
May 25 11:18:12.293368 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.293190    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
May 25 11:18:12.293368 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.293169    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-7c5th\""
May 25 11:18:12.295290 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.295094    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
May 25 11:18:12.295625 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.295599    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
May 25 11:18:12.309061 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.309038    2560 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57b5b68b8-76ldn"]
May 25 11:18:12.309188 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.309153    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-l976t"
May 25 11:18:12.312057 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.312030    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
May 25 11:18:12.313349 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.313327    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-lcnvx\""
May 25 11:18:12.313449 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.313337    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
May 25 11:18:12.314080 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.314060    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
May 25 11:18:12.329058 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.329038    2560 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-76dff48d59-zd54m"]
May 25 11:18:12.329238 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.329200    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-76ldn"
May 25 11:18:12.331786 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.331765    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
May 25 11:18:12.331884 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.331798    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
May 25 11:18:12.332262 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.332098    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-4btzp\""
May 25 11:18:12.332262 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.332107    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
May 25 11:18:12.332262 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.332258    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
May 25 11:18:12.353578 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.353533    2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-65cf6cdd84-r95n8"]
May 25 11:18:12.353578 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.353566    2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-78bdfb888c-rqcll"]
May 25 11:18:12.353578 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.353579    2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-fcqbz"]
May 25 11:18:12.353732 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.353593    2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57b5b68b8-76ldn"]
May 25 11:18:12.353732 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.353606    2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-76dff48d59-zd54m"]
May 25 11:18:12.353732 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.353617    2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-l2xsm"]
May 25 11:18:12.353732 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.353624    2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5f8d6ccbdc-8pc8t"]
May 25 11:18:12.353732 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.353634    2560 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-2mqc7"]
May 25 11:18:12.353732 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.353649    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-76dff48d59-zd54m"
May 25 11:18:12.357510 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.356456    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
May 25 11:18:12.357510 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.356485    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/01fa335c-4082-46b1-b059-8ae4c68ef3d3-image-registry-private-configuration\") pod \"image-registry-5f8d6ccbdc-8pc8t\" (UID: \"01fa335c-4082-46b1-b059-8ae4c68ef3d3\") " pod="openshift-image-registry/image-registry-5f8d6ccbdc-8pc8t"
May 25 11:18:12.357510 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.356532    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/01fa335c-4082-46b1-b059-8ae4c68ef3d3-registry-tls\") pod \"image-registry-5f8d6ccbdc-8pc8t\" (UID: \"01fa335c-4082-46b1-b059-8ae4c68ef3d3\") " pod="openshift-image-registry/image-registry-5f8d6ccbdc-8pc8t"
May 25 11:18:12.357510 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.356558    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/01fa335c-4082-46b1-b059-8ae4c68ef3d3-trusted-ca\") pod \"image-registry-5f8d6ccbdc-8pc8t\" (UID: \"01fa335c-4082-46b1-b059-8ae4c68ef3d3\") " pod="openshift-image-registry/image-registry-5f8d6ccbdc-8pc8t"
May 25 11:18:12.357510 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.356588    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3ca8ede5-d33b-4a16-bd65-73ec9f2a2735-serving-cert\") pod \"insights-operator-78bdfb888c-rqcll\" (UID: \"3ca8ede5-d33b-4a16-bd65-73ec9f2a2735\") " pod="openshift-insights/insights-operator-78bdfb888c-rqcll"
May 25 11:18:12.357510 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.356620    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/01fa335c-4082-46b1-b059-8ae4c68ef3d3-ca-trust-extracted\") pod \"image-registry-5f8d6ccbdc-8pc8t\" (UID: \"01fa335c-4082-46b1-b059-8ae4c68ef3d3\") " pod="openshift-image-registry/image-registry-5f8d6ccbdc-8pc8t"
May 25 11:18:12.357510 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.356644    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/6854cf61-d901-4c43-8718-34de98e26880-stats-auth\") pod \"router-default-65cf6cdd84-r95n8\" (UID: \"6854cf61-d901-4c43-8718-34de98e26880\") " pod="openshift-ingress/router-default-65cf6cdd84-r95n8"
May 25 11:18:12.357510 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.356653    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
May 25 11:18:12.357510 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.356675    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/01fa335c-4082-46b1-b059-8ae4c68ef3d3-bound-sa-token\") pod \"image-registry-5f8d6ccbdc-8pc8t\" (UID: \"01fa335c-4082-46b1-b059-8ae4c68ef3d3\") " pod="openshift-image-registry/image-registry-5f8d6ccbdc-8pc8t"
May 25 11:18:12.357510 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.356700    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7s6wr\" (UniqueName: \"kubernetes.io/projected/251cca37-3cc4-4187-895f-e273d9eac261-kube-api-access-7s6wr\") pod \"cluster-monitoring-operator-6c75959455-n8lh8\" (UID: \"251cca37-3cc4-4187-895f-e273d9eac261\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-n8lh8"
May 25 11:18:12.357510 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.356724    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3ca8ede5-d33b-4a16-bd65-73ec9f2a2735-tmp\") pod \"insights-operator-78bdfb888c-rqcll\" (UID: \"3ca8ede5-d33b-4a16-bd65-73ec9f2a2735\") " pod="openshift-insights/insights-operator-78bdfb888c-rqcll"
May 25 11:18:12.357510 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.356748    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/96664b24-d104-4b57-83e8-63b8c9b3212b-serving-cert\") pod \"kube-storage-version-migrator-operator-565644f49-fcqbz\" (UID: \"96664b24-d104-4b57-83e8-63b8c9b3212b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-fcqbz"
May 25 11:18:12.357510 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.356804    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
May 25 11:18:12.357510 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.357069    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/01fa335c-4082-46b1-b059-8ae4c68ef3d3-ca-trust-extracted\") pod \"image-registry-5f8d6ccbdc-8pc8t\" (UID: \"01fa335c-4082-46b1-b059-8ae4c68ef3d3\") " pod="openshift-image-registry/image-registry-5f8d6ccbdc-8pc8t"
May 25 11:18:12.357510 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.357133    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/3ca8ede5-d33b-4a16-bd65-73ec9f2a2735-snapshots\") pod \"insights-operator-78bdfb888c-rqcll\" (UID: \"3ca8ede5-d33b-4a16-bd65-73ec9f2a2735\") " pod="openshift-insights/insights-operator-78bdfb888c-rqcll"
May 25 11:18:12.357510 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:18:12.357141    2560 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 25 11:18:12.357510 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:18:12.357155    2560 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5f8d6ccbdc-8pc8t: secret "image-registry-tls" not found
May 25 11:18:12.357510 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.357174    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/01fa335c-4082-46b1-b059-8ae4c68ef3d3-installation-pull-secrets\") pod \"image-registry-5f8d6ccbdc-8pc8t\" (UID: \"01fa335c-4082-46b1-b059-8ae4c68ef3d3\") " pod="openshift-image-registry/image-registry-5f8d6ccbdc-8pc8t"
May 25 11:18:12.358342 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:18:12.357225    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/01fa335c-4082-46b1-b059-8ae4c68ef3d3-registry-tls podName:01fa335c-4082-46b1-b059-8ae4c68ef3d3 nodeName:}" failed. No retries permitted until 2026-05-25 11:18:12.857186044 +0000 UTC m=+34.538163997 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/01fa335c-4082-46b1-b059-8ae4c68ef3d3-registry-tls") pod "image-registry-5f8d6ccbdc-8pc8t" (UID: "01fa335c-4082-46b1-b059-8ae4c68ef3d3") : secret "image-registry-tls" not found
May 25 11:18:12.358342 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.357250    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gclc8\" (UniqueName: \"kubernetes.io/projected/6854cf61-d901-4c43-8718-34de98e26880-kube-api-access-gclc8\") pod \"router-default-65cf6cdd84-r95n8\" (UID: \"6854cf61-d901-4c43-8718-34de98e26880\") " pod="openshift-ingress/router-default-65cf6cdd84-r95n8"
May 25 11:18:12.358342 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.357282    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/251cca37-3cc4-4187-895f-e273d9eac261-telemetry-config\") pod \"cluster-monitoring-operator-6c75959455-n8lh8\" (UID: \"251cca37-3cc4-4187-895f-e273d9eac261\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-n8lh8"
May 25 11:18:12.358342 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.357318    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ca8ede5-d33b-4a16-bd65-73ec9f2a2735-trusted-ca-bundle\") pod \"insights-operator-78bdfb888c-rqcll\" (UID: \"3ca8ede5-d33b-4a16-bd65-73ec9f2a2735\") " pod="openshift-insights/insights-operator-78bdfb888c-rqcll"
May 25 11:18:12.358342 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.357348    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6854cf61-d901-4c43-8718-34de98e26880-service-ca-bundle\") pod \"router-default-65cf6cdd84-r95n8\" (UID: \"6854cf61-d901-4c43-8718-34de98e26880\") " pod="openshift-ingress/router-default-65cf6cdd84-r95n8"
May 25 11:18:12.358342 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.357368    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6854cf61-d901-4c43-8718-34de98e26880-metrics-certs\") pod \"router-default-65cf6cdd84-r95n8\" (UID: \"6854cf61-d901-4c43-8718-34de98e26880\") " pod="openshift-ingress/router-default-65cf6cdd84-r95n8"
May 25 11:18:12.358342 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.357393    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/251cca37-3cc4-4187-895f-e273d9eac261-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6c75959455-n8lh8\" (UID: \"251cca37-3cc4-4187-895f-e273d9eac261\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-n8lh8"
May 25 11:18:12.358342 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.357432    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ca8ede5-d33b-4a16-bd65-73ec9f2a2735-service-ca-bundle\") pod \"insights-operator-78bdfb888c-rqcll\" (UID: \"3ca8ede5-d33b-4a16-bd65-73ec9f2a2735\") " pod="openshift-insights/insights-operator-78bdfb888c-rqcll"
May 25 11:18:12.358342 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.357571    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/01fa335c-4082-46b1-b059-8ae4c68ef3d3-registry-certificates\") pod \"image-registry-5f8d6ccbdc-8pc8t\" (UID: \"01fa335c-4082-46b1-b059-8ae4c68ef3d3\") " pod="openshift-image-registry/image-registry-5f8d6ccbdc-8pc8t"
May 25 11:18:12.358342 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.357597    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gx4tq\" (UniqueName: \"kubernetes.io/projected/3ca8ede5-d33b-4a16-bd65-73ec9f2a2735-kube-api-access-gx4tq\") pod \"insights-operator-78bdfb888c-rqcll\" (UID: \"3ca8ede5-d33b-4a16-bd65-73ec9f2a2735\") " pod="openshift-insights/insights-operator-78bdfb888c-rqcll"
May 25 11:18:12.358342 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.357628    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-jbpn8\" (UniqueName: \"kubernetes.io/projected/01fa335c-4082-46b1-b059-8ae4c68ef3d3-kube-api-access-jbpn8\") pod \"image-registry-5f8d6ccbdc-8pc8t\" (UID: \"01fa335c-4082-46b1-b059-8ae4c68ef3d3\") " pod="openshift-image-registry/image-registry-5f8d6ccbdc-8pc8t"
May 25 11:18:12.358342 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.357652    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/96664b24-d104-4b57-83e8-63b8c9b3212b-config\") pod \"kube-storage-version-migrator-operator-565644f49-fcqbz\" (UID: \"96664b24-d104-4b57-83e8-63b8c9b3212b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-fcqbz"
May 25 11:18:12.358342 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.357683    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chh6v\" (UniqueName: \"kubernetes.io/projected/96664b24-d104-4b57-83e8-63b8c9b3212b-kube-api-access-chh6v\") pod \"kube-storage-version-migrator-operator-565644f49-fcqbz\" (UID: \"96664b24-d104-4b57-83e8-63b8c9b3212b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-fcqbz"
May 25 11:18:12.358342 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.357739    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbs9f\" (UniqueName: \"kubernetes.io/projected/f39a5f22-307a-41f0-9ac8-2a37349a345d-kube-api-access-tbs9f\") pod \"volume-data-source-validator-5cf8c89d64-l2xsm\" (UID: \"f39a5f22-307a-41f0-9ac8-2a37349a345d\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-l2xsm"
May 25 11:18:12.358990 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.357773    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/6854cf61-d901-4c43-8718-34de98e26880-default-certificate\") pod \"router-default-65cf6cdd84-r95n8\" (UID: \"6854cf61-d901-4c43-8718-34de98e26880\") " pod="openshift-ingress/router-default-65cf6cdd84-r95n8"
May 25 11:18:12.358990 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.357960    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/01fa335c-4082-46b1-b059-8ae4c68ef3d3-trusted-ca\") pod \"image-registry-5f8d6ccbdc-8pc8t\" (UID: \"01fa335c-4082-46b1-b059-8ae4c68ef3d3\") " pod="openshift-image-registry/image-registry-5f8d6ccbdc-8pc8t"
May 25 11:18:12.358990 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.358319    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/01fa335c-4082-46b1-b059-8ae4c68ef3d3-registry-certificates\") pod \"image-registry-5f8d6ccbdc-8pc8t\" (UID: \"01fa335c-4082-46b1-b059-8ae4c68ef3d3\") " pod="openshift-image-registry/image-registry-5f8d6ccbdc-8pc8t"
May 25 11:18:12.358990 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.358394    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
May 25 11:18:12.358990 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.358457    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-64g6s\""
May 25 11:18:12.361839 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.361818    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/01fa335c-4082-46b1-b059-8ae4c68ef3d3-image-registry-private-configuration\") pod \"image-registry-5f8d6ccbdc-8pc8t\" (UID: \"01fa335c-4082-46b1-b059-8ae4c68ef3d3\") " pod="openshift-image-registry/image-registry-5f8d6ccbdc-8pc8t"
May 25 11:18:12.361983 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.361947    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/01fa335c-4082-46b1-b059-8ae4c68ef3d3-installation-pull-secrets\") pod \"image-registry-5f8d6ccbdc-8pc8t\" (UID: \"01fa335c-4082-46b1-b059-8ae4c68ef3d3\") " pod="openshift-image-registry/image-registry-5f8d6ccbdc-8pc8t"
May 25 11:18:12.367198 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.367143    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/01fa335c-4082-46b1-b059-8ae4c68ef3d3-bound-sa-token\") pod \"image-registry-5f8d6ccbdc-8pc8t\" (UID: \"01fa335c-4082-46b1-b059-8ae4c68ef3d3\") " pod="openshift-image-registry/image-registry-5f8d6ccbdc-8pc8t"
May 25 11:18:12.367519 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.367499    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbpn8\" (UniqueName: \"kubernetes.io/projected/01fa335c-4082-46b1-b059-8ae4c68ef3d3-kube-api-access-jbpn8\") pod \"image-registry-5f8d6ccbdc-8pc8t\" (UID: \"01fa335c-4082-46b1-b059-8ae4c68ef3d3\") " pod="openshift-image-registry/image-registry-5f8d6ccbdc-8pc8t"
May 25 11:18:12.369081 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.369059    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
May 25 11:18:12.377466 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.377446    2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-697587599-l976t"]
May 25 11:18:12.377568 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.377471    2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6c75959455-n8lh8"]
May 25 11:18:12.377568 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.377482    2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-2mqc7"]
May 25 11:18:12.377568 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.377498    2560 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-vvfw5"]
May 25 11:18:12.377710 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.377600    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-2mqc7"
May 25 11:18:12.380199 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.380181    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
May 25 11:18:12.380399 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.380376    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
May 25 11:18:12.380484 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.380470    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
May 25 11:18:12.380547 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.380486    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-kqrg2\""
May 25 11:18:12.397787 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.397765    2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-vvfw5"]
May 25 11:18:12.397900 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.397885    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-vvfw5"
May 25 11:18:12.400698 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.400680    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
May 25 11:18:12.400861 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.400843    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
May 25 11:18:12.400937 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.400700    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
May 25 11:18:12.400993 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.400845    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-9k5hq\""
May 25 11:18:12.401050 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.400681    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
May 25 11:18:12.458362 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.458332    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-tbs9f\" (UniqueName: \"kubernetes.io/projected/f39a5f22-307a-41f0-9ac8-2a37349a345d-kube-api-access-tbs9f\") pod \"volume-data-source-validator-5cf8c89d64-l2xsm\" (UID: \"f39a5f22-307a-41f0-9ac8-2a37349a345d\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-l2xsm"
May 25 11:18:12.458362 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.458361    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/6854cf61-d901-4c43-8718-34de98e26880-default-certificate\") pod \"router-default-65cf6cdd84-r95n8\" (UID: \"6854cf61-d901-4c43-8718-34de98e26880\") " pod="openshift-ingress/router-default-65cf6cdd84-r95n8"
May 25 11:18:12.458593 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.458387    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/96664b24-d104-4b57-83e8-63b8c9b3212b-serving-cert\") pod \"kube-storage-version-migrator-operator-565644f49-fcqbz\" (UID: \"96664b24-d104-4b57-83e8-63b8c9b3212b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-fcqbz"
May 25 11:18:12.458593 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.458408    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3ca8ede5-d33b-4a16-bd65-73ec9f2a2735-serving-cert\") pod \"insights-operator-78bdfb888c-rqcll\" (UID: \"3ca8ede5-d33b-4a16-bd65-73ec9f2a2735\") " pod="openshift-insights/insights-operator-78bdfb888c-rqcll"
May 25 11:18:12.458593 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.458430    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2be32661-9288-4902-9d67-1a5e8f74670d-serving-cert\") pod \"console-operator-76dff48d59-zd54m\" (UID: \"2be32661-9288-4902-9d67-1a5e8f74670d\") " pod="openshift-console-operator/console-operator-76dff48d59-zd54m"
May 25 11:18:12.458593 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.458460    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzx2n\" (UniqueName: \"kubernetes.io/projected/69432372-dd12-48f9-9cf4-8b194109b426-kube-api-access-nzx2n\") pod \"dns-default-vvfw5\" (UID: \"69432372-dd12-48f9-9cf4-8b194109b426\") " pod="openshift-dns/dns-default-vvfw5"
May 25 11:18:12.458593 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.458486    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ed3452ef-2260-4664-92ab-549403628cc8-cert\") pod \"ingress-canary-2mqc7\" (UID: \"ed3452ef-2260-4664-92ab-549403628cc8\") " pod="openshift-ingress-canary/ingress-canary-2mqc7"
May 25 11:18:12.458593 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.458514    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-7s6wr\" (UniqueName: \"kubernetes.io/projected/251cca37-3cc4-4187-895f-e273d9eac261-kube-api-access-7s6wr\") pod \"cluster-monitoring-operator-6c75959455-n8lh8\" (UID: \"251cca37-3cc4-4187-895f-e273d9eac261\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-n8lh8"
May 25 11:18:12.458593 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.458543    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/3ca8ede5-d33b-4a16-bd65-73ec9f2a2735-snapshots\") pod \"insights-operator-78bdfb888c-rqcll\" (UID: \"3ca8ede5-d33b-4a16-bd65-73ec9f2a2735\") " pod="openshift-insights/insights-operator-78bdfb888c-rqcll"
May 25 11:18:12.458593 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.458571    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9e082924-7517-49a1-9595-66c85246eec3-serving-cert\") pod \"service-ca-operator-57b5b68b8-76ldn\" (UID: \"9e082924-7517-49a1-9595-66c85246eec3\") " pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-76ldn"
May 25 11:18:12.458976 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.458598    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/251cca37-3cc4-4187-895f-e273d9eac261-telemetry-config\") pod \"cluster-monitoring-operator-6c75959455-n8lh8\" (UID: \"251cca37-3cc4-4187-895f-e273d9eac261\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-n8lh8"
May 25 11:18:12.458976 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.458706    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69432372-dd12-48f9-9cf4-8b194109b426-tmp-dir\") pod \"dns-default-vvfw5\" (UID: \"69432372-dd12-48f9-9cf4-8b194109b426\") " pod="openshift-dns/dns-default-vvfw5"
May 25 11:18:12.458976 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.458744    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6854cf61-d901-4c43-8718-34de98e26880-metrics-certs\") pod \"router-default-65cf6cdd84-r95n8\" (UID: \"6854cf61-d901-4c43-8718-34de98e26880\") " pod="openshift-ingress/router-default-65cf6cdd84-r95n8"
May 25 11:18:12.458976 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.458770    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/251cca37-3cc4-4187-895f-e273d9eac261-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6c75959455-n8lh8\" (UID: \"251cca37-3cc4-4187-895f-e273d9eac261\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-n8lh8"
May 25 11:18:12.458976 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.458796    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7c5k\" (UniqueName: \"kubernetes.io/projected/f7e56aa6-aa4b-422e-a3a8-a2bbdba923d3-kube-api-access-d7c5k\") pod \"cluster-samples-operator-697587599-l976t\" (UID: \"f7e56aa6-aa4b-422e-a3a8-a2bbdba923d3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-l976t"
May 25 11:18:12.458976 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.458823    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ca8ede5-d33b-4a16-bd65-73ec9f2a2735-service-ca-bundle\") pod \"insights-operator-78bdfb888c-rqcll\" (UID: \"3ca8ede5-d33b-4a16-bd65-73ec9f2a2735\") " pod="openshift-insights/insights-operator-78bdfb888c-rqcll"
May 25 11:18:12.458976 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:18:12.458836    2560 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
May 25 11:18:12.458976 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.458845    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-gx4tq\" (UniqueName: \"kubernetes.io/projected/3ca8ede5-d33b-4a16-bd65-73ec9f2a2735-kube-api-access-gx4tq\") pod \"insights-operator-78bdfb888c-rqcll\" (UID: \"3ca8ede5-d33b-4a16-bd65-73ec9f2a2735\") " pod="openshift-insights/insights-operator-78bdfb888c-rqcll"
May 25 11:18:12.458976 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.458869    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/96664b24-d104-4b57-83e8-63b8c9b3212b-config\") pod \"kube-storage-version-migrator-operator-565644f49-fcqbz\" (UID: \"96664b24-d104-4b57-83e8-63b8c9b3212b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-fcqbz"
May 25 11:18:12.458976 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.458896    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-chh6v\" (UniqueName: \"kubernetes.io/projected/96664b24-d104-4b57-83e8-63b8c9b3212b-kube-api-access-chh6v\") pod \"kube-storage-version-migrator-operator-565644f49-fcqbz\" (UID: \"96664b24-d104-4b57-83e8-63b8c9b3212b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-fcqbz"
May 25 11:18:12.458976 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:18:12.458912    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6854cf61-d901-4c43-8718-34de98e26880-metrics-certs podName:6854cf61-d901-4c43-8718-34de98e26880 nodeName:}" failed. No retries permitted until 2026-05-25 11:18:12.958892251 +0000 UTC m=+34.639870204 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6854cf61-d901-4c43-8718-34de98e26880-metrics-certs") pod "router-default-65cf6cdd84-r95n8" (UID: "6854cf61-d901-4c43-8718-34de98e26880") : secret "router-metrics-certs-default" not found
May 25 11:18:12.458976 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.458961    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f7e56aa6-aa4b-422e-a3a8-a2bbdba923d3-samples-operator-tls\") pod \"cluster-samples-operator-697587599-l976t\" (UID: \"f7e56aa6-aa4b-422e-a3a8-a2bbdba923d3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-l976t"
May 25 11:18:12.459578 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.458994    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2be32661-9288-4902-9d67-1a5e8f74670d-config\") pod \"console-operator-76dff48d59-zd54m\" (UID: \"2be32661-9288-4902-9d67-1a5e8f74670d\") " pod="openshift-console-operator/console-operator-76dff48d59-zd54m"
May 25 11:18:12.459578 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.459025    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v85xv\" (UniqueName: \"kubernetes.io/projected/ed3452ef-2260-4664-92ab-549403628cc8-kube-api-access-v85xv\") pod \"ingress-canary-2mqc7\" (UID: \"ed3452ef-2260-4664-92ab-549403628cc8\") " pod="openshift-ingress-canary/ingress-canary-2mqc7"
May 25 11:18:12.459578 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.459062    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6hmk\" (UniqueName: \"kubernetes.io/projected/9e082924-7517-49a1-9595-66c85246eec3-kube-api-access-h6hmk\") pod \"service-ca-operator-57b5b68b8-76ldn\" (UID: \"9e082924-7517-49a1-9595-66c85246eec3\") " pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-76ldn"
May 25 11:18:12.459578 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.459096    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/69432372-dd12-48f9-9cf4-8b194109b426-metrics-tls\") pod \"dns-default-vvfw5\" (UID: \"69432372-dd12-48f9-9cf4-8b194109b426\") " pod="openshift-dns/dns-default-vvfw5"
May 25 11:18:12.459578 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.459127    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/6854cf61-d901-4c43-8718-34de98e26880-stats-auth\") pod \"router-default-65cf6cdd84-r95n8\" (UID: \"6854cf61-d901-4c43-8718-34de98e26880\") " pod="openshift-ingress/router-default-65cf6cdd84-r95n8"
May 25 11:18:12.459578 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:18:12.459148    2560 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
May 25 11:18:12.459578 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.459152    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3ca8ede5-d33b-4a16-bd65-73ec9f2a2735-tmp\") pod \"insights-operator-78bdfb888c-rqcll\" (UID: \"3ca8ede5-d33b-4a16-bd65-73ec9f2a2735\") " pod="openshift-insights/insights-operator-78bdfb888c-rqcll"
May 25 11:18:12.459578 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.459181    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mr9jk\" (UniqueName: \"kubernetes.io/projected/2be32661-9288-4902-9d67-1a5e8f74670d-kube-api-access-mr9jk\") pod \"console-operator-76dff48d59-zd54m\" (UID: \"2be32661-9288-4902-9d67-1a5e8f74670d\") " pod="openshift-console-operator/console-operator-76dff48d59-zd54m"
May 25 11:18:12.459578 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.459225    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/3ca8ede5-d33b-4a16-bd65-73ec9f2a2735-snapshots\") pod \"insights-operator-78bdfb888c-rqcll\" (UID: \"3ca8ede5-d33b-4a16-bd65-73ec9f2a2735\") " pod="openshift-insights/insights-operator-78bdfb888c-rqcll"
May 25 11:18:12.459578 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.459384    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/251cca37-3cc4-4187-895f-e273d9eac261-telemetry-config\") pod \"cluster-monitoring-operator-6c75959455-n8lh8\" (UID: \"251cca37-3cc4-4187-895f-e273d9eac261\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-n8lh8"
May 25 11:18:12.459578 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:18:12.459417    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/251cca37-3cc4-4187-895f-e273d9eac261-cluster-monitoring-operator-tls podName:251cca37-3cc4-4187-895f-e273d9eac261 nodeName:}" failed. No retries permitted until 2026-05-25 11:18:12.959399483 +0000 UTC m=+34.640377453 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/251cca37-3cc4-4187-895f-e273d9eac261-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6c75959455-n8lh8" (UID: "251cca37-3cc4-4187-895f-e273d9eac261") : secret "cluster-monitoring-operator-tls" not found
May 25 11:18:12.459578 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.459442    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2be32661-9288-4902-9d67-1a5e8f74670d-trusted-ca\") pod \"console-operator-76dff48d59-zd54m\" (UID: \"2be32661-9288-4902-9d67-1a5e8f74670d\") " pod="openshift-console-operator/console-operator-76dff48d59-zd54m"
May 25 11:18:12.459578 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.459479    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/69432372-dd12-48f9-9cf4-8b194109b426-config-volume\") pod \"dns-default-vvfw5\" (UID: \"69432372-dd12-48f9-9cf4-8b194109b426\") " pod="openshift-dns/dns-default-vvfw5"
May 25 11:18:12.459578 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.459516    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-gclc8\" (UniqueName: \"kubernetes.io/projected/6854cf61-d901-4c43-8718-34de98e26880-kube-api-access-gclc8\") pod \"router-default-65cf6cdd84-r95n8\" (UID: \"6854cf61-d901-4c43-8718-34de98e26880\") " pod="openshift-ingress/router-default-65cf6cdd84-r95n8"
May 25 11:18:12.459578 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.459544    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ca8ede5-d33b-4a16-bd65-73ec9f2a2735-trusted-ca-bundle\") pod \"insights-operator-78bdfb888c-rqcll\" (UID: \"3ca8ede5-d33b-4a16-bd65-73ec9f2a2735\") " pod="openshift-insights/insights-operator-78bdfb888c-rqcll"
May 25 11:18:12.459578 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.459577    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6854cf61-d901-4c43-8718-34de98e26880-service-ca-bundle\") pod \"router-default-65cf6cdd84-r95n8\" (UID: \"6854cf61-d901-4c43-8718-34de98e26880\") " pod="openshift-ingress/router-default-65cf6cdd84-r95n8"
May 25 11:18:12.460360 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.459628    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9e082924-7517-49a1-9595-66c85246eec3-config\") pod \"service-ca-operator-57b5b68b8-76ldn\" (UID: \"9e082924-7517-49a1-9595-66c85246eec3\") " pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-76ldn"
May 25 11:18:12.460360 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.459646    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3ca8ede5-d33b-4a16-bd65-73ec9f2a2735-tmp\") pod \"insights-operator-78bdfb888c-rqcll\" (UID: \"3ca8ede5-d33b-4a16-bd65-73ec9f2a2735\") " pod="openshift-insights/insights-operator-78bdfb888c-rqcll"
May 25 11:18:12.460360 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.459701    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ca8ede5-d33b-4a16-bd65-73ec9f2a2735-service-ca-bundle\") pod \"insights-operator-78bdfb888c-rqcll\" (UID: \"3ca8ede5-d33b-4a16-bd65-73ec9f2a2735\") " pod="openshift-insights/insights-operator-78bdfb888c-rqcll"
May 25 11:18:12.460360 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:18:12.459813    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6854cf61-d901-4c43-8718-34de98e26880-service-ca-bundle podName:6854cf61-d901-4c43-8718-34de98e26880 nodeName:}" failed. No retries permitted until 2026-05-25 11:18:12.95979349 +0000 UTC m=+34.640771447 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/6854cf61-d901-4c43-8718-34de98e26880-service-ca-bundle") pod "router-default-65cf6cdd84-r95n8" (UID: "6854cf61-d901-4c43-8718-34de98e26880") : configmap references non-existent config key: service-ca.crt
May 25 11:18:12.460360 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.460319    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/96664b24-d104-4b57-83e8-63b8c9b3212b-config\") pod \"kube-storage-version-migrator-operator-565644f49-fcqbz\" (UID: \"96664b24-d104-4b57-83e8-63b8c9b3212b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-fcqbz"
May 25 11:18:12.460630 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.460567    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ca8ede5-d33b-4a16-bd65-73ec9f2a2735-trusted-ca-bundle\") pod \"insights-operator-78bdfb888c-rqcll\" (UID: \"3ca8ede5-d33b-4a16-bd65-73ec9f2a2735\") " pod="openshift-insights/insights-operator-78bdfb888c-rqcll"
May 25 11:18:12.461250 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.461230    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/96664b24-d104-4b57-83e8-63b8c9b3212b-serving-cert\") pod \"kube-storage-version-migrator-operator-565644f49-fcqbz\" (UID: \"96664b24-d104-4b57-83e8-63b8c9b3212b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-fcqbz"
May 25 11:18:12.461337 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.461257    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3ca8ede5-d33b-4a16-bd65-73ec9f2a2735-serving-cert\") pod \"insights-operator-78bdfb888c-rqcll\" (UID: \"3ca8ede5-d33b-4a16-bd65-73ec9f2a2735\") " pod="openshift-insights/insights-operator-78bdfb888c-rqcll"
May 25 11:18:12.461337 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.461267    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/6854cf61-d901-4c43-8718-34de98e26880-default-certificate\") pod \"router-default-65cf6cdd84-r95n8\" (UID: \"6854cf61-d901-4c43-8718-34de98e26880\") " pod="openshift-ingress/router-default-65cf6cdd84-r95n8"
May 25 11:18:12.461964 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.461948    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/6854cf61-d901-4c43-8718-34de98e26880-stats-auth\") pod \"router-default-65cf6cdd84-r95n8\" (UID: \"6854cf61-d901-4c43-8718-34de98e26880\") " pod="openshift-ingress/router-default-65cf6cdd84-r95n8"
May 25 11:18:12.468354 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.468304    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbs9f\" (UniqueName: \"kubernetes.io/projected/f39a5f22-307a-41f0-9ac8-2a37349a345d-kube-api-access-tbs9f\") pod \"volume-data-source-validator-5cf8c89d64-l2xsm\" (UID: \"f39a5f22-307a-41f0-9ac8-2a37349a345d\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-l2xsm"
May 25 11:18:12.468782 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.468757    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7s6wr\" (UniqueName: \"kubernetes.io/projected/251cca37-3cc4-4187-895f-e273d9eac261-kube-api-access-7s6wr\") pod \"cluster-monitoring-operator-6c75959455-n8lh8\" (UID: \"251cca37-3cc4-4187-895f-e273d9eac261\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-n8lh8"
May 25 11:18:12.468969 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.468943    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gx4tq\" (UniqueName: \"kubernetes.io/projected/3ca8ede5-d33b-4a16-bd65-73ec9f2a2735-kube-api-access-gx4tq\") pod \"insights-operator-78bdfb888c-rqcll\" (UID: \"3ca8ede5-d33b-4a16-bd65-73ec9f2a2735\") " pod="openshift-insights/insights-operator-78bdfb888c-rqcll"
May 25 11:18:12.469174 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.469155    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gclc8\" (UniqueName: \"kubernetes.io/projected/6854cf61-d901-4c43-8718-34de98e26880-kube-api-access-gclc8\") pod \"router-default-65cf6cdd84-r95n8\" (UID: \"6854cf61-d901-4c43-8718-34de98e26880\") " pod="openshift-ingress/router-default-65cf6cdd84-r95n8"
May 25 11:18:12.469594 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.469578    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-chh6v\" (UniqueName: \"kubernetes.io/projected/96664b24-d104-4b57-83e8-63b8c9b3212b-kube-api-access-chh6v\") pod \"kube-storage-version-migrator-operator-565644f49-fcqbz\" (UID: \"96664b24-d104-4b57-83e8-63b8c9b3212b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-fcqbz"
May 25 11:18:12.547042 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.547010    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-78bdfb888c-rqcll"
May 25 11:18:12.560633 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.560474    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0dc07c1-2dce-4b08-9fee-41a5937a7bae-metrics-certs\") pod \"network-metrics-daemon-hxdhk\" (UID: \"b0dc07c1-2dce-4b08-9fee-41a5937a7bae\") " pod="openshift-multus/network-metrics-daemon-hxdhk"
May 25 11:18:12.560752 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.560649    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9e082924-7517-49a1-9595-66c85246eec3-config\") pod \"service-ca-operator-57b5b68b8-76ldn\" (UID: \"9e082924-7517-49a1-9595-66c85246eec3\") " pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-76ldn"
May 25 11:18:12.560752 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.560687    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2be32661-9288-4902-9d67-1a5e8f74670d-serving-cert\") pod \"console-operator-76dff48d59-zd54m\" (UID: \"2be32661-9288-4902-9d67-1a5e8f74670d\") " pod="openshift-console-operator/console-operator-76dff48d59-zd54m"
May 25 11:18:12.560752 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:18:12.560588    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 11:18:12.560752 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.560707    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-nzx2n\" (UniqueName: \"kubernetes.io/projected/69432372-dd12-48f9-9cf4-8b194109b426-kube-api-access-nzx2n\") pod \"dns-default-vvfw5\" (UID: \"69432372-dd12-48f9-9cf4-8b194109b426\") " pod="openshift-dns/dns-default-vvfw5"
May 25 11:18:12.560752 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.560731    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ed3452ef-2260-4664-92ab-549403628cc8-cert\") pod \"ingress-canary-2mqc7\" (UID: \"ed3452ef-2260-4664-92ab-549403628cc8\") " pod="openshift-ingress-canary/ingress-canary-2mqc7"
May 25 11:18:12.561006 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:18:12.560784    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0dc07c1-2dce-4b08-9fee-41a5937a7bae-metrics-certs podName:b0dc07c1-2dce-4b08-9fee-41a5937a7bae nodeName:}" failed. No retries permitted until 2026-05-25 11:18:44.560768126 +0000 UTC m=+66.241746076 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b0dc07c1-2dce-4b08-9fee-41a5937a7bae-metrics-certs") pod "network-metrics-daemon-hxdhk" (UID: "b0dc07c1-2dce-4b08-9fee-41a5937a7bae") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 11:18:12.561006 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:18:12.560817    2560 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 25 11:18:12.561006 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:18:12.560879    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed3452ef-2260-4664-92ab-549403628cc8-cert podName:ed3452ef-2260-4664-92ab-549403628cc8 nodeName:}" failed. No retries permitted until 2026-05-25 11:18:13.060863765 +0000 UTC m=+34.741841717 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/ed3452ef-2260-4664-92ab-549403628cc8-cert") pod "ingress-canary-2mqc7" (UID: "ed3452ef-2260-4664-92ab-549403628cc8") : secret "canary-serving-cert" not found
May 25 11:18:12.561006 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.560927    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9e082924-7517-49a1-9595-66c85246eec3-serving-cert\") pod \"service-ca-operator-57b5b68b8-76ldn\" (UID: \"9e082924-7517-49a1-9595-66c85246eec3\") " pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-76ldn"
May 25 11:18:12.561006 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.560971    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69432372-dd12-48f9-9cf4-8b194109b426-tmp-dir\") pod \"dns-default-vvfw5\" (UID: \"69432372-dd12-48f9-9cf4-8b194109b426\") " pod="openshift-dns/dns-default-vvfw5"
May 25 11:18:12.561278 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.561025    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-d7c5k\" (UniqueName: \"kubernetes.io/projected/f7e56aa6-aa4b-422e-a3a8-a2bbdba923d3-kube-api-access-d7c5k\") pod \"cluster-samples-operator-697587599-l976t\" (UID: \"f7e56aa6-aa4b-422e-a3a8-a2bbdba923d3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-l976t"
May 25 11:18:12.561278 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.561077    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f7e56aa6-aa4b-422e-a3a8-a2bbdba923d3-samples-operator-tls\") pod \"cluster-samples-operator-697587599-l976t\" (UID: \"f7e56aa6-aa4b-422e-a3a8-a2bbdba923d3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-l976t"
May 25 11:18:12.561278 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.561104    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2be32661-9288-4902-9d67-1a5e8f74670d-config\") pod \"console-operator-76dff48d59-zd54m\" (UID: \"2be32661-9288-4902-9d67-1a5e8f74670d\") " pod="openshift-console-operator/console-operator-76dff48d59-zd54m"
May 25 11:18:12.561278 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.561136    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-v85xv\" (UniqueName: \"kubernetes.io/projected/ed3452ef-2260-4664-92ab-549403628cc8-kube-api-access-v85xv\") pod \"ingress-canary-2mqc7\" (UID: \"ed3452ef-2260-4664-92ab-549403628cc8\") " pod="openshift-ingress-canary/ingress-canary-2mqc7"
May 25 11:18:12.561278 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.561169    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-h6hmk\" (UniqueName: \"kubernetes.io/projected/9e082924-7517-49a1-9595-66c85246eec3-kube-api-access-h6hmk\") pod \"service-ca-operator-57b5b68b8-76ldn\" (UID: \"9e082924-7517-49a1-9595-66c85246eec3\") " pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-76ldn"
May 25 11:18:12.561278 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:18:12.561241    2560 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
May 25 11:18:12.561566 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:18:12.561292    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f7e56aa6-aa4b-422e-a3a8-a2bbdba923d3-samples-operator-tls podName:f7e56aa6-aa4b-422e-a3a8-a2bbdba923d3 nodeName:}" failed. No retries permitted until 2026-05-25 11:18:13.061276 +0000 UTC m=+34.742253951 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f7e56aa6-aa4b-422e-a3a8-a2bbdba923d3-samples-operator-tls") pod "cluster-samples-operator-697587599-l976t" (UID: "f7e56aa6-aa4b-422e-a3a8-a2bbdba923d3") : secret "samples-operator-tls" not found
May 25 11:18:12.561566 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.561319    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9e082924-7517-49a1-9595-66c85246eec3-config\") pod \"service-ca-operator-57b5b68b8-76ldn\" (UID: \"9e082924-7517-49a1-9595-66c85246eec3\") " pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-76ldn"
May 25 11:18:12.561566 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.561354    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69432372-dd12-48f9-9cf4-8b194109b426-tmp-dir\") pod \"dns-default-vvfw5\" (UID: \"69432372-dd12-48f9-9cf4-8b194109b426\") " pod="openshift-dns/dns-default-vvfw5"
May 25 11:18:12.561566 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.561386    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/69432372-dd12-48f9-9cf4-8b194109b426-metrics-tls\") pod \"dns-default-vvfw5\" (UID: \"69432372-dd12-48f9-9cf4-8b194109b426\") " pod="openshift-dns/dns-default-vvfw5"
May 25 11:18:12.561566 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.561433    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-mr9jk\" (UniqueName: \"kubernetes.io/projected/2be32661-9288-4902-9d67-1a5e8f74670d-kube-api-access-mr9jk\") pod \"console-operator-76dff48d59-zd54m\" (UID: \"2be32661-9288-4902-9d67-1a5e8f74670d\") " pod="openshift-console-operator/console-operator-76dff48d59-zd54m"
May 25 11:18:12.561566 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.561467    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2be32661-9288-4902-9d67-1a5e8f74670d-trusted-ca\") pod \"console-operator-76dff48d59-zd54m\" (UID: \"2be32661-9288-4902-9d67-1a5e8f74670d\") " pod="openshift-console-operator/console-operator-76dff48d59-zd54m"
May 25 11:18:12.561566 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.561502    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/69432372-dd12-48f9-9cf4-8b194109b426-config-volume\") pod \"dns-default-vvfw5\" (UID: \"69432372-dd12-48f9-9cf4-8b194109b426\") " pod="openshift-dns/dns-default-vvfw5"
May 25 11:18:12.561927 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:18:12.561661    2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 25 11:18:12.561927 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:18:12.561703    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/69432372-dd12-48f9-9cf4-8b194109b426-metrics-tls podName:69432372-dd12-48f9-9cf4-8b194109b426 nodeName:}" failed. No retries permitted until 2026-05-25 11:18:13.061689869 +0000 UTC m=+34.742667821 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/69432372-dd12-48f9-9cf4-8b194109b426-metrics-tls") pod "dns-default-vvfw5" (UID: "69432372-dd12-48f9-9cf4-8b194109b426") : secret "dns-default-metrics-tls" not found
May 25 11:18:12.562036 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.562009    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/69432372-dd12-48f9-9cf4-8b194109b426-config-volume\") pod \"dns-default-vvfw5\" (UID: \"69432372-dd12-48f9-9cf4-8b194109b426\") " pod="openshift-dns/dns-default-vvfw5"
May 25 11:18:12.562526 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.562504    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2be32661-9288-4902-9d67-1a5e8f74670d-config\") pod \"console-operator-76dff48d59-zd54m\" (UID: \"2be32661-9288-4902-9d67-1a5e8f74670d\") " pod="openshift-console-operator/console-operator-76dff48d59-zd54m"
May 25 11:18:12.562597 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.562577    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2be32661-9288-4902-9d67-1a5e8f74670d-trusted-ca\") pod \"console-operator-76dff48d59-zd54m\" (UID: \"2be32661-9288-4902-9d67-1a5e8f74670d\") " pod="openshift-console-operator/console-operator-76dff48d59-zd54m"
May 25 11:18:12.563317 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.563297    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2be32661-9288-4902-9d67-1a5e8f74670d-serving-cert\") pod \"console-operator-76dff48d59-zd54m\" (UID: \"2be32661-9288-4902-9d67-1a5e8f74670d\") " pod="openshift-console-operator/console-operator-76dff48d59-zd54m"
May 25 11:18:12.563414 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.563393    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9e082924-7517-49a1-9595-66c85246eec3-serving-cert\") pod \"service-ca-operator-57b5b68b8-76ldn\" (UID: \"9e082924-7517-49a1-9595-66c85246eec3\") " pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-76ldn"
May 25 11:18:12.573784 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.573756    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6hmk\" (UniqueName: \"kubernetes.io/projected/9e082924-7517-49a1-9595-66c85246eec3-kube-api-access-h6hmk\") pod \"service-ca-operator-57b5b68b8-76ldn\" (UID: \"9e082924-7517-49a1-9595-66c85246eec3\") " pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-76ldn"
May 25 11:18:12.575148 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.575129    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v85xv\" (UniqueName: \"kubernetes.io/projected/ed3452ef-2260-4664-92ab-549403628cc8-kube-api-access-v85xv\") pod \"ingress-canary-2mqc7\" (UID: \"ed3452ef-2260-4664-92ab-549403628cc8\") " pod="openshift-ingress-canary/ingress-canary-2mqc7"
May 25 11:18:12.575502 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.575481    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzx2n\" (UniqueName: \"kubernetes.io/projected/69432372-dd12-48f9-9cf4-8b194109b426-kube-api-access-nzx2n\") pod \"dns-default-vvfw5\" (UID: \"69432372-dd12-48f9-9cf4-8b194109b426\") " pod="openshift-dns/dns-default-vvfw5"
May 25 11:18:12.575581 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.575493    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7c5k\" (UniqueName: \"kubernetes.io/projected/f7e56aa6-aa4b-422e-a3a8-a2bbdba923d3-kube-api-access-d7c5k\") pod \"cluster-samples-operator-697587599-l976t\" (UID: \"f7e56aa6-aa4b-422e-a3a8-a2bbdba923d3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-l976t"
May 25 11:18:12.575906 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.575888    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mr9jk\" (UniqueName: \"kubernetes.io/projected/2be32661-9288-4902-9d67-1a5e8f74670d-kube-api-access-mr9jk\") pod \"console-operator-76dff48d59-zd54m\" (UID: \"2be32661-9288-4902-9d67-1a5e8f74670d\") " pod="openshift-console-operator/console-operator-76dff48d59-zd54m"
May 25 11:18:12.578672 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.578653    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-l2xsm"
May 25 11:18:12.598669 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.598642    2560 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-kbgpq"]
May 25 11:18:12.598803 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.598714    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-fcqbz"
May 25 11:18:12.617601 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.617430    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-kbgpq"
May 25 11:18:12.622778 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.622122    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-j6lnl\""
May 25 11:18:12.639713 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.639358    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-76ldn"
May 25 11:18:12.663690 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.663051    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c0068087-c30b-4ff5-a794-b84738aaa168-original-pull-secret\") pod \"global-pull-secret-syncer-nq67w\" (UID: \"c0068087-c30b-4ff5-a794-b84738aaa168\") " pod="kube-system/global-pull-secret-syncer-nq67w"
May 25 11:18:12.663690 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:18:12.663239    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
May 25 11:18:12.663690 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:18:12.663313    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0068087-c30b-4ff5-a794-b84738aaa168-original-pull-secret podName:c0068087-c30b-4ff5-a794-b84738aaa168 nodeName:}" failed. No retries permitted until 2026-05-25 11:18:44.663294993 +0000 UTC m=+66.344272946 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c0068087-c30b-4ff5-a794-b84738aaa168-original-pull-secret") pod "global-pull-secret-syncer-nq67w" (UID: "c0068087-c30b-4ff5-a794-b84738aaa168") : object "kube-system"/"original-pull-secret" not registered
May 25 11:18:12.675421 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.675388    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-76dff48d59-zd54m"
May 25 11:18:12.741845 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.741818    2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-78bdfb888c-rqcll"]
May 25 11:18:12.747033 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:18:12.747008    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ca8ede5_d33b_4a16_bd65_73ec9f2a2735.slice/crio-a527b9890c24a015c18ab05589992c91b10344716417df5a0296e30f36b88918 WatchSource:0}: Error finding container a527b9890c24a015c18ab05589992c91b10344716417df5a0296e30f36b88918: Status 404 returned error can't find the container with id a527b9890c24a015c18ab05589992c91b10344716417df5a0296e30f36b88918
May 25 11:18:12.758934 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.758912    2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-l2xsm"]
May 25 11:18:12.761369 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:18:12.761344    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf39a5f22_307a_41f0_9ac8_2a37349a345d.slice/crio-d0abfc69fea41d65da43802119d0736b5c37790f9578e4112d4bc594e9cc8877 WatchSource:0}: Error finding container d0abfc69fea41d65da43802119d0736b5c37790f9578e4112d4bc594e9cc8877: Status 404 returned error can't find the container with id d0abfc69fea41d65da43802119d0736b5c37790f9578e4112d4bc594e9cc8877
May 25 11:18:12.764093 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.764069    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-97fvl\" (UniqueName: \"kubernetes.io/projected/0d6828f5-3d2a-417d-ab45-6750dd8cdb92-kube-api-access-97fvl\") pod \"network-check-target-zfljp\" (UID: \"0d6828f5-3d2a-417d-ab45-6750dd8cdb92\") " pod="openshift-network-diagnostics/network-check-target-zfljp"
May 25 11:18:12.764165 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.764129    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8nrj\" (UniqueName: \"kubernetes.io/projected/dc7c628a-0fd2-456b-a47f-b7741b83d46e-kube-api-access-j8nrj\") pod \"node-resolver-kbgpq\" (UID: \"dc7c628a-0fd2-456b-a47f-b7741b83d46e\") " pod="openshift-dns/node-resolver-kbgpq"
May 25 11:18:12.764306 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:18:12.764252    2560 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 25 11:18:12.764306 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:18:12.764273    2560 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 25 11:18:12.764306 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:18:12.764284    2560 projected.go:194] Error preparing data for projected volume kube-api-access-97fvl for pod openshift-network-diagnostics/network-check-target-zfljp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 11:18:12.764442 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.764309    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc7c628a-0fd2-456b-a47f-b7741b83d46e-tmp-dir\") pod \"node-resolver-kbgpq\" (UID: \"dc7c628a-0fd2-456b-a47f-b7741b83d46e\") " pod="openshift-dns/node-resolver-kbgpq"
May 25 11:18:12.764442 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:18:12.764335    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0d6828f5-3d2a-417d-ab45-6750dd8cdb92-kube-api-access-97fvl podName:0d6828f5-3d2a-417d-ab45-6750dd8cdb92 nodeName:}" failed. No retries permitted until 2026-05-25 11:18:44.764317031 +0000 UTC m=+66.445294997 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-97fvl" (UniqueName: "kubernetes.io/projected/0d6828f5-3d2a-417d-ab45-6750dd8cdb92-kube-api-access-97fvl") pod "network-check-target-zfljp" (UID: "0d6828f5-3d2a-417d-ab45-6750dd8cdb92") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 11:18:12.764442 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.764423    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/dc7c628a-0fd2-456b-a47f-b7741b83d46e-hosts-file\") pod \"node-resolver-kbgpq\" (UID: \"dc7c628a-0fd2-456b-a47f-b7741b83d46e\") " pod="openshift-dns/node-resolver-kbgpq"
May 25 11:18:12.774610 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.774587    2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-fcqbz"]
May 25 11:18:12.777824 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:18:12.777794    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96664b24_d104_4b57_83e8_63b8c9b3212b.slice/crio-620ad3d1daf1833d08d8e984a93c42fbdb2178f9306b9d738dfb84e788258aa9 WatchSource:0}: Error finding container 620ad3d1daf1833d08d8e984a93c42fbdb2178f9306b9d738dfb84e788258aa9: Status 404 returned error can't find the container with id 620ad3d1daf1833d08d8e984a93c42fbdb2178f9306b9d738dfb84e788258aa9
May 25 11:18:12.865556 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.865513    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/dc7c628a-0fd2-456b-a47f-b7741b83d46e-hosts-file\") pod \"node-resolver-kbgpq\" (UID: \"dc7c628a-0fd2-456b-a47f-b7741b83d46e\") " pod="openshift-dns/node-resolver-kbgpq"
May 25 11:18:12.865690 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.865623    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/dc7c628a-0fd2-456b-a47f-b7741b83d46e-hosts-file\") pod \"node-resolver-kbgpq\" (UID: \"dc7c628a-0fd2-456b-a47f-b7741b83d46e\") " pod="openshift-dns/node-resolver-kbgpq"
May 25 11:18:12.865690 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.865654    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-j8nrj\" (UniqueName: \"kubernetes.io/projected/dc7c628a-0fd2-456b-a47f-b7741b83d46e-kube-api-access-j8nrj\") pod \"node-resolver-kbgpq\" (UID: \"dc7c628a-0fd2-456b-a47f-b7741b83d46e\") " pod="openshift-dns/node-resolver-kbgpq"
May 25 11:18:12.865802 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.865702    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/01fa335c-4082-46b1-b059-8ae4c68ef3d3-registry-tls\") pod \"image-registry-5f8d6ccbdc-8pc8t\" (UID: \"01fa335c-4082-46b1-b059-8ae4c68ef3d3\") " pod="openshift-image-registry/image-registry-5f8d6ccbdc-8pc8t"
May 25 11:18:12.865802 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.865783    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc7c628a-0fd2-456b-a47f-b7741b83d46e-tmp-dir\") pod \"node-resolver-kbgpq\" (UID: \"dc7c628a-0fd2-456b-a47f-b7741b83d46e\") " pod="openshift-dns/node-resolver-kbgpq"
May 25 11:18:12.866081 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:18:12.866058    2560 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 25 11:18:12.866081 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:18:12.866078    2560 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5f8d6ccbdc-8pc8t: secret "image-registry-tls" not found
May 25 11:18:12.866236 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:18:12.866139    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/01fa335c-4082-46b1-b059-8ae4c68ef3d3-registry-tls podName:01fa335c-4082-46b1-b059-8ae4c68ef3d3 nodeName:}" failed. No retries permitted until 2026-05-25 11:18:13.866120687 +0000 UTC m=+35.547098642 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/01fa335c-4082-46b1-b059-8ae4c68ef3d3-registry-tls") pod "image-registry-5f8d6ccbdc-8pc8t" (UID: "01fa335c-4082-46b1-b059-8ae4c68ef3d3") : secret "image-registry-tls" not found
May 25 11:18:12.866236 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.866197    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc7c628a-0fd2-456b-a47f-b7741b83d46e-tmp-dir\") pod \"node-resolver-kbgpq\" (UID: \"dc7c628a-0fd2-456b-a47f-b7741b83d46e\") " pod="openshift-dns/node-resolver-kbgpq"
May 25 11:18:12.878393 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.878377    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8nrj\" (UniqueName: \"kubernetes.io/projected/dc7c628a-0fd2-456b-a47f-b7741b83d46e-kube-api-access-j8nrj\") pod \"node-resolver-kbgpq\" (UID: \"dc7c628a-0fd2-456b-a47f-b7741b83d46e\") " pod="openshift-dns/node-resolver-kbgpq"
May 25 11:18:12.902852 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.902826    2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-76dff48d59-zd54m"]
May 25 11:18:12.905903 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:18:12.905876    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2be32661_9288_4902_9d67_1a5e8f74670d.slice/crio-1b8bb6e7e35a63255afd24bf9bbac18d3bb76ff5fbee9a144b8bb5a53cffa7c3 WatchSource:0}: Error finding container 1b8bb6e7e35a63255afd24bf9bbac18d3bb76ff5fbee9a144b8bb5a53cffa7c3: Status 404 returned error can't find the container with id 1b8bb6e7e35a63255afd24bf9bbac18d3bb76ff5fbee9a144b8bb5a53cffa7c3
May 25 11:18:12.906701 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.906683    2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57b5b68b8-76ldn"]
May 25 11:18:12.908863 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:18:12.908843    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e082924_7517_49a1_9595_66c85246eec3.slice/crio-65b7495ed8a4f540ecc892ac438930d4b955c1d096402428a54badc344802c9b WatchSource:0}: Error finding container 65b7495ed8a4f540ecc892ac438930d4b955c1d096402428a54badc344802c9b: Status 404 returned error can't find the container with id 65b7495ed8a4f540ecc892ac438930d4b955c1d096402428a54badc344802c9b
May 25 11:18:12.930449 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.930422    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-kbgpq"
May 25 11:18:12.942851 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:18:12.942826    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc7c628a_0fd2_456b_a47f_b7741b83d46e.slice/crio-3660822a6b5d9a3256e245b88aa505e5563f6a451cd4b478a5490c07274ca02d WatchSource:0}: Error finding container 3660822a6b5d9a3256e245b88aa505e5563f6a451cd4b478a5490c07274ca02d: Status 404 returned error can't find the container with id 3660822a6b5d9a3256e245b88aa505e5563f6a451cd4b478a5490c07274ca02d
May 25 11:18:12.966896 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.966665    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6854cf61-d901-4c43-8718-34de98e26880-metrics-certs\") pod \"router-default-65cf6cdd84-r95n8\" (UID: \"6854cf61-d901-4c43-8718-34de98e26880\") " pod="openshift-ingress/router-default-65cf6cdd84-r95n8"
May 25 11:18:12.967052 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.966932    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/251cca37-3cc4-4187-895f-e273d9eac261-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6c75959455-n8lh8\" (UID: \"251cca37-3cc4-4187-895f-e273d9eac261\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-n8lh8"
May 25 11:18:12.967052 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:18:12.966848    2560 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
May 25 11:18:12.967156 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:18:12.967067    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6854cf61-d901-4c43-8718-34de98e26880-metrics-certs podName:6854cf61-d901-4c43-8718-34de98e26880 nodeName:}" failed. No retries permitted until 2026-05-25 11:18:13.967043343 +0000 UTC m=+35.648021319 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6854cf61-d901-4c43-8718-34de98e26880-metrics-certs") pod "router-default-65cf6cdd84-r95n8" (UID: "6854cf61-d901-4c43-8718-34de98e26880") : secret "router-metrics-certs-default" not found
May 25 11:18:12.967156 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:12.967106    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6854cf61-d901-4c43-8718-34de98e26880-service-ca-bundle\") pod \"router-default-65cf6cdd84-r95n8\" (UID: \"6854cf61-d901-4c43-8718-34de98e26880\") " pod="openshift-ingress/router-default-65cf6cdd84-r95n8"
May 25 11:18:12.967156 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:18:12.967122    2560 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
May 25 11:18:12.967328 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:18:12.967188    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/251cca37-3cc4-4187-895f-e273d9eac261-cluster-monitoring-operator-tls podName:251cca37-3cc4-4187-895f-e273d9eac261 nodeName:}" failed. No retries permitted until 2026-05-25 11:18:13.967172298 +0000 UTC m=+35.648150249 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/251cca37-3cc4-4187-895f-e273d9eac261-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6c75959455-n8lh8" (UID: "251cca37-3cc4-4187-895f-e273d9eac261") : secret "cluster-monitoring-operator-tls" not found
May 25 11:18:12.967328 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:18:12.967237    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6854cf61-d901-4c43-8718-34de98e26880-service-ca-bundle podName:6854cf61-d901-4c43-8718-34de98e26880 nodeName:}" failed. No retries permitted until 2026-05-25 11:18:13.967203116 +0000 UTC m=+35.648181070 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/6854cf61-d901-4c43-8718-34de98e26880-service-ca-bundle") pod "router-default-65cf6cdd84-r95n8" (UID: "6854cf61-d901-4c43-8718-34de98e26880") : configmap references non-existent config key: service-ca.crt
May 25 11:18:13.068355 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:13.068330    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ed3452ef-2260-4664-92ab-549403628cc8-cert\") pod \"ingress-canary-2mqc7\" (UID: \"ed3452ef-2260-4664-92ab-549403628cc8\") " pod="openshift-ingress-canary/ingress-canary-2mqc7"
May 25 11:18:13.068465 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:13.068406    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f7e56aa6-aa4b-422e-a3a8-a2bbdba923d3-samples-operator-tls\") pod \"cluster-samples-operator-697587599-l976t\" (UID: \"f7e56aa6-aa4b-422e-a3a8-a2bbdba923d3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-l976t"
May 25 11:18:13.068465 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:13.068459    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/69432372-dd12-48f9-9cf4-8b194109b426-metrics-tls\") pod \"dns-default-vvfw5\" (UID: \"69432372-dd12-48f9-9cf4-8b194109b426\") " pod="openshift-dns/dns-default-vvfw5"
May 25 11:18:13.068539 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:18:13.068481    2560 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 25 11:18:13.068585 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:18:13.068540    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed3452ef-2260-4664-92ab-549403628cc8-cert podName:ed3452ef-2260-4664-92ab-549403628cc8 nodeName:}" failed. No retries permitted until 2026-05-25 11:18:14.068521614 +0000 UTC m=+35.749499588 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/ed3452ef-2260-4664-92ab-549403628cc8-cert") pod "ingress-canary-2mqc7" (UID: "ed3452ef-2260-4664-92ab-549403628cc8") : secret "canary-serving-cert" not found
May 25 11:18:13.068585 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:18:13.068550    2560 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
May 25 11:18:13.068585 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:18:13.068573    2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 25 11:18:13.068705 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:18:13.068618    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f7e56aa6-aa4b-422e-a3a8-a2bbdba923d3-samples-operator-tls podName:f7e56aa6-aa4b-422e-a3a8-a2bbdba923d3 nodeName:}" failed. No retries permitted until 2026-05-25 11:18:14.068597194 +0000 UTC m=+35.749575157 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f7e56aa6-aa4b-422e-a3a8-a2bbdba923d3-samples-operator-tls") pod "cluster-samples-operator-697587599-l976t" (UID: "f7e56aa6-aa4b-422e-a3a8-a2bbdba923d3") : secret "samples-operator-tls" not found
May 25 11:18:13.068705 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:18:13.068635    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/69432372-dd12-48f9-9cf4-8b194109b426-metrics-tls podName:69432372-dd12-48f9-9cf4-8b194109b426 nodeName:}" failed. No retries permitted until 2026-05-25 11:18:14.06862769 +0000 UTC m=+35.749605641 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/69432372-dd12-48f9-9cf4-8b194109b426-metrics-tls") pod "dns-default-vvfw5" (UID: "69432372-dd12-48f9-9cf4-8b194109b426") : secret "dns-default-metrics-tls" not found
May 25 11:18:13.104873 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:13.104842    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-l2xsm" event={"ID":"f39a5f22-307a-41f0-9ac8-2a37349a345d","Type":"ContainerStarted","Data":"d0abfc69fea41d65da43802119d0736b5c37790f9578e4112d4bc594e9cc8877"}
May 25 11:18:13.106120 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:13.106098    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-kbgpq" event={"ID":"dc7c628a-0fd2-456b-a47f-b7741b83d46e","Type":"ContainerStarted","Data":"e37a48ff5428ded894ff6cbab0a027a857df6784c756ecbb18b27bb0b5a2ff3c"}
May 25 11:18:13.106246 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:13.106125    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-kbgpq" event={"ID":"dc7c628a-0fd2-456b-a47f-b7741b83d46e","Type":"ContainerStarted","Data":"3660822a6b5d9a3256e245b88aa505e5563f6a451cd4b478a5490c07274ca02d"}
May 25 11:18:13.107017 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:13.106987    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-fcqbz" event={"ID":"96664b24-d104-4b57-83e8-63b8c9b3212b","Type":"ContainerStarted","Data":"620ad3d1daf1833d08d8e984a93c42fbdb2178f9306b9d738dfb84e788258aa9"}
May 25 11:18:13.109234 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:13.109191    2560 generic.go:358] "Generic (PLEG): container finished" podID="372c0836-0d39-4561-b0fa-7171eecad7ca" containerID="9986d75b651ed31488011bd8c951042c20ba0300a84609834982b31a62e41f9d" exitCode=0
May 25 11:18:13.109325 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:13.109272    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vdfgw" event={"ID":"372c0836-0d39-4561-b0fa-7171eecad7ca","Type":"ContainerDied","Data":"9986d75b651ed31488011bd8c951042c20ba0300a84609834982b31a62e41f9d"}
May 25 11:18:13.110258 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:13.110235    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-78bdfb888c-rqcll" event={"ID":"3ca8ede5-d33b-4a16-bd65-73ec9f2a2735","Type":"ContainerStarted","Data":"a527b9890c24a015c18ab05589992c91b10344716417df5a0296e30f36b88918"}
May 25 11:18:13.111253 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:13.111233    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-76ldn" event={"ID":"9e082924-7517-49a1-9595-66c85246eec3","Type":"ContainerStarted","Data":"65b7495ed8a4f540ecc892ac438930d4b955c1d096402428a54badc344802c9b"}
May 25 11:18:13.112154 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:13.112106    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-76dff48d59-zd54m" event={"ID":"2be32661-9288-4902-9d67-1a5e8f74670d","Type":"ContainerStarted","Data":"1b8bb6e7e35a63255afd24bf9bbac18d3bb76ff5fbee9a144b8bb5a53cffa7c3"}
May 25 11:18:13.122816 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:13.122785    2560 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-dns/node-resolver-kbgpq" podStartSLOduration=1.1227716189999999 podStartE2EDuration="1.122771619s" podCreationTimestamp="2026-05-25 11:18:12 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 11:18:13.121622531 +0000 UTC m=+34.802600507" watchObservedRunningTime="2026-05-25 11:18:13.122771619 +0000 UTC m=+34.803749593"
May 25 11:18:13.878468 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:13.878428    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/01fa335c-4082-46b1-b059-8ae4c68ef3d3-registry-tls\") pod \"image-registry-5f8d6ccbdc-8pc8t\" (UID: \"01fa335c-4082-46b1-b059-8ae4c68ef3d3\") " pod="openshift-image-registry/image-registry-5f8d6ccbdc-8pc8t"
May 25 11:18:13.878765 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:18:13.878749    2560 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 25 11:18:13.878846 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:18:13.878769    2560 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5f8d6ccbdc-8pc8t: secret "image-registry-tls" not found
May 25 11:18:13.878846 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:18:13.878831    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/01fa335c-4082-46b1-b059-8ae4c68ef3d3-registry-tls podName:01fa335c-4082-46b1-b059-8ae4c68ef3d3 nodeName:}" failed. No retries permitted until 2026-05-25 11:18:15.878811167 +0000 UTC m=+37.559789120 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/01fa335c-4082-46b1-b059-8ae4c68ef3d3-registry-tls") pod "image-registry-5f8d6ccbdc-8pc8t" (UID: "01fa335c-4082-46b1-b059-8ae4c68ef3d3") : secret "image-registry-tls" not found
May 25 11:18:13.930843 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:13.930805    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nq67w"
May 25 11:18:13.930843 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:13.930819    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zfljp"
May 25 11:18:13.931560 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:13.930819    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hxdhk"
May 25 11:18:13.933896 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:13.933875    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-mcnj6\""
May 25 11:18:13.935166 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:13.935147    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-lkpxl\""
May 25 11:18:13.935472 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:13.935455    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
May 25 11:18:13.935542 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:13.935492    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
May 25 11:18:13.935738 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:13.935722    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
May 25 11:18:13.935821 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:13.935806    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
May 25 11:18:13.979073 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:13.979038    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6854cf61-d901-4c43-8718-34de98e26880-service-ca-bundle\") pod \"router-default-65cf6cdd84-r95n8\" (UID: \"6854cf61-d901-4c43-8718-34de98e26880\") " pod="openshift-ingress/router-default-65cf6cdd84-r95n8"
May 25 11:18:13.979262 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:13.979156    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6854cf61-d901-4c43-8718-34de98e26880-metrics-certs\") pod \"router-default-65cf6cdd84-r95n8\" (UID: \"6854cf61-d901-4c43-8718-34de98e26880\") " pod="openshift-ingress/router-default-65cf6cdd84-r95n8"
May 25 11:18:13.979262 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:13.979186    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/251cca37-3cc4-4187-895f-e273d9eac261-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6c75959455-n8lh8\" (UID: \"251cca37-3cc4-4187-895f-e273d9eac261\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-n8lh8"
May 25 11:18:13.979716 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:18:13.979409    2560 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
May 25 11:18:13.979716 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:18:13.979476    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/251cca37-3cc4-4187-895f-e273d9eac261-cluster-monitoring-operator-tls podName:251cca37-3cc4-4187-895f-e273d9eac261 nodeName:}" failed. No retries permitted until 2026-05-25 11:18:15.97945694 +0000 UTC m=+37.660434893 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/251cca37-3cc4-4187-895f-e273d9eac261-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6c75959455-n8lh8" (UID: "251cca37-3cc4-4187-895f-e273d9eac261") : secret "cluster-monitoring-operator-tls" not found
May 25 11:18:13.979716 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:18:13.979546    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6854cf61-d901-4c43-8718-34de98e26880-service-ca-bundle podName:6854cf61-d901-4c43-8718-34de98e26880 nodeName:}" failed. No retries permitted until 2026-05-25 11:18:15.979525438 +0000 UTC m=+37.660503404 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/6854cf61-d901-4c43-8718-34de98e26880-service-ca-bundle") pod "router-default-65cf6cdd84-r95n8" (UID: "6854cf61-d901-4c43-8718-34de98e26880") : configmap references non-existent config key: service-ca.crt
May 25 11:18:13.979716 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:18:13.979631    2560 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
May 25 11:18:13.979716 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:18:13.979669    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6854cf61-d901-4c43-8718-34de98e26880-metrics-certs podName:6854cf61-d901-4c43-8718-34de98e26880 nodeName:}" failed. No retries permitted until 2026-05-25 11:18:15.979657591 +0000 UTC m=+37.660635544 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6854cf61-d901-4c43-8718-34de98e26880-metrics-certs") pod "router-default-65cf6cdd84-r95n8" (UID: "6854cf61-d901-4c43-8718-34de98e26880") : secret "router-metrics-certs-default" not found
May 25 11:18:14.080302 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:14.080261    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f7e56aa6-aa4b-422e-a3a8-a2bbdba923d3-samples-operator-tls\") pod \"cluster-samples-operator-697587599-l976t\" (UID: \"f7e56aa6-aa4b-422e-a3a8-a2bbdba923d3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-l976t"
May 25 11:18:14.080494 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:14.080340    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/69432372-dd12-48f9-9cf4-8b194109b426-metrics-tls\") pod \"dns-default-vvfw5\" (UID: \"69432372-dd12-48f9-9cf4-8b194109b426\") " pod="openshift-dns/dns-default-vvfw5"
May 25 11:18:14.080494 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:14.080455    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ed3452ef-2260-4664-92ab-549403628cc8-cert\") pod \"ingress-canary-2mqc7\" (UID: \"ed3452ef-2260-4664-92ab-549403628cc8\") " pod="openshift-ingress-canary/ingress-canary-2mqc7"
May 25 11:18:14.080603 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:18:14.080591    2560 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 25 11:18:14.080670 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:18:14.080656    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed3452ef-2260-4664-92ab-549403628cc8-cert podName:ed3452ef-2260-4664-92ab-549403628cc8 nodeName:}" failed. No retries permitted until 2026-05-25 11:18:16.080635977 +0000 UTC m=+37.761613952 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/ed3452ef-2260-4664-92ab-549403628cc8-cert") pod "ingress-canary-2mqc7" (UID: "ed3452ef-2260-4664-92ab-549403628cc8") : secret "canary-serving-cert" not found
May 25 11:18:14.080753 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:18:14.080737    2560 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
May 25 11:18:14.080807 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:18:14.080778    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f7e56aa6-aa4b-422e-a3a8-a2bbdba923d3-samples-operator-tls podName:f7e56aa6-aa4b-422e-a3a8-a2bbdba923d3 nodeName:}" failed. No retries permitted until 2026-05-25 11:18:16.080765086 +0000 UTC m=+37.761743040 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f7e56aa6-aa4b-422e-a3a8-a2bbdba923d3-samples-operator-tls") pod "cluster-samples-operator-697587599-l976t" (UID: "f7e56aa6-aa4b-422e-a3a8-a2bbdba923d3") : secret "samples-operator-tls" not found
May 25 11:18:14.080863 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:18:14.080834    2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 25 11:18:14.080917 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:18:14.080867    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/69432372-dd12-48f9-9cf4-8b194109b426-metrics-tls podName:69432372-dd12-48f9-9cf4-8b194109b426 nodeName:}" failed. No retries permitted until 2026-05-25 11:18:16.080856445 +0000 UTC m=+37.761834401 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/69432372-dd12-48f9-9cf4-8b194109b426-metrics-tls") pod "dns-default-vvfw5" (UID: "69432372-dd12-48f9-9cf4-8b194109b426") : secret "dns-default-metrics-tls" not found
May 25 11:18:14.119751 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:14.119380    2560 generic.go:358] "Generic (PLEG): container finished" podID="372c0836-0d39-4561-b0fa-7171eecad7ca" containerID="70582db018b6701e7c39d2a055ea58414bd3764bf06fdf32e30242911789537f" exitCode=0
May 25 11:18:14.119751 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:14.119430    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vdfgw" event={"ID":"372c0836-0d39-4561-b0fa-7171eecad7ca","Type":"ContainerDied","Data":"70582db018b6701e7c39d2a055ea58414bd3764bf06fdf32e30242911789537f"}
May 25 11:18:15.897233 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:15.897186    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/01fa335c-4082-46b1-b059-8ae4c68ef3d3-registry-tls\") pod \"image-registry-5f8d6ccbdc-8pc8t\" (UID: \"01fa335c-4082-46b1-b059-8ae4c68ef3d3\") " pod="openshift-image-registry/image-registry-5f8d6ccbdc-8pc8t"
May 25 11:18:15.897662 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:18:15.897324    2560 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 25 11:18:15.897662 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:18:15.897345    2560 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5f8d6ccbdc-8pc8t: secret "image-registry-tls" not found
May 25 11:18:15.897662 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:18:15.897398    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/01fa335c-4082-46b1-b059-8ae4c68ef3d3-registry-tls podName:01fa335c-4082-46b1-b059-8ae4c68ef3d3 nodeName:}" failed. No retries permitted until 2026-05-25 11:18:19.89738374 +0000 UTC m=+41.578361690 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/01fa335c-4082-46b1-b059-8ae4c68ef3d3-registry-tls") pod "image-registry-5f8d6ccbdc-8pc8t" (UID: "01fa335c-4082-46b1-b059-8ae4c68ef3d3") : secret "image-registry-tls" not found
May 25 11:18:15.998576 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:15.998533    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6854cf61-d901-4c43-8718-34de98e26880-service-ca-bundle\") pod \"router-default-65cf6cdd84-r95n8\" (UID: \"6854cf61-d901-4c43-8718-34de98e26880\") " pod="openshift-ingress/router-default-65cf6cdd84-r95n8"
May 25 11:18:15.998754 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:15.998649    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6854cf61-d901-4c43-8718-34de98e26880-metrics-certs\") pod \"router-default-65cf6cdd84-r95n8\" (UID: \"6854cf61-d901-4c43-8718-34de98e26880\") " pod="openshift-ingress/router-default-65cf6cdd84-r95n8"
May 25 11:18:15.998754 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:18:15.998671    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6854cf61-d901-4c43-8718-34de98e26880-service-ca-bundle podName:6854cf61-d901-4c43-8718-34de98e26880 nodeName:}" failed. No retries permitted until 2026-05-25 11:18:19.998648608 +0000 UTC m=+41.679626559 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/6854cf61-d901-4c43-8718-34de98e26880-service-ca-bundle") pod "router-default-65cf6cdd84-r95n8" (UID: "6854cf61-d901-4c43-8718-34de98e26880") : configmap references non-existent config key: service-ca.crt
May 25 11:18:15.998754 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:15.998719    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/251cca37-3cc4-4187-895f-e273d9eac261-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6c75959455-n8lh8\" (UID: \"251cca37-3cc4-4187-895f-e273d9eac261\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-n8lh8"
May 25 11:18:15.998754 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:18:15.998739    2560 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
May 25 11:18:15.998963 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:18:15.998788    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6854cf61-d901-4c43-8718-34de98e26880-metrics-certs podName:6854cf61-d901-4c43-8718-34de98e26880 nodeName:}" failed. No retries permitted until 2026-05-25 11:18:19.998773633 +0000 UTC m=+41.679751586 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6854cf61-d901-4c43-8718-34de98e26880-metrics-certs") pod "router-default-65cf6cdd84-r95n8" (UID: "6854cf61-d901-4c43-8718-34de98e26880") : secret "router-metrics-certs-default" not found
May 25 11:18:15.998963 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:18:15.998857    2560 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
May 25 11:18:15.998963 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:18:15.998903    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/251cca37-3cc4-4187-895f-e273d9eac261-cluster-monitoring-operator-tls podName:251cca37-3cc4-4187-895f-e273d9eac261 nodeName:}" failed. No retries permitted until 2026-05-25 11:18:19.998889886 +0000 UTC m=+41.679867857 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/251cca37-3cc4-4187-895f-e273d9eac261-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6c75959455-n8lh8" (UID: "251cca37-3cc4-4187-895f-e273d9eac261") : secret "cluster-monitoring-operator-tls" not found
May 25 11:18:16.100313 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:16.100270    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/69432372-dd12-48f9-9cf4-8b194109b426-metrics-tls\") pod \"dns-default-vvfw5\" (UID: \"69432372-dd12-48f9-9cf4-8b194109b426\") " pod="openshift-dns/dns-default-vvfw5"
May 25 11:18:16.100500 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:16.100400    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ed3452ef-2260-4664-92ab-549403628cc8-cert\") pod \"ingress-canary-2mqc7\" (UID: \"ed3452ef-2260-4664-92ab-549403628cc8\") " pod="openshift-ingress-canary/ingress-canary-2mqc7"
May 25 11:18:16.100500 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:18:16.100428    2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 25 11:18:16.100500 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:16.100475    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f7e56aa6-aa4b-422e-a3a8-a2bbdba923d3-samples-operator-tls\") pod \"cluster-samples-operator-697587599-l976t\" (UID: \"f7e56aa6-aa4b-422e-a3a8-a2bbdba923d3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-l976t"
May 25 11:18:16.100500 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:18:16.100494    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/69432372-dd12-48f9-9cf4-8b194109b426-metrics-tls podName:69432372-dd12-48f9-9cf4-8b194109b426 nodeName:}" failed. No retries permitted until 2026-05-25 11:18:20.100475334 +0000 UTC m=+41.781453305 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/69432372-dd12-48f9-9cf4-8b194109b426-metrics-tls") pod "dns-default-vvfw5" (UID: "69432372-dd12-48f9-9cf4-8b194109b426") : secret "dns-default-metrics-tls" not found
May 25 11:18:16.100913 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:18:16.100577    2560 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 25 11:18:16.100913 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:18:16.100878    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed3452ef-2260-4664-92ab-549403628cc8-cert podName:ed3452ef-2260-4664-92ab-549403628cc8 nodeName:}" failed. No retries permitted until 2026-05-25 11:18:20.10085627 +0000 UTC m=+41.781834241 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/ed3452ef-2260-4664-92ab-549403628cc8-cert") pod "ingress-canary-2mqc7" (UID: "ed3452ef-2260-4664-92ab-549403628cc8") : secret "canary-serving-cert" not found
May 25 11:18:16.100913 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:18:16.100895    2560 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
May 25 11:18:16.101093 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:18:16.100968    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f7e56aa6-aa4b-422e-a3a8-a2bbdba923d3-samples-operator-tls podName:f7e56aa6-aa4b-422e-a3a8-a2bbdba923d3 nodeName:}" failed. No retries permitted until 2026-05-25 11:18:20.10093979 +0000 UTC m=+41.781917757 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f7e56aa6-aa4b-422e-a3a8-a2bbdba923d3-samples-operator-tls") pod "cluster-samples-operator-697587599-l976t" (UID: "f7e56aa6-aa4b-422e-a3a8-a2bbdba923d3") : secret "samples-operator-tls" not found
May 25 11:18:18.129812 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:18.129593    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-78bdfb888c-rqcll" event={"ID":"3ca8ede5-d33b-4a16-bd65-73ec9f2a2735","Type":"ContainerStarted","Data":"36e1f07b89fca4dfd34ec092594ed7be7bbc1ed67be2134cfd9b8ad97e7ee2d2"}
May 25 11:18:18.131063 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:18.131035    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-76ldn" event={"ID":"9e082924-7517-49a1-9595-66c85246eec3","Type":"ContainerStarted","Data":"b24a628eb96f5714b757690e23b7e57b5932dbad59c552772d756a4e2a61f34b"}
May 25 11:18:18.132567 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:18.132546    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-zd54m_2be32661-9288-4902-9d67-1a5e8f74670d/console-operator/0.log"
May 25 11:18:18.132666 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:18.132587    2560 generic.go:358] "Generic (PLEG): container finished" podID="2be32661-9288-4902-9d67-1a5e8f74670d" containerID="9e5c9d3c29964b04a32741228e318fa5389b93b621195d06b43c5a8a653daa6c" exitCode=255
May 25 11:18:18.132735 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:18.132672    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-76dff48d59-zd54m" event={"ID":"2be32661-9288-4902-9d67-1a5e8f74670d","Type":"ContainerDied","Data":"9e5c9d3c29964b04a32741228e318fa5389b93b621195d06b43c5a8a653daa6c"}
May 25 11:18:18.132860 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:18.132844    2560 scope.go:117] "RemoveContainer" containerID="9e5c9d3c29964b04a32741228e318fa5389b93b621195d06b43c5a8a653daa6c"
May 25 11:18:18.133960 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:18.133940    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-l2xsm" event={"ID":"f39a5f22-307a-41f0-9ac8-2a37349a345d","Type":"ContainerStarted","Data":"73f8b8c5fdf022f86817ea9f635c3ff9de78d2f098e9b47bd5e0e5bcb9038761"}
May 25 11:18:18.135695 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:18.135674    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-fcqbz" event={"ID":"96664b24-d104-4b57-83e8-63b8c9b3212b","Type":"ContainerStarted","Data":"ea9059a63d70d855295b0a5f9735df789e0aeabb6c967b94bdde4baa4768ea22"}
May 25 11:18:18.138912 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:18.138892    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vdfgw" event={"ID":"372c0836-0d39-4561-b0fa-7171eecad7ca","Type":"ContainerStarted","Data":"abe56d76855acf026f16ed02248b2a78d820b21a623bd90d8183c65ae8622af9"}
May 25 11:18:18.194881 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:18.194823    2560 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-insights/insights-operator-78bdfb888c-rqcll" podStartSLOduration=13.628444034 podStartE2EDuration="18.194801126s" podCreationTimestamp="2026-05-25 11:18:00 +0000 UTC" firstStartedPulling="2026-05-25 11:18:12.762543423 +0000 UTC m=+34.443521388" lastFinishedPulling="2026-05-25 11:18:17.328900528 +0000 UTC m=+39.009878480" observedRunningTime="2026-05-25 11:18:18.167531551 +0000 UTC m=+39.848509525" watchObservedRunningTime="2026-05-25 11:18:18.194801126 +0000 UTC m=+39.875779101"
May 25 11:18:18.282163 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:18.282100    2560 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-l2xsm" podStartSLOduration=4.717969862 podStartE2EDuration="9.282081699s" podCreationTimestamp="2026-05-25 11:18:09 +0000 UTC" firstStartedPulling="2026-05-25 11:18:12.764859613 +0000 UTC m=+34.445837565" lastFinishedPulling="2026-05-25 11:18:17.328971445 +0000 UTC m=+39.009949402" observedRunningTime="2026-05-25 11:18:18.19630228 +0000 UTC m=+39.877280266" watchObservedRunningTime="2026-05-25 11:18:18.282081699 +0000 UTC m=+39.963059674"
May 25 11:18:18.413024 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:18.412919    2560 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-76ldn" podStartSLOduration=3.9899732390000002 podStartE2EDuration="8.412900052s" podCreationTimestamp="2026-05-25 11:18:10 +0000 UTC" firstStartedPulling="2026-05-25 11:18:12.91068444 +0000 UTC m=+34.591662391" lastFinishedPulling="2026-05-25 11:18:17.333611239 +0000 UTC m=+39.014589204" observedRunningTime="2026-05-25 11:18:18.282532659 +0000 UTC m=+39.963510632" watchObservedRunningTime="2026-05-25 11:18:18.412900052 +0000 UTC m=+40.093878024"
May 25 11:18:18.413236 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:18.413194    2560 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-vdfgw" podStartSLOduration=8.038093788 podStartE2EDuration="39.413187939s" podCreationTimestamp="2026-05-25 11:17:39 +0000 UTC" firstStartedPulling="2026-05-25 11:17:41.415916962 +0000 UTC m=+3.096894913" lastFinishedPulling="2026-05-25 11:18:12.791011099 +0000 UTC m=+34.471989064" observedRunningTime="2026-05-25 11:18:18.403731372 +0000 UTC m=+40.084709345" watchObservedRunningTime="2026-05-25 11:18:18.413187939 +0000 UTC m=+40.094165913"
May 25 11:18:18.455928 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:18.455874    2560 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-fcqbz" podStartSLOduration=3.90696558 podStartE2EDuration="8.45585733s" podCreationTimestamp="2026-05-25 11:18:10 +0000 UTC" firstStartedPulling="2026-05-25 11:18:12.780022408 +0000 UTC m=+34.461000358" lastFinishedPulling="2026-05-25 11:18:17.328914144 +0000 UTC m=+39.009892108" observedRunningTime="2026-05-25 11:18:18.455798625 +0000 UTC m=+40.136776600" watchObservedRunningTime="2026-05-25 11:18:18.45585733 +0000 UTC m=+40.136835299"
May 25 11:18:19.143056 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:19.143026    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-zd54m_2be32661-9288-4902-9d67-1a5e8f74670d/console-operator/1.log"
May 25 11:18:19.143453 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:19.143374    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-zd54m_2be32661-9288-4902-9d67-1a5e8f74670d/console-operator/0.log"
May 25 11:18:19.143453 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:19.143407    2560 generic.go:358] "Generic (PLEG): container finished" podID="2be32661-9288-4902-9d67-1a5e8f74670d" containerID="8c3250eec3ef03820ae865392ae5d81191348c8c4a74f189b8314099901ddd8d" exitCode=255
May 25 11:18:19.143566 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:19.143534    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-76dff48d59-zd54m" event={"ID":"2be32661-9288-4902-9d67-1a5e8f74670d","Type":"ContainerDied","Data":"8c3250eec3ef03820ae865392ae5d81191348c8c4a74f189b8314099901ddd8d"}
May 25 11:18:19.143622 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:19.143595    2560 scope.go:117] "RemoveContainer" containerID="9e5c9d3c29964b04a32741228e318fa5389b93b621195d06b43c5a8a653daa6c"
May 25 11:18:19.143761 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:19.143745    2560 scope.go:117] "RemoveContainer" containerID="8c3250eec3ef03820ae865392ae5d81191348c8c4a74f189b8314099901ddd8d"
May 25 11:18:19.143979 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:18:19.143926    2560 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-76dff48d59-zd54m_openshift-console-operator(2be32661-9288-4902-9d67-1a5e8f74670d)\"" pod="openshift-console-operator/console-operator-76dff48d59-zd54m" podUID="2be32661-9288-4902-9d67-1a5e8f74670d"
May 25 11:18:19.935121 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:19.935089    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/01fa335c-4082-46b1-b059-8ae4c68ef3d3-registry-tls\") pod \"image-registry-5f8d6ccbdc-8pc8t\" (UID: \"01fa335c-4082-46b1-b059-8ae4c68ef3d3\") " pod="openshift-image-registry/image-registry-5f8d6ccbdc-8pc8t"
May 25 11:18:19.935354 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:18:19.935246    2560 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 25 11:18:19.935354 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:18:19.935263    2560 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5f8d6ccbdc-8pc8t: secret "image-registry-tls" not found
May 25 11:18:19.935354 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:18:19.935314    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/01fa335c-4082-46b1-b059-8ae4c68ef3d3-registry-tls podName:01fa335c-4082-46b1-b059-8ae4c68ef3d3 nodeName:}" failed. No retries permitted until 2026-05-25 11:18:27.935299728 +0000 UTC m=+49.616277679 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/01fa335c-4082-46b1-b059-8ae4c68ef3d3-registry-tls") pod "image-registry-5f8d6ccbdc-8pc8t" (UID: "01fa335c-4082-46b1-b059-8ae4c68ef3d3") : secret "image-registry-tls" not found
May 25 11:18:20.035854 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:20.035821    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6854cf61-d901-4c43-8718-34de98e26880-service-ca-bundle\") pod \"router-default-65cf6cdd84-r95n8\" (UID: \"6854cf61-d901-4c43-8718-34de98e26880\") " pod="openshift-ingress/router-default-65cf6cdd84-r95n8"
May 25 11:18:20.036029 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:20.035906    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6854cf61-d901-4c43-8718-34de98e26880-metrics-certs\") pod \"router-default-65cf6cdd84-r95n8\" (UID: \"6854cf61-d901-4c43-8718-34de98e26880\") " pod="openshift-ingress/router-default-65cf6cdd84-r95n8"
May 25 11:18:20.036029 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:20.035927    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/251cca37-3cc4-4187-895f-e273d9eac261-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6c75959455-n8lh8\" (UID: \"251cca37-3cc4-4187-895f-e273d9eac261\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-n8lh8"
May 25 11:18:20.036029 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:18:20.035974    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6854cf61-d901-4c43-8718-34de98e26880-service-ca-bundle podName:6854cf61-d901-4c43-8718-34de98e26880 nodeName:}" failed. No retries permitted until 2026-05-25 11:18:28.03595641 +0000 UTC m=+49.716934361 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/6854cf61-d901-4c43-8718-34de98e26880-service-ca-bundle") pod "router-default-65cf6cdd84-r95n8" (UID: "6854cf61-d901-4c43-8718-34de98e26880") : configmap references non-existent config key: service-ca.crt
May 25 11:18:20.036029 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:18:20.036025    2560 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
May 25 11:18:20.036286 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:18:20.036033    2560 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
May 25 11:18:20.036286 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:18:20.036097    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6854cf61-d901-4c43-8718-34de98e26880-metrics-certs podName:6854cf61-d901-4c43-8718-34de98e26880 nodeName:}" failed. No retries permitted until 2026-05-25 11:18:28.036083155 +0000 UTC m=+49.717061106 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6854cf61-d901-4c43-8718-34de98e26880-metrics-certs") pod "router-default-65cf6cdd84-r95n8" (UID: "6854cf61-d901-4c43-8718-34de98e26880") : secret "router-metrics-certs-default" not found
May 25 11:18:20.036286 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:18:20.036112    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/251cca37-3cc4-4187-895f-e273d9eac261-cluster-monitoring-operator-tls podName:251cca37-3cc4-4187-895f-e273d9eac261 nodeName:}" failed. No retries permitted until 2026-05-25 11:18:28.036104774 +0000 UTC m=+49.717082725 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/251cca37-3cc4-4187-895f-e273d9eac261-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6c75959455-n8lh8" (UID: "251cca37-3cc4-4187-895f-e273d9eac261") : secret "cluster-monitoring-operator-tls" not found
May 25 11:18:20.136520 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:20.136485    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ed3452ef-2260-4664-92ab-549403628cc8-cert\") pod \"ingress-canary-2mqc7\" (UID: \"ed3452ef-2260-4664-92ab-549403628cc8\") " pod="openshift-ingress-canary/ingress-canary-2mqc7"
May 25 11:18:20.136694 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:20.136572    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f7e56aa6-aa4b-422e-a3a8-a2bbdba923d3-samples-operator-tls\") pod \"cluster-samples-operator-697587599-l976t\" (UID: \"f7e56aa6-aa4b-422e-a3a8-a2bbdba923d3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-l976t"
May 25 11:18:20.136694 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:20.136621    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/69432372-dd12-48f9-9cf4-8b194109b426-metrics-tls\") pod \"dns-default-vvfw5\" (UID: \"69432372-dd12-48f9-9cf4-8b194109b426\") " pod="openshift-dns/dns-default-vvfw5"
May 25 11:18:20.136694 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:18:20.136633    2560 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 25 11:18:20.136849 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:18:20.136696    2560 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
May 25 11:18:20.136849 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:18:20.136697    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed3452ef-2260-4664-92ab-549403628cc8-cert podName:ed3452ef-2260-4664-92ab-549403628cc8 nodeName:}" failed. No retries permitted until 2026-05-25 11:18:28.136679024 +0000 UTC m=+49.817656995 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/ed3452ef-2260-4664-92ab-549403628cc8-cert") pod "ingress-canary-2mqc7" (UID: "ed3452ef-2260-4664-92ab-549403628cc8") : secret "canary-serving-cert" not found
May 25 11:18:20.136849 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:18:20.136741    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f7e56aa6-aa4b-422e-a3a8-a2bbdba923d3-samples-operator-tls podName:f7e56aa6-aa4b-422e-a3a8-a2bbdba923d3 nodeName:}" failed. No retries permitted until 2026-05-25 11:18:28.136730191 +0000 UTC m=+49.817708149 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f7e56aa6-aa4b-422e-a3a8-a2bbdba923d3-samples-operator-tls") pod "cluster-samples-operator-697587599-l976t" (UID: "f7e56aa6-aa4b-422e-a3a8-a2bbdba923d3") : secret "samples-operator-tls" not found
May 25 11:18:20.136849 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:18:20.136763    2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 25 11:18:20.136849 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:18:20.136822    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/69432372-dd12-48f9-9cf4-8b194109b426-metrics-tls podName:69432372-dd12-48f9-9cf4-8b194109b426 nodeName:}" failed. No retries permitted until 2026-05-25 11:18:28.13680376 +0000 UTC m=+49.817781726 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/69432372-dd12-48f9-9cf4-8b194109b426-metrics-tls") pod "dns-default-vvfw5" (UID: "69432372-dd12-48f9-9cf4-8b194109b426") : secret "dns-default-metrics-tls" not found
May 25 11:18:20.151065 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:20.151041    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-zd54m_2be32661-9288-4902-9d67-1a5e8f74670d/console-operator/1.log"
May 25 11:18:20.151400 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:20.151373    2560 scope.go:117] "RemoveContainer" containerID="8c3250eec3ef03820ae865392ae5d81191348c8c4a74f189b8314099901ddd8d"
May 25 11:18:20.151557 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:18:20.151539    2560 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-76dff48d59-zd54m_openshift-console-operator(2be32661-9288-4902-9d67-1a5e8f74670d)\"" pod="openshift-console-operator/console-operator-76dff48d59-zd54m" podUID="2be32661-9288-4902-9d67-1a5e8f74670d"
May 25 11:18:20.879093 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:20.879069    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-kbgpq_dc7c628a-0fd2-456b-a47f-b7741b83d46e/dns-node-resolver/0.log"
May 25 11:18:21.287840 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:21.287771    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-8vzrv_2d4d0de1-9c41-46ef-89e3-9a0615a665af/node-ca/0.log"
May 25 11:18:22.675963 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:22.675928    2560 kubelet.go:2658] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-76dff48d59-zd54m"
May 25 11:18:22.675963 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:22.675966    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-76dff48d59-zd54m"
May 25 11:18:22.676363 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:22.676308    2560 scope.go:117] "RemoveContainer" containerID="8c3250eec3ef03820ae865392ae5d81191348c8c4a74f189b8314099901ddd8d"
May 25 11:18:22.676478 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:18:22.676462    2560 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-76dff48d59-zd54m_openshift-console-operator(2be32661-9288-4902-9d67-1a5e8f74670d)\"" pod="openshift-console-operator/console-operator-76dff48d59-zd54m" podUID="2be32661-9288-4902-9d67-1a5e8f74670d"
May 25 11:18:28.002568 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:28.002530    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/01fa335c-4082-46b1-b059-8ae4c68ef3d3-registry-tls\") pod \"image-registry-5f8d6ccbdc-8pc8t\" (UID: \"01fa335c-4082-46b1-b059-8ae4c68ef3d3\") " pod="openshift-image-registry/image-registry-5f8d6ccbdc-8pc8t"
May 25 11:18:28.002937 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:18:28.002652    2560 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 25 11:18:28.002937 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:18:28.002664    2560 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5f8d6ccbdc-8pc8t: secret "image-registry-tls" not found
May 25 11:18:28.002937 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:18:28.002714    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/01fa335c-4082-46b1-b059-8ae4c68ef3d3-registry-tls podName:01fa335c-4082-46b1-b059-8ae4c68ef3d3 nodeName:}" failed. No retries permitted until 2026-05-25 11:18:44.002699366 +0000 UTC m=+65.683677317 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/01fa335c-4082-46b1-b059-8ae4c68ef3d3-registry-tls") pod "image-registry-5f8d6ccbdc-8pc8t" (UID: "01fa335c-4082-46b1-b059-8ae4c68ef3d3") : secret "image-registry-tls" not found
May 25 11:18:28.103869 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:28.103834    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6854cf61-d901-4c43-8718-34de98e26880-service-ca-bundle\") pod \"router-default-65cf6cdd84-r95n8\" (UID: \"6854cf61-d901-4c43-8718-34de98e26880\") " pod="openshift-ingress/router-default-65cf6cdd84-r95n8"
May 25 11:18:28.104021 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:28.103912    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6854cf61-d901-4c43-8718-34de98e26880-metrics-certs\") pod \"router-default-65cf6cdd84-r95n8\" (UID: \"6854cf61-d901-4c43-8718-34de98e26880\") " pod="openshift-ingress/router-default-65cf6cdd84-r95n8"
May 25 11:18:28.104021 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:28.103932    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/251cca37-3cc4-4187-895f-e273d9eac261-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6c75959455-n8lh8\" (UID: \"251cca37-3cc4-4187-895f-e273d9eac261\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-n8lh8"
May 25 11:18:28.104021 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:18:28.104008    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6854cf61-d901-4c43-8718-34de98e26880-service-ca-bundle podName:6854cf61-d901-4c43-8718-34de98e26880 nodeName:}" failed. No retries permitted until 2026-05-25 11:18:44.103984729 +0000 UTC m=+65.784962685 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/6854cf61-d901-4c43-8718-34de98e26880-service-ca-bundle") pod "router-default-65cf6cdd84-r95n8" (UID: "6854cf61-d901-4c43-8718-34de98e26880") : configmap references non-existent config key: service-ca.crt
May 25 11:18:28.104174 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:18:28.104062    2560 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
May 25 11:18:28.104174 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:18:28.104068    2560 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
May 25 11:18:28.104174 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:18:28.104106    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/251cca37-3cc4-4187-895f-e273d9eac261-cluster-monitoring-operator-tls podName:251cca37-3cc4-4187-895f-e273d9eac261 nodeName:}" failed. No retries permitted until 2026-05-25 11:18:44.104095129 +0000 UTC m=+65.785073085 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/251cca37-3cc4-4187-895f-e273d9eac261-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6c75959455-n8lh8" (UID: "251cca37-3cc4-4187-895f-e273d9eac261") : secret "cluster-monitoring-operator-tls" not found
May 25 11:18:28.104174 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:18:28.104145    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6854cf61-d901-4c43-8718-34de98e26880-metrics-certs podName:6854cf61-d901-4c43-8718-34de98e26880 nodeName:}" failed. No retries permitted until 2026-05-25 11:18:44.104131906 +0000 UTC m=+65.785109868 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6854cf61-d901-4c43-8718-34de98e26880-metrics-certs") pod "router-default-65cf6cdd84-r95n8" (UID: "6854cf61-d901-4c43-8718-34de98e26880") : secret "router-metrics-certs-default" not found
May 25 11:18:28.204776 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:28.204740    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/69432372-dd12-48f9-9cf4-8b194109b426-metrics-tls\") pod \"dns-default-vvfw5\" (UID: \"69432372-dd12-48f9-9cf4-8b194109b426\") " pod="openshift-dns/dns-default-vvfw5"
May 25 11:18:28.204936 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:28.204824    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ed3452ef-2260-4664-92ab-549403628cc8-cert\") pod \"ingress-canary-2mqc7\" (UID: \"ed3452ef-2260-4664-92ab-549403628cc8\") " pod="openshift-ingress-canary/ingress-canary-2mqc7"
May 25 11:18:28.204936 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:28.204872    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f7e56aa6-aa4b-422e-a3a8-a2bbdba923d3-samples-operator-tls\") pod \"cluster-samples-operator-697587599-l976t\" (UID: \"f7e56aa6-aa4b-422e-a3a8-a2bbdba923d3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-l976t"
May 25 11:18:28.204936 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:18:28.204891    2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 25 11:18:28.205029 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:18:28.204952    2560 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 25 11:18:28.205029 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:18:28.204983    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/69432372-dd12-48f9-9cf4-8b194109b426-metrics-tls podName:69432372-dd12-48f9-9cf4-8b194109b426 nodeName:}" failed. No retries permitted until 2026-05-25 11:18:44.204967279 +0000 UTC m=+65.885945230 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/69432372-dd12-48f9-9cf4-8b194109b426-metrics-tls") pod "dns-default-vvfw5" (UID: "69432372-dd12-48f9-9cf4-8b194109b426") : secret "dns-default-metrics-tls" not found
May 25 11:18:28.205029 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:18:28.205000    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed3452ef-2260-4664-92ab-549403628cc8-cert podName:ed3452ef-2260-4664-92ab-549403628cc8 nodeName:}" failed. No retries permitted until 2026-05-25 11:18:44.204991791 +0000 UTC m=+65.885969742 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/ed3452ef-2260-4664-92ab-549403628cc8-cert") pod "ingress-canary-2mqc7" (UID: "ed3452ef-2260-4664-92ab-549403628cc8") : secret "canary-serving-cert" not found
May 25 11:18:28.208832 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:28.208801    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f7e56aa6-aa4b-422e-a3a8-a2bbdba923d3-samples-operator-tls\") pod \"cluster-samples-operator-697587599-l976t\" (UID: \"f7e56aa6-aa4b-422e-a3a8-a2bbdba923d3\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-l976t"
May 25 11:18:28.218544 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:28.218523    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-l976t"
May 25 11:18:28.348292 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:28.348262    2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-697587599-l976t"]
May 25 11:18:29.174003 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:29.173971    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-l976t" event={"ID":"f7e56aa6-aa4b-422e-a3a8-a2bbdba923d3","Type":"ContainerStarted","Data":"00622a10916dd8c26e4a8906b65749f348f376ad7cb6f4b18949780caa26abc6"}
May 25 11:18:31.181611 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:31.181577    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-l976t" event={"ID":"f7e56aa6-aa4b-422e-a3a8-a2bbdba923d3","Type":"ContainerStarted","Data":"db3c13d31c7f3132355a100f16f0336c8374394ef7dddfc373977642a7fc1a9d"}
May 25 11:18:31.181611 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:31.181614    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-l976t" event={"ID":"f7e56aa6-aa4b-422e-a3a8-a2bbdba923d3","Type":"ContainerStarted","Data":"4c44f1cbd481f917e23f11dd1e032dfcca750e3650eac49b712067e2654be5fa"}
May 25 11:18:31.205592 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:31.205552    2560 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-l976t" podStartSLOduration=19.217408009 podStartE2EDuration="21.205538347s" podCreationTimestamp="2026-05-25 11:18:10 +0000 UTC" firstStartedPulling="2026-05-25 11:18:28.428964275 +0000 UTC m=+50.109942226" lastFinishedPulling="2026-05-25 11:18:30.417094613 +0000 UTC m=+52.098072564" observedRunningTime="2026-05-25 11:18:31.204733309 +0000 UTC m=+52.885711281" watchObservedRunningTime="2026-05-25 11:18:31.205538347 +0000 UTC m=+52.886516328"
May 25 11:18:37.103040 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:37.103010    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-m55v2"
May 25 11:18:37.930739 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:37.930708    2560 scope.go:117] "RemoveContainer" containerID="8c3250eec3ef03820ae865392ae5d81191348c8c4a74f189b8314099901ddd8d"
May 25 11:18:38.200056 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:38.199968    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-zd54m_2be32661-9288-4902-9d67-1a5e8f74670d/console-operator/1.log"
May 25 11:18:38.200514 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:38.200053    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-76dff48d59-zd54m" event={"ID":"2be32661-9288-4902-9d67-1a5e8f74670d","Type":"ContainerStarted","Data":"d538b4fa1c60c9d6df621750b543f8a8db51ace4d06f7b404ee17f0b68b259dc"}
May 25 11:18:38.200514 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:38.200432    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-76dff48d59-zd54m"
May 25 11:18:38.220392 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:38.220351    2560 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console-operator/console-operator-76dff48d59-zd54m" podStartSLOduration=23.799057917 podStartE2EDuration="28.220337817s" podCreationTimestamp="2026-05-25 11:18:10 +0000 UTC" firstStartedPulling="2026-05-25 11:18:12.908142624 +0000 UTC m=+34.589120575" lastFinishedPulling="2026-05-25 11:18:17.329422524 +0000 UTC m=+39.010400475" observedRunningTime="2026-05-25 11:18:38.220148694 +0000 UTC m=+59.901126667" watchObservedRunningTime="2026-05-25 11:18:38.220337817 +0000 UTC m=+59.901315836"
May 25 11:18:38.886224 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:38.886181    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-76dff48d59-zd54m"
May 25 11:18:44.029791 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:44.029751    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/01fa335c-4082-46b1-b059-8ae4c68ef3d3-registry-tls\") pod \"image-registry-5f8d6ccbdc-8pc8t\" (UID: \"01fa335c-4082-46b1-b059-8ae4c68ef3d3\") " pod="openshift-image-registry/image-registry-5f8d6ccbdc-8pc8t"
May 25 11:18:44.032188 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:44.032167    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/01fa335c-4082-46b1-b059-8ae4c68ef3d3-registry-tls\") pod \"image-registry-5f8d6ccbdc-8pc8t\" (UID: \"01fa335c-4082-46b1-b059-8ae4c68ef3d3\") " pod="openshift-image-registry/image-registry-5f8d6ccbdc-8pc8t"
May 25 11:18:44.130221 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:44.130170    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6854cf61-d901-4c43-8718-34de98e26880-metrics-certs\") pod \"router-default-65cf6cdd84-r95n8\" (UID: \"6854cf61-d901-4c43-8718-34de98e26880\") " pod="openshift-ingress/router-default-65cf6cdd84-r95n8"
May 25 11:18:44.130221 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:44.130224    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/251cca37-3cc4-4187-895f-e273d9eac261-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6c75959455-n8lh8\" (UID: \"251cca37-3cc4-4187-895f-e273d9eac261\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-n8lh8"
May 25 11:18:44.130462 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:44.130293    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6854cf61-d901-4c43-8718-34de98e26880-service-ca-bundle\") pod \"router-default-65cf6cdd84-r95n8\" (UID: \"6854cf61-d901-4c43-8718-34de98e26880\") " pod="openshift-ingress/router-default-65cf6cdd84-r95n8"
May 25 11:18:44.130943 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:44.130922    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6854cf61-d901-4c43-8718-34de98e26880-service-ca-bundle\") pod \"router-default-65cf6cdd84-r95n8\" (UID: \"6854cf61-d901-4c43-8718-34de98e26880\") " pod="openshift-ingress/router-default-65cf6cdd84-r95n8"
May 25 11:18:44.132656 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:44.132630    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6854cf61-d901-4c43-8718-34de98e26880-metrics-certs\") pod \"router-default-65cf6cdd84-r95n8\" (UID: \"6854cf61-d901-4c43-8718-34de98e26880\") " pod="openshift-ingress/router-default-65cf6cdd84-r95n8"
May 25 11:18:44.132735 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:44.132641    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/251cca37-3cc4-4187-895f-e273d9eac261-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6c75959455-n8lh8\" (UID: \"251cca37-3cc4-4187-895f-e273d9eac261\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-n8lh8"
May 25 11:18:44.231508 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:44.231478    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/69432372-dd12-48f9-9cf4-8b194109b426-metrics-tls\") pod \"dns-default-vvfw5\" (UID: \"69432372-dd12-48f9-9cf4-8b194109b426\") " pod="openshift-dns/dns-default-vvfw5"
May 25 11:18:44.231662 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:44.231544    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ed3452ef-2260-4664-92ab-549403628cc8-cert\") pod \"ingress-canary-2mqc7\" (UID: \"ed3452ef-2260-4664-92ab-549403628cc8\") " pod="openshift-ingress-canary/ingress-canary-2mqc7"
May 25 11:18:44.233904 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:44.233870    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/69432372-dd12-48f9-9cf4-8b194109b426-metrics-tls\") pod \"dns-default-vvfw5\" (UID: \"69432372-dd12-48f9-9cf4-8b194109b426\") " pod="openshift-dns/dns-default-vvfw5"
May 25 11:18:44.234003 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:44.233922    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ed3452ef-2260-4664-92ab-549403628cc8-cert\") pod \"ingress-canary-2mqc7\" (UID: \"ed3452ef-2260-4664-92ab-549403628cc8\") " pod="openshift-ingress-canary/ingress-canary-2mqc7"
May 25 11:18:44.330822 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:44.330746    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-lh8tr\""
May 25 11:18:44.339036 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:44.339013    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5f8d6ccbdc-8pc8t"
May 25 11:18:44.344082 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:44.344063    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-bzkz7\""
May 25 11:18:44.352477 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:44.352459    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-65cf6cdd84-r95n8"
May 25 11:18:44.358522 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:44.358503    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-vvc8x\""
May 25 11:18:44.366790 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:44.366770    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-n8lh8"
May 25 11:18:44.490174 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:44.490119    2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5f8d6ccbdc-8pc8t"]
May 25 11:18:44.490691 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:44.490664    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-kqrg2\""
May 25 11:18:44.494746 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:18:44.494718    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01fa335c_4082_46b1_b059_8ae4c68ef3d3.slice/crio-77230f888307d67a7adbb07402e93a8ec501d56d81822bb6ce683101ee1cbce1 WatchSource:0}: Error finding container 77230f888307d67a7adbb07402e93a8ec501d56d81822bb6ce683101ee1cbce1: Status 404 returned error can't find the container with id 77230f888307d67a7adbb07402e93a8ec501d56d81822bb6ce683101ee1cbce1
May 25 11:18:44.497742 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:44.497722    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-2mqc7"
May 25 11:18:44.505831 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:44.505699    2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-65cf6cdd84-r95n8"]
May 25 11:18:44.508477 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:18:44.508446    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6854cf61_d901_4c43_8718_34de98e26880.slice/crio-8e025eb9f39f2187b6c09cfd02a0e5042712c45c7cf86db1b469335a6c1b5067 WatchSource:0}: Error finding container 8e025eb9f39f2187b6c09cfd02a0e5042712c45c7cf86db1b469335a6c1b5067: Status 404 returned error can't find the container with id 8e025eb9f39f2187b6c09cfd02a0e5042712c45c7cf86db1b469335a6c1b5067
May 25 11:18:44.509688 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:44.509671    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-9k5hq\""
May 25 11:18:44.517565 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:44.517541    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-vvfw5"
May 25 11:18:44.523035 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:44.523014    2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6c75959455-n8lh8"]
May 25 11:18:44.525431 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:18:44.525408    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod251cca37_3cc4_4187_895f_e273d9eac261.slice/crio-c461f09579e687434f9bd873cec0acc2cfa035c2a004a7eafc7bf4f11a1e0e06 WatchSource:0}: Error finding container c461f09579e687434f9bd873cec0acc2cfa035c2a004a7eafc7bf4f11a1e0e06: Status 404 returned error can't find the container with id c461f09579e687434f9bd873cec0acc2cfa035c2a004a7eafc7bf4f11a1e0e06
May 25 11:18:44.634358 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:44.634331    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0dc07c1-2dce-4b08-9fee-41a5937a7bae-metrics-certs\") pod \"network-metrics-daemon-hxdhk\" (UID: \"b0dc07c1-2dce-4b08-9fee-41a5937a7bae\") " pod="openshift-multus/network-metrics-daemon-hxdhk"
May 25 11:18:44.636824 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:44.636802    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
May 25 11:18:44.642319 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:44.642295    2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-2mqc7"]
May 25 11:18:44.646202 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:18:44.646169    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded3452ef_2260_4664_92ab_549403628cc8.slice/crio-7c0383c746b72cfcb01960d245f976e2b94b7f07f7db64f1bbe2403dcdf43d61 WatchSource:0}: Error finding container 7c0383c746b72cfcb01960d245f976e2b94b7f07f7db64f1bbe2403dcdf43d61: Status 404 returned error can't find the container with id 7c0383c746b72cfcb01960d245f976e2b94b7f07f7db64f1bbe2403dcdf43d61
May 25 11:18:44.647628 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:44.647594    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0dc07c1-2dce-4b08-9fee-41a5937a7bae-metrics-certs\") pod \"network-metrics-daemon-hxdhk\" (UID: \"b0dc07c1-2dce-4b08-9fee-41a5937a7bae\") " pod="openshift-multus/network-metrics-daemon-hxdhk"
May 25 11:18:44.668146 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:44.668121    2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-vvfw5"]
May 25 11:18:44.671178 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:18:44.671139    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69432372_dd12_48f9_9cf4_8b194109b426.slice/crio-97fb55e8c21f73c00cc151120bda0887f24f4539fda5e45aa8c63797fce6e4a0 WatchSource:0}: Error finding container 97fb55e8c21f73c00cc151120bda0887f24f4539fda5e45aa8c63797fce6e4a0: Status 404 returned error can't find the container with id 97fb55e8c21f73c00cc151120bda0887f24f4539fda5e45aa8c63797fce6e4a0
May 25 11:18:44.735063 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:44.735029    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c0068087-c30b-4ff5-a794-b84738aaa168-original-pull-secret\") pod \"global-pull-secret-syncer-nq67w\" (UID: \"c0068087-c30b-4ff5-a794-b84738aaa168\") " pod="kube-system/global-pull-secret-syncer-nq67w"
May 25 11:18:44.738889 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:44.738868    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
May 25 11:18:44.748366 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:44.748346    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c0068087-c30b-4ff5-a794-b84738aaa168-original-pull-secret\") pod \"global-pull-secret-syncer-nq67w\" (UID: \"c0068087-c30b-4ff5-a794-b84738aaa168\") " pod="kube-system/global-pull-secret-syncer-nq67w"
May 25 11:18:44.835979 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:44.835940    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-97fvl\" (UniqueName: \"kubernetes.io/projected/0d6828f5-3d2a-417d-ab45-6750dd8cdb92-kube-api-access-97fvl\") pod \"network-check-target-zfljp\" (UID: \"0d6828f5-3d2a-417d-ab45-6750dd8cdb92\") " pod="openshift-network-diagnostics/network-check-target-zfljp"
May 25 11:18:44.838827 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:44.838802    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
May 25 11:18:44.847111 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:44.847052    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nq67w"
May 25 11:18:44.849433 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:44.849413    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
May 25 11:18:44.860023 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:44.859998    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-97fvl\" (UniqueName: \"kubernetes.io/projected/0d6828f5-3d2a-417d-ab45-6750dd8cdb92-kube-api-access-97fvl\") pod \"network-check-target-zfljp\" (UID: \"0d6828f5-3d2a-417d-ab45-6750dd8cdb92\") " pod="openshift-network-diagnostics/network-check-target-zfljp"
May 25 11:18:44.860442 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:44.860426    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-lkpxl\""
May 25 11:18:44.867846 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:44.867807    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-mcnj6\""
May 25 11:18:44.867971 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:44.867888    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hxdhk"
May 25 11:18:44.875608 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:44.875548    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zfljp"
May 25 11:18:45.030103 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:45.030068    2560 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-7dgx7"]
May 25 11:18:45.040588 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:45.040560    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-7dgx7"
May 25 11:18:45.045352 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:45.045172    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
May 25 11:18:45.045538 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:45.045371    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
May 25 11:18:45.045538 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:45.045403    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-m9btq\""
May 25 11:18:45.050199 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:45.049710    2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-7dgx7"]
May 25 11:18:45.060992 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:45.060944    2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-nq67w"]
May 25 11:18:45.067575 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:18:45.067549    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0068087_c30b_4ff5_a794_b84738aaa168.slice/crio-fbb7c72681f64641b90221a97f9c364f81e21b7786c6e4f4d0fa5d174b452756 WatchSource:0}: Error finding container fbb7c72681f64641b90221a97f9c364f81e21b7786c6e4f4d0fa5d174b452756: Status 404 returned error can't find the container with id fbb7c72681f64641b90221a97f9c364f81e21b7786c6e4f4d0fa5d174b452756
May 25 11:18:45.138137 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:45.138013    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/64750c18-e28d-484d-891f-44b12392fe2c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7dgx7\" (UID: \"64750c18-e28d-484d-891f-44b12392fe2c\") " pod="openshift-insights/insights-runtime-extractor-7dgx7"
May 25 11:18:45.138137 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:45.138067    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhs4j\" (UniqueName: \"kubernetes.io/projected/64750c18-e28d-484d-891f-44b12392fe2c-kube-api-access-vhs4j\") pod \"insights-runtime-extractor-7dgx7\" (UID: \"64750c18-e28d-484d-891f-44b12392fe2c\") " pod="openshift-insights/insights-runtime-extractor-7dgx7"
May 25 11:18:45.138398 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:45.138155    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/64750c18-e28d-484d-891f-44b12392fe2c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7dgx7\" (UID: \"64750c18-e28d-484d-891f-44b12392fe2c\") " pod="openshift-insights/insights-runtime-extractor-7dgx7"
May 25 11:18:45.138398 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:45.138226    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/64750c18-e28d-484d-891f-44b12392fe2c-crio-socket\") pod \"insights-runtime-extractor-7dgx7\" (UID: \"64750c18-e28d-484d-891f-44b12392fe2c\") " pod="openshift-insights/insights-runtime-extractor-7dgx7"
May 25 11:18:45.138398 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:45.138274    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/64750c18-e28d-484d-891f-44b12392fe2c-data-volume\") pod \"insights-runtime-extractor-7dgx7\" (UID: \"64750c18-e28d-484d-891f-44b12392fe2c\") " pod="openshift-insights/insights-runtime-extractor-7dgx7"
May 25 11:18:45.222437 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:45.222378    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-vvfw5" event={"ID":"69432372-dd12-48f9-9cf4-8b194109b426","Type":"ContainerStarted","Data":"97fb55e8c21f73c00cc151120bda0887f24f4539fda5e45aa8c63797fce6e4a0"}
May 25 11:18:45.223831 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:45.223787    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-n8lh8" event={"ID":"251cca37-3cc4-4187-895f-e273d9eac261","Type":"ContainerStarted","Data":"c461f09579e687434f9bd873cec0acc2cfa035c2a004a7eafc7bf4f11a1e0e06"}
May 25 11:18:45.227381 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:45.227349    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-65cf6cdd84-r95n8" event={"ID":"6854cf61-d901-4c43-8718-34de98e26880","Type":"ContainerStarted","Data":"589db8f9bfa35db7a9c2336dd59488bc00acf1b88d35a0f4fe18ac6299845449"}
May 25 11:18:45.227484 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:45.227394    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-65cf6cdd84-r95n8" event={"ID":"6854cf61-d901-4c43-8718-34de98e26880","Type":"ContainerStarted","Data":"8e025eb9f39f2187b6c09cfd02a0e5042712c45c7cf86db1b469335a6c1b5067"}
May 25 11:18:45.230088 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:45.230060    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5f8d6ccbdc-8pc8t" event={"ID":"01fa335c-4082-46b1-b059-8ae4c68ef3d3","Type":"ContainerStarted","Data":"eccd0e95681a9441d7c26eb9cc095618004f4a0f0b5118863d8c663e3d5c7e7d"}
May 25 11:18:45.230194 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:45.230096    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5f8d6ccbdc-8pc8t" event={"ID":"01fa335c-4082-46b1-b059-8ae4c68ef3d3","Type":"ContainerStarted","Data":"77230f888307d67a7adbb07402e93a8ec501d56d81822bb6ce683101ee1cbce1"}
May 25 11:18:45.230194 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:45.230153    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5f8d6ccbdc-8pc8t"
May 25 11:18:45.232112 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:45.232072    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-nq67w" event={"ID":"c0068087-c30b-4ff5-a794-b84738aaa168","Type":"ContainerStarted","Data":"fbb7c72681f64641b90221a97f9c364f81e21b7786c6e4f4d0fa5d174b452756"}
May 25 11:18:45.233493 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:45.233452    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-2mqc7" event={"ID":"ed3452ef-2260-4664-92ab-549403628cc8","Type":"ContainerStarted","Data":"7c0383c746b72cfcb01960d245f976e2b94b7f07f7db64f1bbe2403dcdf43d61"}
May 25 11:18:45.240383 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:45.239376    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/64750c18-e28d-484d-891f-44b12392fe2c-data-volume\") pod \"insights-runtime-extractor-7dgx7\" (UID: \"64750c18-e28d-484d-891f-44b12392fe2c\") " pod="openshift-insights/insights-runtime-extractor-7dgx7"
May 25 11:18:45.240383 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:45.239473    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/64750c18-e28d-484d-891f-44b12392fe2c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7dgx7\" (UID: \"64750c18-e28d-484d-891f-44b12392fe2c\") " pod="openshift-insights/insights-runtime-extractor-7dgx7"
May 25 11:18:45.240383 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:45.239516    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-vhs4j\" (UniqueName: \"kubernetes.io/projected/64750c18-e28d-484d-891f-44b12392fe2c-kube-api-access-vhs4j\") pod \"insights-runtime-extractor-7dgx7\" (UID: \"64750c18-e28d-484d-891f-44b12392fe2c\") " pod="openshift-insights/insights-runtime-extractor-7dgx7"
May 25 11:18:45.240383 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:45.239551    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/64750c18-e28d-484d-891f-44b12392fe2c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7dgx7\" (UID: \"64750c18-e28d-484d-891f-44b12392fe2c\") " pod="openshift-insights/insights-runtime-extractor-7dgx7"
May 25 11:18:45.240383 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:45.239601    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/64750c18-e28d-484d-891f-44b12392fe2c-crio-socket\") pod \"insights-runtime-extractor-7dgx7\" (UID: \"64750c18-e28d-484d-891f-44b12392fe2c\") " pod="openshift-insights/insights-runtime-extractor-7dgx7"
May 25 11:18:45.240383 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:45.239703    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/64750c18-e28d-484d-891f-44b12392fe2c-crio-socket\") pod \"insights-runtime-extractor-7dgx7\" (UID: \"64750c18-e28d-484d-891f-44b12392fe2c\") " pod="openshift-insights/insights-runtime-extractor-7dgx7"
May 25 11:18:45.240383 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:45.240003    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/64750c18-e28d-484d-891f-44b12392fe2c-data-volume\") pod \"insights-runtime-extractor-7dgx7\" (UID: \"64750c18-e28d-484d-891f-44b12392fe2c\") " pod="openshift-insights/insights-runtime-extractor-7dgx7"
May 25 11:18:45.240908 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:45.240871    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/64750c18-e28d-484d-891f-44b12392fe2c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7dgx7\" (UID: \"64750c18-e28d-484d-891f-44b12392fe2c\") " pod="openshift-insights/insights-runtime-extractor-7dgx7"
May 25 11:18:45.246473 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:45.246429    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/64750c18-e28d-484d-891f-44b12392fe2c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7dgx7\" (UID: \"64750c18-e28d-484d-891f-44b12392fe2c\") " pod="openshift-insights/insights-runtime-extractor-7dgx7"
May 25 11:18:45.287329 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:45.287265    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhs4j\" (UniqueName: \"kubernetes.io/projected/64750c18-e28d-484d-891f-44b12392fe2c-kube-api-access-vhs4j\") pod \"insights-runtime-extractor-7dgx7\" (UID: \"64750c18-e28d-484d-891f-44b12392fe2c\") " pod="openshift-insights/insights-runtime-extractor-7dgx7"
May 25 11:18:45.308946 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:45.308705    2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-zfljp"]
May 25 11:18:45.317162 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:45.315794    2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-hxdhk"]
May 25 11:18:45.321668 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:18:45.321638    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0dc07c1_2dce_4b08_9fee_41a5937a7bae.slice/crio-ffeb830ad446e6313628323772f4035ab4b662c5c0c01e52a44c2b2e47d7e349 WatchSource:0}: Error finding container ffeb830ad446e6313628323772f4035ab4b662c5c0c01e52a44c2b2e47d7e349: Status 404 returned error can't find the container with id ffeb830ad446e6313628323772f4035ab4b662c5c0c01e52a44c2b2e47d7e349
May 25 11:18:45.353607 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:45.353560    2560 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-65cf6cdd84-r95n8"
May 25 11:18:45.356315 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:45.356279    2560 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-65cf6cdd84-r95n8"
May 25 11:18:45.369080 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:45.368671    2560 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ingress/router-default-65cf6cdd84-r95n8" podStartSLOduration=46.368655979 podStartE2EDuration="46.368655979s" podCreationTimestamp="2026-05-25 11:17:59 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 11:18:45.321249327 +0000 UTC m=+67.002227300" watchObservedRunningTime="2026-05-25 11:18:45.368655979 +0000 UTC m=+67.049633952"
May 25 11:18:45.383250 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:45.383198    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-7dgx7"
May 25 11:18:45.425088 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:45.425032    2560 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5f8d6ccbdc-8pc8t" podStartSLOduration=66.42501397 podStartE2EDuration="1m6.42501397s" podCreationTimestamp="2026-05-25 11:17:39 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 11:18:45.368577959 +0000 UTC m=+67.049555932" watchObservedRunningTime="2026-05-25 11:18:45.42501397 +0000 UTC m=+67.105991946"
May 25 11:18:45.567302 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:45.567242    2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-7dgx7"]
May 25 11:18:45.571261 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:18:45.571229    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64750c18_e28d_484d_891f_44b12392fe2c.slice/crio-595a3427165c29d89be4db33652bd4776012e1bca243d30738484bc9df6d3183 WatchSource:0}: Error finding container 595a3427165c29d89be4db33652bd4776012e1bca243d30738484bc9df6d3183: Status 404 returned error can't find the container with id 595a3427165c29d89be4db33652bd4776012e1bca243d30738484bc9df6d3183
May 25 11:18:46.239353 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:46.239313    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-zfljp" event={"ID":"0d6828f5-3d2a-417d-ab45-6750dd8cdb92","Type":"ContainerStarted","Data":"31d1c189d17c5e58a3bc4891af5aba9cabb1291d1dbe9062f67aa8e7660015dd"}
May 25 11:18:46.240788 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:46.240758    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-hxdhk" event={"ID":"b0dc07c1-2dce-4b08-9fee-41a5937a7bae","Type":"ContainerStarted","Data":"ffeb830ad446e6313628323772f4035ab4b662c5c0c01e52a44c2b2e47d7e349"}
May 25 11:18:46.243115 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:46.243089    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7dgx7" event={"ID":"64750c18-e28d-484d-891f-44b12392fe2c","Type":"ContainerStarted","Data":"c3d1a160ccfce519d1f524db539a0b8c370678e1455254f69fed1b0040726d2a"}
May 25 11:18:46.243236 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:46.243124    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7dgx7" event={"ID":"64750c18-e28d-484d-891f-44b12392fe2c","Type":"ContainerStarted","Data":"595a3427165c29d89be4db33652bd4776012e1bca243d30738484bc9df6d3183"}
May 25 11:18:46.243448 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:46.243425    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-65cf6cdd84-r95n8"
May 25 11:18:46.244966 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:46.244798    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-65cf6cdd84-r95n8"
May 25 11:18:51.261571 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:51.261468    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-nq67w" event={"ID":"c0068087-c30b-4ff5-a794-b84738aaa168","Type":"ContainerStarted","Data":"fabca1472d27dc161c490d2271d4737d805bfb7cc75405431387d93137465ea3"}
May 25 11:18:51.263868 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:51.263823    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-2mqc7" event={"ID":"ed3452ef-2260-4664-92ab-549403628cc8","Type":"ContainerStarted","Data":"7fbdf732c67507c2c12f70f5790a71f71d9fc84aeacb357a56c47c8393e1d236"}
May 25 11:18:51.265883 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:51.265859    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-zfljp" event={"ID":"0d6828f5-3d2a-417d-ab45-6750dd8cdb92","Type":"ContainerStarted","Data":"7aa4d3904b832489829f302602f2268fd6ea791167f151aab5f570b2ee5a35de"}
May 25 11:18:51.267327 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:51.267177    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-zfljp"
May 25 11:18:51.269093 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:51.269065    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-hxdhk" event={"ID":"b0dc07c1-2dce-4b08-9fee-41a5937a7bae","Type":"ContainerStarted","Data":"65403a860ee64d475a3b440070deb7b9fbd2ed99f08d1c2c9241bd1ada743390"}
May 25 11:18:51.271446 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:51.271404    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-vvfw5" event={"ID":"69432372-dd12-48f9-9cf4-8b194109b426","Type":"ContainerStarted","Data":"6bddfd0f81a4b9f763f4dd6827d06a69b53922fd2f9682406e7cced0163091c7"}
May 25 11:18:51.273108 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:51.273088    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-n8lh8" event={"ID":"251cca37-3cc4-4187-895f-e273d9eac261","Type":"ContainerStarted","Data":"86e980c00bc170ab8c7125390eaeb8a2c08a153157b57c0e29331672a062239a"}
May 25 11:18:51.276178 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:51.276145    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7dgx7" event={"ID":"64750c18-e28d-484d-891f-44b12392fe2c","Type":"ContainerStarted","Data":"bc08cadd3db7151a5127cbad2ef2eb50e202f317f1f4dd209fe535452fe3b528"}
May 25 11:18:51.312809 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:51.312759    2560 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-zfljp" podStartSLOduration=66.653475985 podStartE2EDuration="1m12.312743151s" podCreationTimestamp="2026-05-25 11:17:39 +0000 UTC" firstStartedPulling="2026-05-25 11:18:45.316106449 +0000 UTC m=+66.997084414" lastFinishedPulling="2026-05-25 11:18:50.975373615 +0000 UTC m=+72.656351580" observedRunningTime="2026-05-25 11:18:51.312532376 +0000 UTC m=+72.993510386" watchObservedRunningTime="2026-05-25 11:18:51.312743151 +0000 UTC m=+72.993721123"
May 25 11:18:51.313192 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:51.313153    2560 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-nq67w" podStartSLOduration=65.408416671 podStartE2EDuration="1m11.313143814s" podCreationTimestamp="2026-05-25 11:17:40 +0000 UTC" firstStartedPulling="2026-05-25 11:18:45.070820986 +0000 UTC m=+66.751798939" lastFinishedPulling="2026-05-25 11:18:50.975548129 +0000 UTC m=+72.656526082" observedRunningTime="2026-05-25 11:18:51.287067705 +0000 UTC m=+72.968045679" watchObservedRunningTime="2026-05-25 11:18:51.313143814 +0000 UTC m=+72.994121841"
May 25 11:18:51.338041 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:51.338000    2560 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-2mqc7" podStartSLOduration=33.012608893 podStartE2EDuration="39.337985253s" podCreationTimestamp="2026-05-25 11:18:12 +0000 UTC" firstStartedPulling="2026-05-25 11:18:44.648524493 +0000 UTC m=+66.329502444" lastFinishedPulling="2026-05-25 11:18:50.973900843 +0000 UTC m=+72.654878804" observedRunningTime="2026-05-25 11:18:51.337949479 +0000 UTC m=+73.018927452" watchObservedRunningTime="2026-05-25 11:18:51.337985253 +0000 UTC m=+73.018963225"
May 25 11:18:51.380004 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:51.379930    2560 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-n8lh8" podStartSLOduration=45.932229614 podStartE2EDuration="52.379910284s" podCreationTimestamp="2026-05-25 11:17:59 +0000 UTC" firstStartedPulling="2026-05-25 11:18:44.52771919 +0000 UTC m=+66.208697140" lastFinishedPulling="2026-05-25 11:18:50.975399852 +0000 UTC m=+72.656377810" observedRunningTime="2026-05-25 11:18:51.376367916 +0000 UTC m=+73.057345890" watchObservedRunningTime="2026-05-25 11:18:51.379910284 +0000 UTC m=+73.060888297"
May 25 11:18:52.281157 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:52.281130    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-vvfw5" event={"ID":"69432372-dd12-48f9-9cf4-8b194109b426","Type":"ContainerStarted","Data":"c925622002a504e4eee0763ca5f7ad01f4518e0c00071fdd0f57ded0852e3a76"}
May 25 11:18:52.281522 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:52.281238    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-vvfw5"
May 25 11:18:52.282685 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:52.282660    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-hxdhk" event={"ID":"b0dc07c1-2dce-4b08-9fee-41a5937a7bae","Type":"ContainerStarted","Data":"fc47f353f256dfb81f1879e5f029d2165ec178b867312dabfc06ba553e27f6df"}
May 25 11:18:52.330761 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:52.330717    2560 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-dns/dns-default-vvfw5" podStartSLOduration=34.029784035 podStartE2EDuration="40.330702391s" podCreationTimestamp="2026-05-25 11:18:12 +0000 UTC" firstStartedPulling="2026-05-25 11:18:44.67298559 +0000 UTC m=+66.353963545" lastFinishedPulling="2026-05-25 11:18:50.973903949 +0000 UTC m=+72.654881901" observedRunningTime="2026-05-25 11:18:52.325264216 +0000 UTC m=+74.006242189" watchObservedRunningTime="2026-05-25 11:18:52.330702391 +0000 UTC m=+74.011680363"
May 25 11:18:52.352947 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:52.352903    2560 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-hxdhk" podStartSLOduration=67.701884659 podStartE2EDuration="1m13.352889239s" podCreationTimestamp="2026-05-25 11:17:39 +0000 UTC" firstStartedPulling="2026-05-25 11:18:45.324482894 +0000 UTC m=+67.005460847" lastFinishedPulling="2026-05-25 11:18:50.97548747 +0000 UTC m=+72.656465427" observedRunningTime="2026-05-25 11:18:52.351542238 +0000 UTC m=+74.032520211" watchObservedRunningTime="2026-05-25 11:18:52.352889239 +0000 UTC m=+74.033867245"
May 25 11:18:53.289722 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:53.289675    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7dgx7" event={"ID":"64750c18-e28d-484d-891f-44b12392fe2c","Type":"ContainerStarted","Data":"54bb2c1321edcebec0cd08c8001e6355b1ff0db93cfe88dafa6dac2f35e24c0d"}
May 25 11:18:53.309281 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:53.309204    2560 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-7dgx7" podStartSLOduration=1.754032907 podStartE2EDuration="8.3091903s" podCreationTimestamp="2026-05-25 11:18:45 +0000 UTC" firstStartedPulling="2026-05-25 11:18:45.704014651 +0000 UTC m=+67.384992607" lastFinishedPulling="2026-05-25 11:18:52.259172046 +0000 UTC m=+73.940150000" observedRunningTime="2026-05-25 11:18:53.308908382 +0000 UTC m=+74.989886357" watchObservedRunningTime="2026-05-25 11:18:53.3091903 +0000 UTC m=+74.990168272"
May 25 11:18:55.781625 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:55.781589    2560 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console/console-676875fdf9-drjfl"]
May 25 11:18:55.783506 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:55.783490    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-676875fdf9-drjfl"
May 25 11:18:55.787488 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:55.787454    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
May 25 11:18:55.787488 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:55.787454    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-hcjk4\""
May 25 11:18:55.787662 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:55.787488    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
May 25 11:18:55.787662 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:55.787503    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
May 25 11:18:55.787662 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:55.787454    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
May 25 11:18:55.787662 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:55.787454    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
May 25 11:18:55.787662 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:55.787596    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
May 25 11:18:55.787932 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:55.787880    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
May 25 11:18:55.792573 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:55.792551    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
May 25 11:18:55.795956 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:55.795938    2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-676875fdf9-drjfl"]
May 25 11:18:55.828200 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:55.828172    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xrlp\" (UniqueName: \"kubernetes.io/projected/21962877-1f1b-4993-8c1f-56042776afb4-kube-api-access-4xrlp\") pod \"console-676875fdf9-drjfl\" (UID: \"21962877-1f1b-4993-8c1f-56042776afb4\") " pod="openshift-console/console-676875fdf9-drjfl"
May 25 11:18:55.828350 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:55.828229    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/21962877-1f1b-4993-8c1f-56042776afb4-console-serving-cert\") pod \"console-676875fdf9-drjfl\" (UID: \"21962877-1f1b-4993-8c1f-56042776afb4\") " pod="openshift-console/console-676875fdf9-drjfl"
May 25 11:18:55.828350 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:55.828326    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/21962877-1f1b-4993-8c1f-56042776afb4-service-ca\") pod \"console-676875fdf9-drjfl\" (UID: \"21962877-1f1b-4993-8c1f-56042776afb4\") " pod="openshift-console/console-676875fdf9-drjfl"
May 25 11:18:55.828458 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:55.828377    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/21962877-1f1b-4993-8c1f-56042776afb4-console-config\") pod \"console-676875fdf9-drjfl\" (UID: \"21962877-1f1b-4993-8c1f-56042776afb4\") " pod="openshift-console/console-676875fdf9-drjfl"
May 25 11:18:55.828458 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:55.828409    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/21962877-1f1b-4993-8c1f-56042776afb4-oauth-serving-cert\") pod \"console-676875fdf9-drjfl\" (UID: \"21962877-1f1b-4993-8c1f-56042776afb4\") " pod="openshift-console/console-676875fdf9-drjfl"
May 25 11:18:55.828547 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:55.828455    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/21962877-1f1b-4993-8c1f-56042776afb4-console-oauth-config\") pod \"console-676875fdf9-drjfl\" (UID: \"21962877-1f1b-4993-8c1f-56042776afb4\") " pod="openshift-console/console-676875fdf9-drjfl"
May 25 11:18:55.828547 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:55.828489    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/21962877-1f1b-4993-8c1f-56042776afb4-trusted-ca-bundle\") pod \"console-676875fdf9-drjfl\" (UID: \"21962877-1f1b-4993-8c1f-56042776afb4\") " pod="openshift-console/console-676875fdf9-drjfl"
May 25 11:18:55.929762 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:55.929726    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-4xrlp\" (UniqueName: \"kubernetes.io/projected/21962877-1f1b-4993-8c1f-56042776afb4-kube-api-access-4xrlp\") pod \"console-676875fdf9-drjfl\" (UID: \"21962877-1f1b-4993-8c1f-56042776afb4\") " pod="openshift-console/console-676875fdf9-drjfl"
May 25 11:18:55.929900 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:55.929778    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/21962877-1f1b-4993-8c1f-56042776afb4-console-serving-cert\") pod \"console-676875fdf9-drjfl\" (UID: \"21962877-1f1b-4993-8c1f-56042776afb4\") " pod="openshift-console/console-676875fdf9-drjfl"
May 25 11:18:55.929900 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:55.929818    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/21962877-1f1b-4993-8c1f-56042776afb4-service-ca\") pod \"console-676875fdf9-drjfl\" (UID: \"21962877-1f1b-4993-8c1f-56042776afb4\") " pod="openshift-console/console-676875fdf9-drjfl"
May 25 11:18:55.929900 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:55.929851    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/21962877-1f1b-4993-8c1f-56042776afb4-console-config\") pod \"console-676875fdf9-drjfl\" (UID: \"21962877-1f1b-4993-8c1f-56042776afb4\") " pod="openshift-console/console-676875fdf9-drjfl"
May 25 11:18:55.930027 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:55.930003    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/21962877-1f1b-4993-8c1f-56042776afb4-oauth-serving-cert\") pod \"console-676875fdf9-drjfl\" (UID: \"21962877-1f1b-4993-8c1f-56042776afb4\") " pod="openshift-console/console-676875fdf9-drjfl"
May 25 11:18:55.930085 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:55.930068    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/21962877-1f1b-4993-8c1f-56042776afb4-console-oauth-config\") pod \"console-676875fdf9-drjfl\" (UID: \"21962877-1f1b-4993-8c1f-56042776afb4\") " pod="openshift-console/console-676875fdf9-drjfl"
May 25 11:18:55.930138 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:55.930112    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/21962877-1f1b-4993-8c1f-56042776afb4-trusted-ca-bundle\") pod \"console-676875fdf9-drjfl\" (UID: \"21962877-1f1b-4993-8c1f-56042776afb4\") " pod="openshift-console/console-676875fdf9-drjfl"
May 25 11:18:55.930668 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:55.930631    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/21962877-1f1b-4993-8c1f-56042776afb4-oauth-serving-cert\") pod \"console-676875fdf9-drjfl\" (UID: \"21962877-1f1b-4993-8c1f-56042776afb4\") " pod="openshift-console/console-676875fdf9-drjfl"
May 25 11:18:55.930668 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:55.930634    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/21962877-1f1b-4993-8c1f-56042776afb4-service-ca\") pod \"console-676875fdf9-drjfl\" (UID: \"21962877-1f1b-4993-8c1f-56042776afb4\") " pod="openshift-console/console-676875fdf9-drjfl"
May 25 11:18:55.930844 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:55.930686    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/21962877-1f1b-4993-8c1f-56042776afb4-console-config\") pod \"console-676875fdf9-drjfl\" (UID: \"21962877-1f1b-4993-8c1f-56042776afb4\") " pod="openshift-console/console-676875fdf9-drjfl"
May 25 11:18:55.930899 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:55.930854    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/21962877-1f1b-4993-8c1f-56042776afb4-trusted-ca-bundle\") pod \"console-676875fdf9-drjfl\" (UID: \"21962877-1f1b-4993-8c1f-56042776afb4\") " pod="openshift-console/console-676875fdf9-drjfl"
May 25 11:18:55.932461 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:55.932444    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/21962877-1f1b-4993-8c1f-56042776afb4-console-oauth-config\") pod \"console-676875fdf9-drjfl\" (UID: \"21962877-1f1b-4993-8c1f-56042776afb4\") " pod="openshift-console/console-676875fdf9-drjfl"
May 25 11:18:55.932461 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:55.932453    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/21962877-1f1b-4993-8c1f-56042776afb4-console-serving-cert\") pod \"console-676875fdf9-drjfl\" (UID: \"21962877-1f1b-4993-8c1f-56042776afb4\") " pod="openshift-console/console-676875fdf9-drjfl"
May 25 11:18:55.937473 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:55.937452    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xrlp\" (UniqueName: \"kubernetes.io/projected/21962877-1f1b-4993-8c1f-56042776afb4-kube-api-access-4xrlp\") pod \"console-676875fdf9-drjfl\" (UID: \"21962877-1f1b-4993-8c1f-56042776afb4\") " pod="openshift-console/console-676875fdf9-drjfl"
May 25 11:18:56.093528 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:56.093498    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-676875fdf9-drjfl"
May 25 11:18:56.218300 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:56.218268    2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-676875fdf9-drjfl"]
May 25 11:18:56.221139 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:18:56.221107    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21962877_1f1b_4993_8c1f_56042776afb4.slice/crio-b9fb781582f01c7951f89850bd675ad0d7ddabc5ca7d31888aa77432e4d984a2 WatchSource:0}: Error finding container b9fb781582f01c7951f89850bd675ad0d7ddabc5ca7d31888aa77432e4d984a2: Status 404 returned error can't find the container with id b9fb781582f01c7951f89850bd675ad0d7ddabc5ca7d31888aa77432e4d984a2
May 25 11:18:56.298679 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:56.298646    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-676875fdf9-drjfl" event={"ID":"21962877-1f1b-4993-8c1f-56042776afb4","Type":"ContainerStarted","Data":"b9fb781582f01c7951f89850bd675ad0d7ddabc5ca7d31888aa77432e4d984a2"}
May 25 11:18:59.309524 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:59.309485    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-676875fdf9-drjfl" event={"ID":"21962877-1f1b-4993-8c1f-56042776afb4","Type":"ContainerStarted","Data":"6083a1027f825663cff3884cd6c154397353b0f2d9eabf8e99ec0996f1e42622"}
May 25 11:18:59.329356 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:18:59.329306    2560 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console/console-676875fdf9-drjfl" podStartSLOduration=1.911609908 podStartE2EDuration="4.329291502s" podCreationTimestamp="2026-05-25 11:18:55 +0000 UTC" firstStartedPulling="2026-05-25 11:18:56.223099741 +0000 UTC m=+77.904077692" lastFinishedPulling="2026-05-25 11:18:58.640781335 +0000 UTC m=+80.321759286" observedRunningTime="2026-05-25 11:18:59.328863053 +0000 UTC m=+81.009841028" watchObservedRunningTime="2026-05-25 11:18:59.329291502 +0000 UTC m=+81.010269474"
May 25 11:19:02.292077 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:19:02.292044    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-vvfw5"
May 25 11:19:06.094579 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:19:06.094544    2560 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-676875fdf9-drjfl"
May 25 11:19:06.095012 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:19:06.094591    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-676875fdf9-drjfl"
May 25 11:19:06.100707 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:19:06.100682    2560 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-676875fdf9-drjfl"
May 25 11:19:06.247710 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:19:06.247682    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5f8d6ccbdc-8pc8t"
May 25 11:19:06.333874 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:19:06.333844    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-676875fdf9-drjfl"
May 25 11:19:06.956386 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:19:06.956356    2560 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5f8d6ccbdc-8pc8t"]
May 25 11:19:07.151723 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:19:07.151692    2560 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-9c959559f-x8bkd"]
May 25 11:19:07.156125 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:19:07.156104    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-9c959559f-x8bkd"
May 25 11:19:07.159335 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:19:07.159309    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
May 25 11:19:07.159461 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:19:07.159309    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-tf92g\""
May 25 11:19:07.159461 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:19:07.159351    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
May 25 11:19:07.159547 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:19:07.159530    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
May 25 11:19:07.176470 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:19:07.176436    2560 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-rk5fb"]
May 25 11:19:07.179658 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:19:07.179638    2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-9c959559f-x8bkd"]
May 25 11:19:07.179769 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:19:07.179753    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-rk5fb"
May 25 11:19:07.182497 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:19:07.182465    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
May 25 11:19:07.182610 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:19:07.182466    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
May 25 11:19:07.182677 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:19:07.182640    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-qlxnt\""
May 25 11:19:07.182760 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:19:07.182741    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
May 25 11:19:07.226464 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:19:07.226386    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/1b402763-7a26-428e-8165-0248e508c1f5-openshift-state-metrics-tls\") pod \"openshift-state-metrics-9c959559f-x8bkd\" (UID: \"1b402763-7a26-428e-8165-0248e508c1f5\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-x8bkd"
May 25 11:19:07.226464 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:19:07.226426    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tq68\" (UniqueName: \"kubernetes.io/projected/1b402763-7a26-428e-8165-0248e508c1f5-kube-api-access-2tq68\") pod \"openshift-state-metrics-9c959559f-x8bkd\" (UID: \"1b402763-7a26-428e-8165-0248e508c1f5\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-x8bkd"
May 25 11:19:07.226464 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:19:07.226450    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1b402763-7a26-428e-8165-0248e508c1f5-metrics-client-ca\") pod \"openshift-state-metrics-9c959559f-x8bkd\" (UID: \"1b402763-7a26-428e-8165-0248e508c1f5\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-x8bkd"
May 25 11:19:07.226652 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:19:07.226523    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1b402763-7a26-428e-8165-0248e508c1f5-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-9c959559f-x8bkd\" (UID: \"1b402763-7a26-428e-8165-0248e508c1f5\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-x8bkd"
May 25 11:19:07.327095 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:19:07.327067    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/1b402763-7a26-428e-8165-0248e508c1f5-openshift-state-metrics-tls\") pod \"openshift-state-metrics-9c959559f-x8bkd\" (UID: \"1b402763-7a26-428e-8165-0248e508c1f5\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-x8bkd"
May 25 11:19:07.327095 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:19:07.327099    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6089fc6e-f837-4d7f-ae29-cc5588936435-sys\") pod \"node-exporter-rk5fb\" (UID: \"6089fc6e-f837-4d7f-ae29-cc5588936435\") " pod="openshift-monitoring/node-exporter-rk5fb"
May 25 11:19:07.327319 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:19:07.327120    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5xkk\" (UniqueName: \"kubernetes.io/projected/6089fc6e-f837-4d7f-ae29-cc5588936435-kube-api-access-p5xkk\") pod \"node-exporter-rk5fb\" (UID: \"6089fc6e-f837-4d7f-ae29-cc5588936435\") " pod="openshift-monitoring/node-exporter-rk5fb"
May 25 11:19:07.327319 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:19:07.327150    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-2tq68\" (UniqueName: \"kubernetes.io/projected/1b402763-7a26-428e-8165-0248e508c1f5-kube-api-access-2tq68\") pod \"openshift-state-metrics-9c959559f-x8bkd\" (UID: \"1b402763-7a26-428e-8165-0248e508c1f5\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-x8bkd"
May 25 11:19:07.327319 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:19:07.327239    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6089fc6e-f837-4d7f-ae29-cc5588936435-node-exporter-accelerators-collector-config\") pod \"node-exporter-rk5fb\" (UID: \"6089fc6e-f837-4d7f-ae29-cc5588936435\") " pod="openshift-monitoring/node-exporter-rk5fb"
May 25 11:19:07.327319 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:19:07.327286    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1b402763-7a26-428e-8165-0248e508c1f5-metrics-client-ca\") pod \"openshift-state-metrics-9c959559f-x8bkd\" (UID: \"1b402763-7a26-428e-8165-0248e508c1f5\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-x8bkd"
May 25 11:19:07.327470 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:19:07.327330    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6089fc6e-f837-4d7f-ae29-cc5588936435-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rk5fb\" (UID: \"6089fc6e-f837-4d7f-ae29-cc5588936435\") " pod="openshift-monitoring/node-exporter-rk5fb"
May 25 11:19:07.327470 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:19:07.327370    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6089fc6e-f837-4d7f-ae29-cc5588936435-node-exporter-textfile\") pod \"node-exporter-rk5fb\" (UID: \"6089fc6e-f837-4d7f-ae29-cc5588936435\") " pod="openshift-monitoring/node-exporter-rk5fb"
May 25 11:19:07.327470 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:19:07.327435    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1b402763-7a26-428e-8165-0248e508c1f5-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-9c959559f-x8bkd\" (UID: \"1b402763-7a26-428e-8165-0248e508c1f5\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-x8bkd"
May 25 11:19:07.327613 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:19:07.327475    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6089fc6e-f837-4d7f-ae29-cc5588936435-node-exporter-tls\") pod \"node-exporter-rk5fb\" (UID: \"6089fc6e-f837-4d7f-ae29-cc5588936435\") " pod="openshift-monitoring/node-exporter-rk5fb"
May 25 11:19:07.327613 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:19:07.327499    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6089fc6e-f837-4d7f-ae29-cc5588936435-node-exporter-wtmp\") pod \"node-exporter-rk5fb\" (UID: \"6089fc6e-f837-4d7f-ae29-cc5588936435\") " pod="openshift-monitoring/node-exporter-rk5fb"
May 25 11:19:07.327613 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:19:07.327542    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6089fc6e-f837-4d7f-ae29-cc5588936435-root\") pod \"node-exporter-rk5fb\" (UID: \"6089fc6e-f837-4d7f-ae29-cc5588936435\") " pod="openshift-monitoring/node-exporter-rk5fb"
May 25 11:19:07.327613 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:19:07.327597    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6089fc6e-f837-4d7f-ae29-cc5588936435-metrics-client-ca\") pod \"node-exporter-rk5fb\" (UID: \"6089fc6e-f837-4d7f-ae29-cc5588936435\") " pod="openshift-monitoring/node-exporter-rk5fb"
May 25 11:19:07.327873 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:19:07.327855    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1b402763-7a26-428e-8165-0248e508c1f5-metrics-client-ca\") pod \"openshift-state-metrics-9c959559f-x8bkd\" (UID: \"1b402763-7a26-428e-8165-0248e508c1f5\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-x8bkd"
May 25 11:19:07.329788 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:19:07.329763    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/1b402763-7a26-428e-8165-0248e508c1f5-openshift-state-metrics-tls\") pod \"openshift-state-metrics-9c959559f-x8bkd\" (UID: \"1b402763-7a26-428e-8165-0248e508c1f5\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-x8bkd"
May 25 11:19:07.329884 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:19:07.329792    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1b402763-7a26-428e-8165-0248e508c1f5-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-9c959559f-x8bkd\" (UID: \"1b402763-7a26-428e-8165-0248e508c1f5\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-x8bkd"
May 25 11:19:07.334753 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:19:07.334734    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tq68\" (UniqueName: \"kubernetes.io/projected/1b402763-7a26-428e-8165-0248e508c1f5-kube-api-access-2tq68\") pod \"openshift-state-metrics-9c959559f-x8bkd\" (UID: \"1b402763-7a26-428e-8165-0248e508c1f5\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-x8bkd"
May 25 11:19:07.428416 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:19:07.428384    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6089fc6e-f837-4d7f-ae29-cc5588936435-metrics-client-ca\") pod \"node-exporter-rk5fb\" (UID: \"6089fc6e-f837-4d7f-ae29-cc5588936435\") " pod="openshift-monitoring/node-exporter-rk5fb"
May 25 11:19:07.428569 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:19:07.428432    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6089fc6e-f837-4d7f-ae29-cc5588936435-sys\") pod \"node-exporter-rk5fb\" (UID: \"6089fc6e-f837-4d7f-ae29-cc5588936435\") " pod="openshift-monitoring/node-exporter-rk5fb"
May 25 11:19:07.428569 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:19:07.428470    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-p5xkk\" (UniqueName: \"kubernetes.io/projected/6089fc6e-f837-4d7f-ae29-cc5588936435-kube-api-access-p5xkk\") pod \"node-exporter-rk5fb\" (UID: \"6089fc6e-f837-4d7f-ae29-cc5588936435\") " pod="openshift-monitoring/node-exporter-rk5fb"
May 25 11:19:07.428569 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:19:07.428499    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6089fc6e-f837-4d7f-ae29-cc5588936435-sys\") pod \"node-exporter-rk5fb\" (UID: \"6089fc6e-f837-4d7f-ae29-cc5588936435\") " pod="openshift-monitoring/node-exporter-rk5fb"
May 25 11:19:07.428569 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:19:07.428509    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6089fc6e-f837-4d7f-ae29-cc5588936435-node-exporter-accelerators-collector-config\") pod \"node-exporter-rk5fb\" (UID: \"6089fc6e-f837-4d7f-ae29-cc5588936435\") " pod="openshift-monitoring/node-exporter-rk5fb"
May 25 11:19:07.428766 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:19:07.428609    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6089fc6e-f837-4d7f-ae29-cc5588936435-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rk5fb\" (UID: \"6089fc6e-f837-4d7f-ae29-cc5588936435\") " pod="openshift-monitoring/node-exporter-rk5fb"
May 25 11:19:07.428828 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:19:07.428798    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6089fc6e-f837-4d7f-ae29-cc5588936435-node-exporter-textfile\") pod \"node-exporter-rk5fb\" (UID: \"6089fc6e-f837-4d7f-ae29-cc5588936435\") " pod="openshift-monitoring/node-exporter-rk5fb"
May 25 11:19:07.428885 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:19:07.428838    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6089fc6e-f837-4d7f-ae29-cc5588936435-node-exporter-tls\") pod \"node-exporter-rk5fb\" (UID: \"6089fc6e-f837-4d7f-ae29-cc5588936435\") " pod="openshift-monitoring/node-exporter-rk5fb"
May 25 11:19:07.428938 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:19:07.428884    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6089fc6e-f837-4d7f-ae29-cc5588936435-node-exporter-wtmp\") pod \"node-exporter-rk5fb\" (UID: \"6089fc6e-f837-4d7f-ae29-cc5588936435\") " pod="openshift-monitoring/node-exporter-rk5fb"
May 25 11:19:07.428938 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:19:07.428926    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6089fc6e-f837-4d7f-ae29-cc5588936435-root\") pod \"node-exporter-rk5fb\" (UID: \"6089fc6e-f837-4d7f-ae29-cc5588936435\") " pod="openshift-monitoring/node-exporter-rk5fb"
May 25 11:19:07.429236 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:19:07.429188    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6089fc6e-f837-4d7f-ae29-cc5588936435-node-exporter-wtmp\") pod \"node-exporter-rk5fb\" (UID: \"6089fc6e-f837-4d7f-ae29-cc5588936435\") " pod="openshift-monitoring/node-exporter-rk5fb"
May 25 11:19:07.429418 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:19:07.429249    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6089fc6e-f837-4d7f-ae29-cc5588936435-metrics-client-ca\") pod \"node-exporter-rk5fb\" (UID: \"6089fc6e-f837-4d7f-ae29-cc5588936435\") " pod="openshift-monitoring/node-exporter-rk5fb"
May 25 11:19:07.429493 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:19:07.429293    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6089fc6e-f837-4d7f-ae29-cc5588936435-node-exporter-accelerators-collector-config\") pod \"node-exporter-rk5fb\" (UID: \"6089fc6e-f837-4d7f-ae29-cc5588936435\") " pod="openshift-monitoring/node-exporter-rk5fb"
May 25 11:19:07.429493 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:19:07.429312    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6089fc6e-f837-4d7f-ae29-cc5588936435-root\") pod \"node-exporter-rk5fb\" (UID: \"6089fc6e-f837-4d7f-ae29-cc5588936435\") " pod="openshift-monitoring/node-exporter-rk5fb"
May 25 11:19:07.429493 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:19:07.429462    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6089fc6e-f837-4d7f-ae29-cc5588936435-node-exporter-textfile\") pod \"node-exporter-rk5fb\" (UID: \"6089fc6e-f837-4d7f-ae29-cc5588936435\") " pod="openshift-monitoring/node-exporter-rk5fb"
May 25 11:19:07.430961 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:19:07.430937    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6089fc6e-f837-4d7f-ae29-cc5588936435-node-exporter-tls\") pod \"node-exporter-rk5fb\" (UID: \"6089fc6e-f837-4d7f-ae29-cc5588936435\") " pod="openshift-monitoring/node-exporter-rk5fb"
May 25 11:19:07.431059 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:19:07.431021    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6089fc6e-f837-4d7f-ae29-cc5588936435-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rk5fb\" (UID: \"6089fc6e-f837-4d7f-ae29-cc5588936435\") " pod="openshift-monitoring/node-exporter-rk5fb"
May 25 11:19:07.437054 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:19:07.437034    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5xkk\" (UniqueName: \"kubernetes.io/projected/6089fc6e-f837-4d7f-ae29-cc5588936435-kube-api-access-p5xkk\") pod \"node-exporter-rk5fb\" (UID: \"6089fc6e-f837-4d7f-ae29-cc5588936435\") " pod="openshift-monitoring/node-exporter-rk5fb"
May 25 11:19:07.464972 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:19:07.464948    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-9c959559f-x8bkd"
May 25 11:19:07.489423 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:19:07.489355    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-rk5fb"
May 25 11:19:07.499703 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:19:07.499660    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6089fc6e_f837_4d7f_ae29_cc5588936435.slice/crio-64322fcd3cce8fc008b13d34968c0a3c79ab88837c492f07aff5895c9b1612ac WatchSource:0}: Error finding container 64322fcd3cce8fc008b13d34968c0a3c79ab88837c492f07aff5895c9b1612ac: Status 404 returned error can't find the container with id 64322fcd3cce8fc008b13d34968c0a3c79ab88837c492f07aff5895c9b1612ac
May 25 11:19:07.590861 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:19:07.590817    2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-9c959559f-x8bkd"]
May 25 11:19:07.594291 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:19:07.594267    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b402763_7a26_428e_8165_0248e508c1f5.slice/crio-895ec77198fefa292be2596b361a31fa1e560a034c1b9a867cdb0fc23b36d887 WatchSource:0}: Error finding container 895ec77198fefa292be2596b361a31fa1e560a034c1b9a867cdb0fc23b36d887: Status 404 returned error can't find the container with id 895ec77198fefa292be2596b361a31fa1e560a034c1b9a867cdb0fc23b36d887
May 25 11:19:08.339572 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:19:08.339519    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rk5fb" event={"ID":"6089fc6e-f837-4d7f-ae29-cc5588936435","Type":"ContainerStarted","Data":"64322fcd3cce8fc008b13d34968c0a3c79ab88837c492f07aff5895c9b1612ac"}
May 25 11:19:08.341624 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:19:08.341598    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-9c959559f-x8bkd" event={"ID":"1b402763-7a26-428e-8165-0248e508c1f5","Type":"ContainerStarted","Data":"777ac191249a6c3254a4acaab73ad02a77fc5b932955980e6e0c5b45882159d4"}
May 25 11:19:08.341742 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:19:08.341630    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-9c959559f-x8bkd" event={"ID":"1b402763-7a26-428e-8165-0248e508c1f5","Type":"ContainerStarted","Data":"f013c8681c946ce1c7162f84c622d3c169181d6203df39a70a08da8f4f4c4a54"}
May 25 11:19:08.341742 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:19:08.341644    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-9c959559f-x8bkd" event={"ID":"1b402763-7a26-428e-8165-0248e508c1f5","Type":"ContainerStarted","Data":"895ec77198fefa292be2596b361a31fa1e560a034c1b9a867cdb0fc23b36d887"}
May 25 11:19:09.350172 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:19:09.350130    2560 generic.go:358] "Generic (PLEG): container finished" podID="6089fc6e-f837-4d7f-ae29-cc5588936435" containerID="3044f382f57cbf3e0ce29fc46636abcf01f1a5aa6de025c0be5427388704f68a" exitCode=0
May 25 11:19:09.350608 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:19:09.350227    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rk5fb" event={"ID":"6089fc6e-f837-4d7f-ae29-cc5588936435","Type":"ContainerDied","Data":"3044f382f57cbf3e0ce29fc46636abcf01f1a5aa6de025c0be5427388704f68a"}
May 25 11:19:09.352136 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:19:09.352115    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-9c959559f-x8bkd" event={"ID":"1b402763-7a26-428e-8165-0248e508c1f5","Type":"ContainerStarted","Data":"d9f4acbeb02af5d96062668147b7f2e5c1f7cfc7b668a12c2e6128c8c81febc9"}
May 25 11:19:09.385739 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:19:09.385685    2560 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-9c959559f-x8bkd" podStartSLOduration=1.333221461 podStartE2EDuration="2.385668858s" podCreationTimestamp="2026-05-25 11:19:07 +0000 UTC" firstStartedPulling="2026-05-25 11:19:07.708412244 +0000 UTC m=+89.389390194" lastFinishedPulling="2026-05-25 11:19:08.760859637 +0000 UTC m=+90.441837591" observedRunningTime="2026-05-25 11:19:09.385120123 +0000 UTC m=+91.066098098" watchObservedRunningTime="2026-05-25 11:19:09.385668858 +0000 UTC m=+91.066646832"
May 25 11:19:10.357442 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:19:10.357403    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rk5fb" event={"ID":"6089fc6e-f837-4d7f-ae29-cc5588936435","Type":"ContainerStarted","Data":"ff903397b81d943abd03e1bfb566a5ca9a15f4a8dd977657c2fe25865d5f8244"}
May 25 11:19:10.357442 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:19:10.357447    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rk5fb" event={"ID":"6089fc6e-f837-4d7f-ae29-cc5588936435","Type":"ContainerStarted","Data":"c86ece9d4465d57c01efe9f2f8d36853bdab4b8e8854cea82f776297468c5421"}
May 25 11:19:10.378353 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:19:10.378309    2560 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-rk5fb" podStartSLOduration=2.121321441 podStartE2EDuration="3.378293336s" podCreationTimestamp="2026-05-25 11:19:07 +0000 UTC" firstStartedPulling="2026-05-25 11:19:07.501720187 +0000 UTC m=+89.182698151" lastFinishedPulling="2026-05-25 11:19:08.758692095 +0000 UTC m=+90.439670046" observedRunningTime="2026-05-25 11:19:10.376663125 +0000 UTC m=+92.057641097" watchObservedRunningTime="2026-05-25 11:19:10.378293336 +0000 UTC m=+92.059271309"
May 25 11:19:11.864350 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:19:11.864313    2560 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console/console-768d57dcfd-m4gtl"]
May 25 11:19:11.868033 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:19:11.868010    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-768d57dcfd-m4gtl"
May 25 11:19:11.878579 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:19:11.878555    2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-768d57dcfd-m4gtl"]
May 25 11:19:11.973098 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:19:11.973059    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8bb8e0f0-519e-49bd-a871-e5c13624fd0a-service-ca\") pod \"console-768d57dcfd-m4gtl\" (UID: \"8bb8e0f0-519e-49bd-a871-e5c13624fd0a\") " pod="openshift-console/console-768d57dcfd-m4gtl"
May 25 11:19:11.973098 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:19:11.973102    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8bb8e0f0-519e-49bd-a871-e5c13624fd0a-console-config\") pod \"console-768d57dcfd-m4gtl\" (UID: \"8bb8e0f0-519e-49bd-a871-e5c13624fd0a\") " pod="openshift-console/console-768d57dcfd-m4gtl"
May 25 11:19:11.973375 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:19:11.973121    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8bb8e0f0-519e-49bd-a871-e5c13624fd0a-oauth-serving-cert\") pod \"console-768d57dcfd-m4gtl\" (UID: \"8bb8e0f0-519e-49bd-a871-e5c13624fd0a\") " pod="openshift-console/console-768d57dcfd-m4gtl"
May 25 11:19:11.973375 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:19:11.973146    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8bb8e0f0-519e-49bd-a871-e5c13624fd0a-console-serving-cert\") pod \"console-768d57dcfd-m4gtl\" (UID: \"8bb8e0f0-519e-49bd-a871-e5c13624fd0a\") " pod="openshift-console/console-768d57dcfd-m4gtl"
May 25 11:19:11.973375 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:19:11.973188    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8bb8e0f0-519e-49bd-a871-e5c13624fd0a-console-oauth-config\") pod \"console-768d57dcfd-m4gtl\" (UID: \"8bb8e0f0-519e-49bd-a871-e5c13624fd0a\") " pod="openshift-console/console-768d57dcfd-m4gtl"
May 25 11:19:11.973375 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:19:11.973270    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8bb8e0f0-519e-49bd-a871-e5c13624fd0a-trusted-ca-bundle\") pod \"console-768d57dcfd-m4gtl\" (UID: \"8bb8e0f0-519e-49bd-a871-e5c13624fd0a\") " pod="openshift-console/console-768d57dcfd-m4gtl"
May 25 11:19:11.973375 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:19:11.973309    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r26kt\" (UniqueName: \"kubernetes.io/projected/8bb8e0f0-519e-49bd-a871-e5c13624fd0a-kube-api-access-r26kt\") pod \"console-768d57dcfd-m4gtl\" (UID: \"8bb8e0f0-519e-49bd-a871-e5c13624fd0a\") " pod="openshift-console/console-768d57dcfd-m4gtl"
May 25 11:19:12.074318 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:19:12.074283    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8bb8e0f0-519e-49bd-a871-e5c13624fd0a-console-oauth-config\") pod \"console-768d57dcfd-m4gtl\" (UID: \"8bb8e0f0-519e-49bd-a871-e5c13624fd0a\") " pod="openshift-console/console-768d57dcfd-m4gtl"
May 25 11:19:12.074318 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:19:12.074324    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8bb8e0f0-519e-49bd-a871-e5c13624fd0a-trusted-ca-bundle\") pod \"console-768d57dcfd-m4gtl\" (UID: \"8bb8e0f0-519e-49bd-a871-e5c13624fd0a\") " pod="openshift-console/console-768d57dcfd-m4gtl"
May 25 11:19:12.074540 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:19:12.074440    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-r26kt\" (UniqueName: \"kubernetes.io/projected/8bb8e0f0-519e-49bd-a871-e5c13624fd0a-kube-api-access-r26kt\") pod \"console-768d57dcfd-m4gtl\" (UID: \"8bb8e0f0-519e-49bd-a871-e5c13624fd0a\") " pod="openshift-console/console-768d57dcfd-m4gtl"
May 25 11:19:12.074540 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:19:12.074484    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8bb8e0f0-519e-49bd-a871-e5c13624fd0a-service-ca\") pod \"console-768d57dcfd-m4gtl\" (UID: \"8bb8e0f0-519e-49bd-a871-e5c13624fd0a\") " pod="openshift-console/console-768d57dcfd-m4gtl"
May 25 11:19:12.074540 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:19:12.074508    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8bb8e0f0-519e-49bd-a871-e5c13624fd0a-console-config\") pod \"console-768d57dcfd-m4gtl\" (UID: \"8bb8e0f0-519e-49bd-a871-e5c13624fd0a\") " pod="openshift-console/console-768d57dcfd-m4gtl"
May 25 11:19:12.074540 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:19:12.074525    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8bb8e0f0-519e-49bd-a871-e5c13624fd0a-oauth-serving-cert\") pod \"console-768d57dcfd-m4gtl\" (UID: \"8bb8e0f0-519e-49bd-a871-e5c13624fd0a\") " pod="openshift-console/console-768d57dcfd-m4gtl"
May 25 11:19:12.074739 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:19:12.074554    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8bb8e0f0-519e-49bd-a871-e5c13624fd0a-console-serving-cert\") pod \"console-768d57dcfd-m4gtl\" (UID: \"8bb8e0f0-519e-49bd-a871-e5c13624fd0a\") " pod="openshift-console/console-768d57dcfd-m4gtl"
May 25 11:19:12.075256 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:19:12.075200    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8bb8e0f0-519e-49bd-a871-e5c13624fd0a-console-config\") pod \"console-768d57dcfd-m4gtl\" (UID: \"8bb8e0f0-519e-49bd-a871-e5c13624fd0a\") " pod="openshift-console/console-768d57dcfd-m4gtl"
May 25 11:19:12.075367 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:19:12.075307    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8bb8e0f0-519e-49bd-a871-e5c13624fd0a-trusted-ca-bundle\") pod \"console-768d57dcfd-m4gtl\" (UID: \"8bb8e0f0-519e-49bd-a871-e5c13624fd0a\") " pod="openshift-console/console-768d57dcfd-m4gtl"
May 25 11:19:12.075367 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:19:12.075331    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8bb8e0f0-519e-49bd-a871-e5c13624fd0a-oauth-serving-cert\") pod \"console-768d57dcfd-m4gtl\" (UID: \"8bb8e0f0-519e-49bd-a871-e5c13624fd0a\") " pod="openshift-console/console-768d57dcfd-m4gtl"
May 25 11:19:12.075450 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:19:12.075382    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8bb8e0f0-519e-49bd-a871-e5c13624fd0a-service-ca\") pod \"console-768d57dcfd-m4gtl\" (UID: \"8bb8e0f0-519e-49bd-a871-e5c13624fd0a\") " pod="openshift-console/console-768d57dcfd-m4gtl"
May 25 11:19:12.076765 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:19:12.076742    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8bb8e0f0-519e-49bd-a871-e5c13624fd0a-console-oauth-config\") pod \"console-768d57dcfd-m4gtl\" (UID: \"8bb8e0f0-519e-49bd-a871-e5c13624fd0a\") " pod="openshift-console/console-768d57dcfd-m4gtl"
May 25 11:19:12.077007 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:19:12.076986    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8bb8e0f0-519e-49bd-a871-e5c13624fd0a-console-serving-cert\") pod \"console-768d57dcfd-m4gtl\" (UID: \"8bb8e0f0-519e-49bd-a871-e5c13624fd0a\") " pod="openshift-console/console-768d57dcfd-m4gtl"
May 25 11:19:12.082782 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:19:12.082759    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r26kt\" (UniqueName: \"kubernetes.io/projected/8bb8e0f0-519e-49bd-a871-e5c13624fd0a-kube-api-access-r26kt\") pod \"console-768d57dcfd-m4gtl\" (UID: \"8bb8e0f0-519e-49bd-a871-e5c13624fd0a\") " pod="openshift-console/console-768d57dcfd-m4gtl"
May 25 11:19:12.178968 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:19:12.178872    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-768d57dcfd-m4gtl"
May 25 11:19:12.305435 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:19:12.305396    2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-768d57dcfd-m4gtl"]
May 25 11:19:12.307504 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:19:12.307467    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bb8e0f0_519e_49bd_a871_e5c13624fd0a.slice/crio-a5e87bb4fc5ba480dce1308d092b35a4af103d9e93279c54a00d43635139d490 WatchSource:0}: Error finding container a5e87bb4fc5ba480dce1308d092b35a4af103d9e93279c54a00d43635139d490: Status 404 returned error can't find the container with id a5e87bb4fc5ba480dce1308d092b35a4af103d9e93279c54a00d43635139d490
May 25 11:19:12.364868 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:19:12.364839    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-768d57dcfd-m4gtl" event={"ID":"8bb8e0f0-519e-49bd-a871-e5c13624fd0a","Type":"ContainerStarted","Data":"a5e87bb4fc5ba480dce1308d092b35a4af103d9e93279c54a00d43635139d490"}
May 25 11:19:13.368936 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:19:13.368897    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-768d57dcfd-m4gtl" event={"ID":"8bb8e0f0-519e-49bd-a871-e5c13624fd0a","Type":"ContainerStarted","Data":"ae662f2d9abff807a13021aa500788d9ec847dc6df7428eccec6f97230aabe93"}
May 25 11:19:13.388128 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:19:13.388073    2560 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console/console-768d57dcfd-m4gtl" podStartSLOduration=2.388048459 podStartE2EDuration="2.388048459s" podCreationTimestamp="2026-05-25 11:19:11 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 11:19:13.386466502 +0000 UTC m=+95.067444474" watchObservedRunningTime="2026-05-25 11:19:13.388048459 +0000 UTC m=+95.069026432"
May 25 11:19:22.179886 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:19:22.179849    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-768d57dcfd-m4gtl"
May 25 11:19:22.180293 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:19:22.179901    2560 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-768d57dcfd-m4gtl"
May 25 11:19:22.184673 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:19:22.184651    2560 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-768d57dcfd-m4gtl"
May 25 11:19:22.396549 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:19:22.396517    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-768d57dcfd-m4gtl"
May 25 11:19:22.446444 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:19:22.446371    2560 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-676875fdf9-drjfl"]
May 25 11:19:23.292459 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:19:23.292422    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-zfljp"
May 25 11:19:28.411860 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:19:28.411770    2560 generic.go:358] "Generic (PLEG): container finished" podID="9e082924-7517-49a1-9595-66c85246eec3" containerID="b24a628eb96f5714b757690e23b7e57b5932dbad59c552772d756a4e2a61f34b" exitCode=0
May 25 11:19:28.411860 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:19:28.411849    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-76ldn" event={"ID":"9e082924-7517-49a1-9595-66c85246eec3","Type":"ContainerDied","Data":"b24a628eb96f5714b757690e23b7e57b5932dbad59c552772d756a4e2a61f34b"}
May 25 11:19:28.412444 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:19:28.412234    2560 scope.go:117] "RemoveContainer" containerID="b24a628eb96f5714b757690e23b7e57b5932dbad59c552772d756a4e2a61f34b"
May 25 11:19:28.413260 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:19:28.413239    2560 generic.go:358] "Generic (PLEG): container finished" podID="96664b24-d104-4b57-83e8-63b8c9b3212b" containerID="ea9059a63d70d855295b0a5f9735df789e0aeabb6c967b94bdde4baa4768ea22" exitCode=0
May 25 11:19:28.413377 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:19:28.413315    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-fcqbz" event={"ID":"96664b24-d104-4b57-83e8-63b8c9b3212b","Type":"ContainerDied","Data":"ea9059a63d70d855295b0a5f9735df789e0aeabb6c967b94bdde4baa4768ea22"}
May 25 11:19:28.413612 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:19:28.413591    2560 scope.go:117] "RemoveContainer" containerID="ea9059a63d70d855295b0a5f9735df789e0aeabb6c967b94bdde4baa4768ea22"
May 25 11:19:28.414788 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:19:28.414739    2560 generic.go:358] "Generic (PLEG): container finished" podID="3ca8ede5-d33b-4a16-bd65-73ec9f2a2735" containerID="36e1f07b89fca4dfd34ec092594ed7be7bbc1ed67be2134cfd9b8ad97e7ee2d2" exitCode=0
May 25 11:19:28.414788 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:19:28.414771    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-78bdfb888c-rqcll" event={"ID":"3ca8ede5-d33b-4a16-bd65-73ec9f2a2735","Type":"ContainerDied","Data":"36e1f07b89fca4dfd34ec092594ed7be7bbc1ed67be2134cfd9b8ad97e7ee2d2"}
May 25 11:19:28.415047 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:19:28.415005    2560 scope.go:117] "RemoveContainer" containerID="36e1f07b89fca4dfd34ec092594ed7be7bbc1ed67be2134cfd9b8ad97e7ee2d2"
May 25 11:19:29.419684 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:19:29.419646    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-76ldn" event={"ID":"9e082924-7517-49a1-9595-66c85246eec3","Type":"ContainerStarted","Data":"4e025f1219b1e924819dd202e04531acc7ee3f3c9928a683c1d8b811fb991327"}
May 25 11:19:29.421465 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:19:29.421435    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-fcqbz" event={"ID":"96664b24-d104-4b57-83e8-63b8c9b3212b","Type":"ContainerStarted","Data":"0e2d3e91becd6e8d4437f03f38ccd0f6d7bedd3dd82a9ddb905ce5f53ba4f73a"}
May 25 11:19:29.422945 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:19:29.422926    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-78bdfb888c-rqcll" event={"ID":"3ca8ede5-d33b-4a16-bd65-73ec9f2a2735","Type":"ContainerStarted","Data":"76226c21101a9b040474e7fc03ef3212d5516a715a7eef57bea1da051f0e1bae"}
May 25 11:19:31.975937 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:19:31.975866    2560 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-image-registry/image-registry-5f8d6ccbdc-8pc8t" podUID="01fa335c-4082-46b1-b059-8ae4c68ef3d3" containerName="registry" containerID="cri-o://eccd0e95681a9441d7c26eb9cc095618004f4a0f0b5118863d8c663e3d5c7e7d" gracePeriod=30
May 25 11:19:32.237739 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:19:32.237717    2560 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5f8d6ccbdc-8pc8t"
May 25 11:19:32.343576 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:19:32.343546    2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jbpn8\" (UniqueName: \"kubernetes.io/projected/01fa335c-4082-46b1-b059-8ae4c68ef3d3-kube-api-access-jbpn8\") pod \"01fa335c-4082-46b1-b059-8ae4c68ef3d3\" (UID: \"01fa335c-4082-46b1-b059-8ae4c68ef3d3\") "
May 25 11:19:32.343751 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:19:32.343599    2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/01fa335c-4082-46b1-b059-8ae4c68ef3d3-image-registry-private-configuration\") pod \"01fa335c-4082-46b1-b059-8ae4c68ef3d3\" (UID: \"01fa335c-4082-46b1-b059-8ae4c68ef3d3\") "
May 25 11:19:32.343751 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:19:32.343628    2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/01fa335c-4082-46b1-b059-8ae4c68ef3d3-bound-sa-token\") pod \"01fa335c-4082-46b1-b059-8ae4c68ef3d3\" (UID: \"01fa335c-4082-46b1-b059-8ae4c68ef3d3\") "
May 25 11:19:32.343751 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:19:32.343654    2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/01fa335c-4082-46b1-b059-8ae4c68ef3d3-ca-trust-extracted\") pod \"01fa335c-4082-46b1-b059-8ae4c68ef3d3\" (UID: \"01fa335c-4082-46b1-b059-8ae4c68ef3d3\") "
May 25 11:19:32.343751 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:19:32.343670    2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/01fa335c-4082-46b1-b059-8ae4c68ef3d3-registry-tls\") pod \"01fa335c-4082-46b1-b059-8ae4c68ef3d3\" (UID: \"01fa335c-4082-46b1-b059-8ae4c68ef3d3\") "
May 25 11:19:32.343751 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:19:32.343694    2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/01fa335c-4082-46b1-b059-8ae4c68ef3d3-trusted-ca\") pod \"01fa335c-4082-46b1-b059-8ae4c68ef3d3\" (UID: \"01fa335c-4082-46b1-b059-8ae4c68ef3d3\") "
May 25 11:19:32.343751 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:19:32.343716    2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/01fa335c-4082-46b1-b059-8ae4c68ef3d3-installation-pull-secrets\") pod \"01fa335c-4082-46b1-b059-8ae4c68ef3d3\" (UID: \"01fa335c-4082-46b1-b059-8ae4c68ef3d3\") "
May 25 11:19:32.343751 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:19:32.343740    2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/01fa335c-4082-46b1-b059-8ae4c68ef3d3-registry-certificates\") pod \"01fa335c-4082-46b1-b059-8ae4c68ef3d3\" (UID: \"01fa335c-4082-46b1-b059-8ae4c68ef3d3\") "
May 25 11:19:32.344304 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:19:32.344276    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01fa335c-4082-46b1-b059-8ae4c68ef3d3-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "01fa335c-4082-46b1-b059-8ae4c68ef3d3" (UID: "01fa335c-4082-46b1-b059-8ae4c68ef3d3"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 11:19:32.344434 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:19:32.344322    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01fa335c-4082-46b1-b059-8ae4c68ef3d3-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "01fa335c-4082-46b1-b059-8ae4c68ef3d3" (UID: "01fa335c-4082-46b1-b059-8ae4c68ef3d3"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 11:19:32.346180 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:19:32.346148    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01fa335c-4082-46b1-b059-8ae4c68ef3d3-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "01fa335c-4082-46b1-b059-8ae4c68ef3d3" (UID: "01fa335c-4082-46b1-b059-8ae4c68ef3d3"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 11:19:32.346322 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:19:32.346299    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01fa335c-4082-46b1-b059-8ae4c68ef3d3-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "01fa335c-4082-46b1-b059-8ae4c68ef3d3" (UID: "01fa335c-4082-46b1-b059-8ae4c68ef3d3"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 11:19:32.346400 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:19:32.346385    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01fa335c-4082-46b1-b059-8ae4c68ef3d3-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "01fa335c-4082-46b1-b059-8ae4c68ef3d3" (UID: "01fa335c-4082-46b1-b059-8ae4c68ef3d3"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 11:19:32.346446 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:19:32.346382    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01fa335c-4082-46b1-b059-8ae4c68ef3d3-kube-api-access-jbpn8" (OuterVolumeSpecName: "kube-api-access-jbpn8") pod "01fa335c-4082-46b1-b059-8ae4c68ef3d3" (UID: "01fa335c-4082-46b1-b059-8ae4c68ef3d3"). InnerVolumeSpecName "kube-api-access-jbpn8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 11:19:32.346631 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:19:32.346610    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01fa335c-4082-46b1-b059-8ae4c68ef3d3-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "01fa335c-4082-46b1-b059-8ae4c68ef3d3" (UID: "01fa335c-4082-46b1-b059-8ae4c68ef3d3"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 11:19:32.352681 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:19:32.352658    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01fa335c-4082-46b1-b059-8ae4c68ef3d3-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "01fa335c-4082-46b1-b059-8ae4c68ef3d3" (UID: "01fa335c-4082-46b1-b059-8ae4c68ef3d3"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:19:32.433228 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:19:32.433180    2560 generic.go:358] "Generic (PLEG): container finished" podID="01fa335c-4082-46b1-b059-8ae4c68ef3d3" containerID="eccd0e95681a9441d7c26eb9cc095618004f4a0f0b5118863d8c663e3d5c7e7d" exitCode=0
May 25 11:19:32.433384 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:19:32.433251    2560 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5f8d6ccbdc-8pc8t"
May 25 11:19:32.433384 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:19:32.433244    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5f8d6ccbdc-8pc8t" event={"ID":"01fa335c-4082-46b1-b059-8ae4c68ef3d3","Type":"ContainerDied","Data":"eccd0e95681a9441d7c26eb9cc095618004f4a0f0b5118863d8c663e3d5c7e7d"}
May 25 11:19:32.433384 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:19:32.433360    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5f8d6ccbdc-8pc8t" event={"ID":"01fa335c-4082-46b1-b059-8ae4c68ef3d3","Type":"ContainerDied","Data":"77230f888307d67a7adbb07402e93a8ec501d56d81822bb6ce683101ee1cbce1"}
May 25 11:19:32.433384 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:19:32.433382    2560 scope.go:117] "RemoveContainer" containerID="eccd0e95681a9441d7c26eb9cc095618004f4a0f0b5118863d8c663e3d5c7e7d"
May 25 11:19:32.442201 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:19:32.442186    2560 scope.go:117] "RemoveContainer" containerID="eccd0e95681a9441d7c26eb9cc095618004f4a0f0b5118863d8c663e3d5c7e7d"
May 25 11:19:32.442490 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:19:32.442470    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eccd0e95681a9441d7c26eb9cc095618004f4a0f0b5118863d8c663e3d5c7e7d\": container with ID starting with eccd0e95681a9441d7c26eb9cc095618004f4a0f0b5118863d8c663e3d5c7e7d not found: ID does not exist" containerID="eccd0e95681a9441d7c26eb9cc095618004f4a0f0b5118863d8c663e3d5c7e7d"
May 25 11:19:32.442544 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:19:32.442503    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eccd0e95681a9441d7c26eb9cc095618004f4a0f0b5118863d8c663e3d5c7e7d"} err="failed to get container status \"eccd0e95681a9441d7c26eb9cc095618004f4a0f0b5118863d8c663e3d5c7e7d\": rpc error: code = NotFound desc = could not find container \"eccd0e95681a9441d7c26eb9cc095618004f4a0f0b5118863d8c663e3d5c7e7d\": container with ID starting with eccd0e95681a9441d7c26eb9cc095618004f4a0f0b5118863d8c663e3d5c7e7d not found: ID does not exist"
May 25 11:19:32.444944 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:19:32.444926    2560 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/01fa335c-4082-46b1-b059-8ae4c68ef3d3-ca-trust-extracted\") on node \"ip-10-0-128-146.ec2.internal\" DevicePath \"\""
May 25 11:19:32.445041 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:19:32.444949    2560 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/01fa335c-4082-46b1-b059-8ae4c68ef3d3-registry-tls\") on node \"ip-10-0-128-146.ec2.internal\" DevicePath \"\""
May 25 11:19:32.445041 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:19:32.444964    2560 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/01fa335c-4082-46b1-b059-8ae4c68ef3d3-trusted-ca\") on node \"ip-10-0-128-146.ec2.internal\" DevicePath \"\""
May 25 11:19:32.445041 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:19:32.444977    2560 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/01fa335c-4082-46b1-b059-8ae4c68ef3d3-installation-pull-secrets\") on node \"ip-10-0-128-146.ec2.internal\" DevicePath \"\""
May 25 11:19:32.445041 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:19:32.444991    2560 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/01fa335c-4082-46b1-b059-8ae4c68ef3d3-registry-certificates\") on node \"ip-10-0-128-146.ec2.internal\" DevicePath \"\""
May 25 11:19:32.445041 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:19:32.445005    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jbpn8\" (UniqueName: \"kubernetes.io/projected/01fa335c-4082-46b1-b059-8ae4c68ef3d3-kube-api-access-jbpn8\") on node \"ip-10-0-128-146.ec2.internal\" DevicePath \"\""
May 25 11:19:32.445041 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:19:32.445020    2560 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/01fa335c-4082-46b1-b059-8ae4c68ef3d3-image-registry-private-configuration\") on node \"ip-10-0-128-146.ec2.internal\" DevicePath \"\""
May 25 11:19:32.445041 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:19:32.445033    2560 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/01fa335c-4082-46b1-b059-8ae4c68ef3d3-bound-sa-token\") on node \"ip-10-0-128-146.ec2.internal\" DevicePath \"\""
May 25 11:19:32.466992 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:19:32.466962    2560 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5f8d6ccbdc-8pc8t"]
May 25 11:19:32.471952 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:19:32.471932    2560 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-5f8d6ccbdc-8pc8t"]
May 25 11:19:32.935681 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:19:32.935648    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="01fa335c-4082-46b1-b059-8ae4c68ef3d3" path="/var/lib/kubelet/pods/01fa335c-4082-46b1-b059-8ae4c68ef3d3/volumes"
May 25 11:19:47.470100 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:19:47.470037    2560 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-console/console-676875fdf9-drjfl" podUID="21962877-1f1b-4993-8c1f-56042776afb4" containerName="console" containerID="cri-o://6083a1027f825663cff3884cd6c154397353b0f2d9eabf8e99ec0996f1e42622" gracePeriod=15
May 25 11:19:47.715263 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:19:47.715242    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-676875fdf9-drjfl_21962877-1f1b-4993-8c1f-56042776afb4/console/0.log"
May 25 11:19:47.715378 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:19:47.715302    2560 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-676875fdf9-drjfl"
May 25 11:19:47.771055 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:19:47.770976    2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4xrlp\" (UniqueName: \"kubernetes.io/projected/21962877-1f1b-4993-8c1f-56042776afb4-kube-api-access-4xrlp\") pod \"21962877-1f1b-4993-8c1f-56042776afb4\" (UID: \"21962877-1f1b-4993-8c1f-56042776afb4\") "
May 25 11:19:47.771055 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:19:47.771014    2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/21962877-1f1b-4993-8c1f-56042776afb4-trusted-ca-bundle\") pod \"21962877-1f1b-4993-8c1f-56042776afb4\" (UID: \"21962877-1f1b-4993-8c1f-56042776afb4\") "
May 25 11:19:47.771055 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:19:47.771042    2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/21962877-1f1b-4993-8c1f-56042776afb4-console-oauth-config\") pod \"21962877-1f1b-4993-8c1f-56042776afb4\" (UID: \"21962877-1f1b-4993-8c1f-56042776afb4\") "
May 25 11:19:47.771350 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:19:47.771060    2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/21962877-1f1b-4993-8c1f-56042776afb4-service-ca\") pod \"21962877-1f1b-4993-8c1f-56042776afb4\" (UID: \"21962877-1f1b-4993-8c1f-56042776afb4\") "
May 25 11:19:47.771350 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:19:47.771095    2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/21962877-1f1b-4993-8c1f-56042776afb4-oauth-serving-cert\") pod \"21962877-1f1b-4993-8c1f-56042776afb4\" (UID: \"21962877-1f1b-4993-8c1f-56042776afb4\") "
May 25 11:19:47.771350 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:19:47.771157    2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/21962877-1f1b-4993-8c1f-56042776afb4-console-config\") pod \"21962877-1f1b-4993-8c1f-56042776afb4\" (UID: \"21962877-1f1b-4993-8c1f-56042776afb4\") "
May 25 11:19:47.771350 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:19:47.771187    2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/21962877-1f1b-4993-8c1f-56042776afb4-console-serving-cert\") pod \"21962877-1f1b-4993-8c1f-56042776afb4\" (UID: \"21962877-1f1b-4993-8c1f-56042776afb4\") "
May 25 11:19:47.771576 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:19:47.771548    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21962877-1f1b-4993-8c1f-56042776afb4-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "21962877-1f1b-4993-8c1f-56042776afb4" (UID: "21962877-1f1b-4993-8c1f-56042776afb4"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 11:19:47.771634 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:19:47.771609    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21962877-1f1b-4993-8c1f-56042776afb4-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "21962877-1f1b-4993-8c1f-56042776afb4" (UID: "21962877-1f1b-4993-8c1f-56042776afb4"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 11:19:47.771672 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:19:47.771625    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21962877-1f1b-4993-8c1f-56042776afb4-service-ca" (OuterVolumeSpecName: "service-ca") pod "21962877-1f1b-4993-8c1f-56042776afb4" (UID: "21962877-1f1b-4993-8c1f-56042776afb4"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 11:19:47.771672 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:19:47.771613    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21962877-1f1b-4993-8c1f-56042776afb4-console-config" (OuterVolumeSpecName: "console-config") pod "21962877-1f1b-4993-8c1f-56042776afb4" (UID: "21962877-1f1b-4993-8c1f-56042776afb4"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 11:19:47.773498 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:19:47.773476    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21962877-1f1b-4993-8c1f-56042776afb4-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "21962877-1f1b-4993-8c1f-56042776afb4" (UID: "21962877-1f1b-4993-8c1f-56042776afb4"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 11:19:47.773682 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:19:47.773661    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21962877-1f1b-4993-8c1f-56042776afb4-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "21962877-1f1b-4993-8c1f-56042776afb4" (UID: "21962877-1f1b-4993-8c1f-56042776afb4"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 11:19:47.773740 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:19:47.773663    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21962877-1f1b-4993-8c1f-56042776afb4-kube-api-access-4xrlp" (OuterVolumeSpecName: "kube-api-access-4xrlp") pod "21962877-1f1b-4993-8c1f-56042776afb4" (UID: "21962877-1f1b-4993-8c1f-56042776afb4"). InnerVolumeSpecName "kube-api-access-4xrlp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 11:19:47.872042 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:19:47.872002    2560 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/21962877-1f1b-4993-8c1f-56042776afb4-oauth-serving-cert\") on node \"ip-10-0-128-146.ec2.internal\" DevicePath \"\""
May 25 11:19:47.872042 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:19:47.872034    2560 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/21962877-1f1b-4993-8c1f-56042776afb4-console-config\") on node \"ip-10-0-128-146.ec2.internal\" DevicePath \"\""
May 25 11:19:47.872042 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:19:47.872044    2560 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/21962877-1f1b-4993-8c1f-56042776afb4-console-serving-cert\") on node \"ip-10-0-128-146.ec2.internal\" DevicePath \"\""
May 25 11:19:47.872305 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:19:47.872052    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4xrlp\" (UniqueName: \"kubernetes.io/projected/21962877-1f1b-4993-8c1f-56042776afb4-kube-api-access-4xrlp\") on node \"ip-10-0-128-146.ec2.internal\" DevicePath \"\""
May 25 11:19:47.872305 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:19:47.872063    2560 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/21962877-1f1b-4993-8c1f-56042776afb4-trusted-ca-bundle\") on node \"ip-10-0-128-146.ec2.internal\" DevicePath \"\""
May 25 11:19:47.872305 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:19:47.872071    2560 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/21962877-1f1b-4993-8c1f-56042776afb4-console-oauth-config\") on node \"ip-10-0-128-146.ec2.internal\" DevicePath \"\""
May 25 11:19:47.872305 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:19:47.872081    2560 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/21962877-1f1b-4993-8c1f-56042776afb4-service-ca\") on node \"ip-10-0-128-146.ec2.internal\" DevicePath \"\""
May 25 11:19:48.482144 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:19:48.482115    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-676875fdf9-drjfl_21962877-1f1b-4993-8c1f-56042776afb4/console/0.log"
May 25 11:19:48.482551 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:19:48.482156    2560 generic.go:358] "Generic (PLEG): container finished" podID="21962877-1f1b-4993-8c1f-56042776afb4" containerID="6083a1027f825663cff3884cd6c154397353b0f2d9eabf8e99ec0996f1e42622" exitCode=2
May 25 11:19:48.482551 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:19:48.482244    2560 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-676875fdf9-drjfl"
May 25 11:19:48.482551 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:19:48.482244    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-676875fdf9-drjfl" event={"ID":"21962877-1f1b-4993-8c1f-56042776afb4","Type":"ContainerDied","Data":"6083a1027f825663cff3884cd6c154397353b0f2d9eabf8e99ec0996f1e42622"}
May 25 11:19:48.482551 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:19:48.482347    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-676875fdf9-drjfl" event={"ID":"21962877-1f1b-4993-8c1f-56042776afb4","Type":"ContainerDied","Data":"b9fb781582f01c7951f89850bd675ad0d7ddabc5ca7d31888aa77432e4d984a2"}
May 25 11:19:48.482551 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:19:48.482364    2560 scope.go:117] "RemoveContainer" containerID="6083a1027f825663cff3884cd6c154397353b0f2d9eabf8e99ec0996f1e42622"
May 25 11:19:48.491181 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:19:48.491163    2560 scope.go:117] "RemoveContainer" containerID="6083a1027f825663cff3884cd6c154397353b0f2d9eabf8e99ec0996f1e42622"
May 25 11:19:48.491450 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:19:48.491431    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6083a1027f825663cff3884cd6c154397353b0f2d9eabf8e99ec0996f1e42622\": container with ID starting with 6083a1027f825663cff3884cd6c154397353b0f2d9eabf8e99ec0996f1e42622 not found: ID does not exist" containerID="6083a1027f825663cff3884cd6c154397353b0f2d9eabf8e99ec0996f1e42622"
May 25 11:19:48.491510 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:19:48.491464    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6083a1027f825663cff3884cd6c154397353b0f2d9eabf8e99ec0996f1e42622"} err="failed to get container status \"6083a1027f825663cff3884cd6c154397353b0f2d9eabf8e99ec0996f1e42622\": rpc error: code = NotFound desc = could not find container \"6083a1027f825663cff3884cd6c154397353b0f2d9eabf8e99ec0996f1e42622\": container with ID starting with 6083a1027f825663cff3884cd6c154397353b0f2d9eabf8e99ec0996f1e42622 not found: ID does not exist"
May 25 11:19:48.504237 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:19:48.504194    2560 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-676875fdf9-drjfl"]
May 25 11:19:48.508025 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:19:48.507994    2560 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-676875fdf9-drjfl"]
May 25 11:19:48.934179 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:19:48.934148    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="21962877-1f1b-4993-8c1f-56042776afb4" path="/var/lib/kubelet/pods/21962877-1f1b-4993-8c1f-56042776afb4/volumes"
May 25 11:20:31.553984 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:20:31.553899    2560 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-68b857dfd5-xdvx6"]
May 25 11:20:31.554418 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:20:31.554197    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21962877-1f1b-4993-8c1f-56042776afb4" containerName="console"
May 25 11:20:31.554418 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:20:31.554235    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="21962877-1f1b-4993-8c1f-56042776afb4" containerName="console"
May 25 11:20:31.554418 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:20:31.554258    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01fa335c-4082-46b1-b059-8ae4c68ef3d3" containerName="registry"
May 25 11:20:31.554418 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:20:31.554267    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="01fa335c-4082-46b1-b059-8ae4c68ef3d3" containerName="registry"
May 25 11:20:31.554418 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:20:31.554323    2560 memory_manager.go:356] "RemoveStaleState removing state" podUID="21962877-1f1b-4993-8c1f-56042776afb4" containerName="console"
May 25 11:20:31.554418 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:20:31.554332    2560 memory_manager.go:356] "RemoveStaleState removing state" podUID="01fa335c-4082-46b1-b059-8ae4c68ef3d3" containerName="registry"
May 25 11:20:31.557595 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:20:31.557575    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-68b857dfd5-xdvx6"
May 25 11:20:31.560504 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:20:31.560482    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
May 25 11:20:31.560635 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:20:31.560586    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
May 25 11:20:31.560635 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:20:31.560588    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
May 25 11:20:31.560635 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:20:31.560633    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
May 25 11:20:31.561302 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:20:31.561281    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
May 25 11:20:31.561395 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:20:31.561307    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-gnc2v\""
May 25 11:20:31.566731 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:20:31.566698    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
May 25 11:20:31.568888 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:20:31.568867    2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-68b857dfd5-xdvx6"]
May 25 11:20:31.623195 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:20:31.623163    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3c3fed2e-c3d2-4e64-94a5-07eed2348adc-serving-certs-ca-bundle\") pod \"telemeter-client-68b857dfd5-xdvx6\" (UID: \"3c3fed2e-c3d2-4e64-94a5-07eed2348adc\") " pod="openshift-monitoring/telemeter-client-68b857dfd5-xdvx6"
May 25 11:20:31.623390 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:20:31.623204    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65ntc\" (UniqueName: \"kubernetes.io/projected/3c3fed2e-c3d2-4e64-94a5-07eed2348adc-kube-api-access-65ntc\") pod \"telemeter-client-68b857dfd5-xdvx6\" (UID: \"3c3fed2e-c3d2-4e64-94a5-07eed2348adc\") " pod="openshift-monitoring/telemeter-client-68b857dfd5-xdvx6"
May 25 11:20:31.623390 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:20:31.623257    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3c3fed2e-c3d2-4e64-94a5-07eed2348adc-telemeter-trusted-ca-bundle\") pod \"telemeter-client-68b857dfd5-xdvx6\" (UID: \"3c3fed2e-c3d2-4e64-94a5-07eed2348adc\") " pod="openshift-monitoring/telemeter-client-68b857dfd5-xdvx6"
May 25 11:20:31.623390 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:20:31.623275    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/3c3fed2e-c3d2-4e64-94a5-07eed2348adc-federate-client-tls\") pod \"telemeter-client-68b857dfd5-xdvx6\" (UID: \"3c3fed2e-c3d2-4e64-94a5-07eed2348adc\") " pod="openshift-monitoring/telemeter-client-68b857dfd5-xdvx6"
May 25 11:20:31.623390 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:20:31.623307    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3c3fed2e-c3d2-4e64-94a5-07eed2348adc-metrics-client-ca\") pod \"telemeter-client-68b857dfd5-xdvx6\" (UID: \"3c3fed2e-c3d2-4e64-94a5-07eed2348adc\") " pod="openshift-monitoring/telemeter-client-68b857dfd5-xdvx6"
May 25 11:20:31.623390 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:20:31.623344    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/3c3fed2e-c3d2-4e64-94a5-07eed2348adc-telemeter-client-tls\") pod \"telemeter-client-68b857dfd5-xdvx6\" (UID: \"3c3fed2e-c3d2-4e64-94a5-07eed2348adc\") " pod="openshift-monitoring/telemeter-client-68b857dfd5-xdvx6"
May 25 11:20:31.623390 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:20:31.623362    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/3c3fed2e-c3d2-4e64-94a5-07eed2348adc-secret-telemeter-client\") pod \"telemeter-client-68b857dfd5-xdvx6\" (UID: \"3c3fed2e-c3d2-4e64-94a5-07eed2348adc\") " pod="openshift-monitoring/telemeter-client-68b857dfd5-xdvx6"
May 25 11:20:31.623390 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:20:31.623380    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3c3fed2e-c3d2-4e64-94a5-07eed2348adc-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-68b857dfd5-xdvx6\" (UID: \"3c3fed2e-c3d2-4e64-94a5-07eed2348adc\") " pod="openshift-monitoring/telemeter-client-68b857dfd5-xdvx6"
May 25 11:20:31.724522 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:20:31.724486    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/3c3fed2e-c3d2-4e64-94a5-07eed2348adc-telemeter-client-tls\") pod \"telemeter-client-68b857dfd5-xdvx6\" (UID: \"3c3fed2e-c3d2-4e64-94a5-07eed2348adc\") " pod="openshift-monitoring/telemeter-client-68b857dfd5-xdvx6"
May 25 11:20:31.724686 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:20:31.724527    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/3c3fed2e-c3d2-4e64-94a5-07eed2348adc-secret-telemeter-client\") pod \"telemeter-client-68b857dfd5-xdvx6\" (UID: \"3c3fed2e-c3d2-4e64-94a5-07eed2348adc\") " pod="openshift-monitoring/telemeter-client-68b857dfd5-xdvx6"
May 25 11:20:31.724686 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:20:31.724557    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3c3fed2e-c3d2-4e64-94a5-07eed2348adc-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-68b857dfd5-xdvx6\" (UID: \"3c3fed2e-c3d2-4e64-94a5-07eed2348adc\") " pod="openshift-monitoring/telemeter-client-68b857dfd5-xdvx6"
May 25 11:20:31.724686 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:20:31.724605    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3c3fed2e-c3d2-4e64-94a5-07eed2348adc-serving-certs-ca-bundle\") pod \"telemeter-client-68b857dfd5-xdvx6\" (UID: \"3c3fed2e-c3d2-4e64-94a5-07eed2348adc\") " pod="openshift-monitoring/telemeter-client-68b857dfd5-xdvx6"
May 25 11:20:31.724686 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:20:31.724646    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-65ntc\" (UniqueName: \"kubernetes.io/projected/3c3fed2e-c3d2-4e64-94a5-07eed2348adc-kube-api-access-65ntc\") pod \"telemeter-client-68b857dfd5-xdvx6\" (UID: \"3c3fed2e-c3d2-4e64-94a5-07eed2348adc\") " pod="openshift-monitoring/telemeter-client-68b857dfd5-xdvx6"
May 25 11:20:31.724875 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:20:31.724699    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3c3fed2e-c3d2-4e64-94a5-07eed2348adc-telemeter-trusted-ca-bundle\") pod \"telemeter-client-68b857dfd5-xdvx6\" (UID: \"3c3fed2e-c3d2-4e64-94a5-07eed2348adc\") " pod="openshift-monitoring/telemeter-client-68b857dfd5-xdvx6"
May 25 11:20:31.724875 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:20:31.724732    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/3c3fed2e-c3d2-4e64-94a5-07eed2348adc-federate-client-tls\") pod \"telemeter-client-68b857dfd5-xdvx6\" (UID: \"3c3fed2e-c3d2-4e64-94a5-07eed2348adc\") " pod="openshift-monitoring/telemeter-client-68b857dfd5-xdvx6"
May 25 11:20:31.724875 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:20:31.724778    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3c3fed2e-c3d2-4e64-94a5-07eed2348adc-metrics-client-ca\") pod \"telemeter-client-68b857dfd5-xdvx6\" (UID: \"3c3fed2e-c3d2-4e64-94a5-07eed2348adc\") " pod="openshift-monitoring/telemeter-client-68b857dfd5-xdvx6"
May 25 11:20:31.725595 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:20:31.725567    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3c3fed2e-c3d2-4e64-94a5-07eed2348adc-metrics-client-ca\") pod \"telemeter-client-68b857dfd5-xdvx6\" (UID: \"3c3fed2e-c3d2-4e64-94a5-07eed2348adc\") " pod="openshift-monitoring/telemeter-client-68b857dfd5-xdvx6"
May 25 11:20:31.725752 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:20:31.725647    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3c3fed2e-c3d2-4e64-94a5-07eed2348adc-telemeter-trusted-ca-bundle\") pod \"telemeter-client-68b857dfd5-xdvx6\" (UID: \"3c3fed2e-c3d2-4e64-94a5-07eed2348adc\") " pod="openshift-monitoring/telemeter-client-68b857dfd5-xdvx6"
May 25 11:20:31.725752 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:20:31.725659    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3c3fed2e-c3d2-4e64-94a5-07eed2348adc-serving-certs-ca-bundle\") pod \"telemeter-client-68b857dfd5-xdvx6\" (UID: \"3c3fed2e-c3d2-4e64-94a5-07eed2348adc\") " pod="openshift-monitoring/telemeter-client-68b857dfd5-xdvx6"
May 25 11:20:31.727165 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:20:31.727139    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/3c3fed2e-c3d2-4e64-94a5-07eed2348adc-federate-client-tls\") pod \"telemeter-client-68b857dfd5-xdvx6\" (UID: \"3c3fed2e-c3d2-4e64-94a5-07eed2348adc\") " pod="openshift-monitoring/telemeter-client-68b857dfd5-xdvx6"
May 25 11:20:31.727317 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:20:31.727179    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3c3fed2e-c3d2-4e64-94a5-07eed2348adc-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-68b857dfd5-xdvx6\" (UID: \"3c3fed2e-c3d2-4e64-94a5-07eed2348adc\") " pod="openshift-monitoring/telemeter-client-68b857dfd5-xdvx6"
May 25 11:20:31.727380 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:20:31.727318    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/3c3fed2e-c3d2-4e64-94a5-07eed2348adc-secret-telemeter-client\") pod \"telemeter-client-68b857dfd5-xdvx6\" (UID: \"3c3fed2e-c3d2-4e64-94a5-07eed2348adc\") " pod="openshift-monitoring/telemeter-client-68b857dfd5-xdvx6"
May 25 11:20:31.727416 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:20:31.727401    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/3c3fed2e-c3d2-4e64-94a5-07eed2348adc-telemeter-client-tls\") pod \"telemeter-client-68b857dfd5-xdvx6\" (UID: \"3c3fed2e-c3d2-4e64-94a5-07eed2348adc\") " pod="openshift-monitoring/telemeter-client-68b857dfd5-xdvx6"
May 25 11:20:31.733035 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:20:31.733013    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-65ntc\" (UniqueName: \"kubernetes.io/projected/3c3fed2e-c3d2-4e64-94a5-07eed2348adc-kube-api-access-65ntc\") pod \"telemeter-client-68b857dfd5-xdvx6\" (UID: \"3c3fed2e-c3d2-4e64-94a5-07eed2348adc\") " pod="openshift-monitoring/telemeter-client-68b857dfd5-xdvx6"
May 25 11:20:31.867723 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:20:31.867682    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-68b857dfd5-xdvx6"
May 25 11:20:31.997977 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:20:31.997751    2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-68b857dfd5-xdvx6"]
May 25 11:20:32.000222 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:20:32.000181    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c3fed2e_c3d2_4e64_94a5_07eed2348adc.slice/crio-3923d584e690cf03f3e6a7ebb89f13e544e6de568531efa8e5b32387232d86ac WatchSource:0}: Error finding container 3923d584e690cf03f3e6a7ebb89f13e544e6de568531efa8e5b32387232d86ac: Status 404 returned error can't find the container with id 3923d584e690cf03f3e6a7ebb89f13e544e6de568531efa8e5b32387232d86ac
May 25 11:20:32.604669 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:20:32.604636    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-68b857dfd5-xdvx6" event={"ID":"3c3fed2e-c3d2-4e64-94a5-07eed2348adc","Type":"ContainerStarted","Data":"3923d584e690cf03f3e6a7ebb89f13e544e6de568531efa8e5b32387232d86ac"}
May 25 11:20:34.611191 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:20:34.611157    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-68b857dfd5-xdvx6" event={"ID":"3c3fed2e-c3d2-4e64-94a5-07eed2348adc","Type":"ContainerStarted","Data":"9e5fcf208e0446e7527644e9e19bbbb6d2890afaee05f9c03982d988397cee4a"}
May 25 11:20:35.616139 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:20:35.616103    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-68b857dfd5-xdvx6" event={"ID":"3c3fed2e-c3d2-4e64-94a5-07eed2348adc","Type":"ContainerStarted","Data":"a9b12aae56c0d55904eddb3185d04bb52689b6c16e53c4c9ffcc50df1e29986a"}
May 25 11:20:35.616139 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:20:35.616142    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-68b857dfd5-xdvx6" event={"ID":"3c3fed2e-c3d2-4e64-94a5-07eed2348adc","Type":"ContainerStarted","Data":"18ba2e0c8fdbcf3c93d74cb454162a5173040d4a93f58a2e01960c7429fd7f37"}
May 25 11:20:35.639830 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:20:35.639778    2560 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-68b857dfd5-xdvx6" podStartSLOduration=1.994043413 podStartE2EDuration="4.63976552s" podCreationTimestamp="2026-05-25 11:20:31 +0000 UTC" firstStartedPulling="2026-05-25 11:20:32.002017448 +0000 UTC m=+173.682995399" lastFinishedPulling="2026-05-25 11:20:34.647739539 +0000 UTC m=+176.328717506" observedRunningTime="2026-05-25 11:20:35.638469005 +0000 UTC m=+177.319446978" watchObservedRunningTime="2026-05-25 11:20:35.63976552 +0000 UTC m=+177.320743529"
May 25 11:20:36.347811 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:20:36.347780    2560 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console/console-c6655d68d-wsh5g"]
May 25 11:20:36.351259 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:20:36.351237    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c6655d68d-wsh5g"
May 25 11:20:36.362336 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:20:36.362311    2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-c6655d68d-wsh5g"]
May 25 11:20:36.466229 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:20:36.466182    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a363e903-30ca-4436-bcdf-1a541027e8d9-console-serving-cert\") pod \"console-c6655d68d-wsh5g\" (UID: \"a363e903-30ca-4436-bcdf-1a541027e8d9\") " pod="openshift-console/console-c6655d68d-wsh5g"
May 25 11:20:36.466384 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:20:36.466241    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fh6l4\" (UniqueName: \"kubernetes.io/projected/a363e903-30ca-4436-bcdf-1a541027e8d9-kube-api-access-fh6l4\") pod \"console-c6655d68d-wsh5g\" (UID: \"a363e903-30ca-4436-bcdf-1a541027e8d9\") " pod="openshift-console/console-c6655d68d-wsh5g"
May 25 11:20:36.466384 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:20:36.466291    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a363e903-30ca-4436-bcdf-1a541027e8d9-service-ca\") pod \"console-c6655d68d-wsh5g\" (UID: \"a363e903-30ca-4436-bcdf-1a541027e8d9\") " pod="openshift-console/console-c6655d68d-wsh5g"
May 25 11:20:36.466384 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:20:36.466308    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a363e903-30ca-4436-bcdf-1a541027e8d9-console-config\") pod \"console-c6655d68d-wsh5g\" (UID: \"a363e903-30ca-4436-bcdf-1a541027e8d9\") " pod="openshift-console/console-c6655d68d-wsh5g"
May 25 11:20:36.466384 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:20:36.466323    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a363e903-30ca-4436-bcdf-1a541027e8d9-trusted-ca-bundle\") pod \"console-c6655d68d-wsh5g\" (UID: \"a363e903-30ca-4436-bcdf-1a541027e8d9\") " pod="openshift-console/console-c6655d68d-wsh5g"
May 25 11:20:36.466384 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:20:36.466348    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a363e903-30ca-4436-bcdf-1a541027e8d9-console-oauth-config\") pod \"console-c6655d68d-wsh5g\" (UID: \"a363e903-30ca-4436-bcdf-1a541027e8d9\") " pod="openshift-console/console-c6655d68d-wsh5g"
May 25 11:20:36.466384 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:20:36.466379    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a363e903-30ca-4436-bcdf-1a541027e8d9-oauth-serving-cert\") pod \"console-c6655d68d-wsh5g\" (UID: \"a363e903-30ca-4436-bcdf-1a541027e8d9\") " pod="openshift-console/console-c6655d68d-wsh5g"
May 25 11:20:36.567189 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:20:36.567153    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a363e903-30ca-4436-bcdf-1a541027e8d9-service-ca\") pod \"console-c6655d68d-wsh5g\" (UID: \"a363e903-30ca-4436-bcdf-1a541027e8d9\") " pod="openshift-console/console-c6655d68d-wsh5g"
May 25 11:20:36.567189 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:20:36.567189    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a363e903-30ca-4436-bcdf-1a541027e8d9-console-config\") pod \"console-c6655d68d-wsh5g\" (UID: \"a363e903-30ca-4436-bcdf-1a541027e8d9\") " pod="openshift-console/console-c6655d68d-wsh5g"
May 25 11:20:36.567404 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:20:36.567228    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a363e903-30ca-4436-bcdf-1a541027e8d9-trusted-ca-bundle\") pod \"console-c6655d68d-wsh5g\" (UID: \"a363e903-30ca-4436-bcdf-1a541027e8d9\") " pod="openshift-console/console-c6655d68d-wsh5g"
May 25 11:20:36.567404 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:20:36.567265    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a363e903-30ca-4436-bcdf-1a541027e8d9-console-oauth-config\") pod \"console-c6655d68d-wsh5g\" (UID: \"a363e903-30ca-4436-bcdf-1a541027e8d9\") " pod="openshift-console/console-c6655d68d-wsh5g"
May 25 11:20:36.567404 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:20:36.567312    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a363e903-30ca-4436-bcdf-1a541027e8d9-oauth-serving-cert\") pod \"console-c6655d68d-wsh5g\" (UID: \"a363e903-30ca-4436-bcdf-1a541027e8d9\") " pod="openshift-console/console-c6655d68d-wsh5g"
May 25 11:20:36.567404 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:20:36.567359    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a363e903-30ca-4436-bcdf-1a541027e8d9-console-serving-cert\") pod \"console-c6655d68d-wsh5g\" (UID: \"a363e903-30ca-4436-bcdf-1a541027e8d9\") " pod="openshift-console/console-c6655d68d-wsh5g"
May 25 11:20:36.567404 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:20:36.567396    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-fh6l4\" (UniqueName: \"kubernetes.io/projected/a363e903-30ca-4436-bcdf-1a541027e8d9-kube-api-access-fh6l4\") pod \"console-c6655d68d-wsh5g\" (UID: \"a363e903-30ca-4436-bcdf-1a541027e8d9\") " pod="openshift-console/console-c6655d68d-wsh5g"
May 25 11:20:36.568036 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:20:36.568001    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a363e903-30ca-4436-bcdf-1a541027e8d9-console-config\") pod \"console-c6655d68d-wsh5g\" (UID: \"a363e903-30ca-4436-bcdf-1a541027e8d9\") " pod="openshift-console/console-c6655d68d-wsh5g"
May 25 11:20:36.568177 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:20:36.568123    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a363e903-30ca-4436-bcdf-1a541027e8d9-service-ca\") pod \"console-c6655d68d-wsh5g\" (UID: \"a363e903-30ca-4436-bcdf-1a541027e8d9\") " pod="openshift-console/console-c6655d68d-wsh5g"
May 25 11:20:36.568280 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:20:36.568257    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a363e903-30ca-4436-bcdf-1a541027e8d9-oauth-serving-cert\") pod \"console-c6655d68d-wsh5g\" (UID: \"a363e903-30ca-4436-bcdf-1a541027e8d9\") " pod="openshift-console/console-c6655d68d-wsh5g"
May 25 11:20:36.568345 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:20:36.568304    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a363e903-30ca-4436-bcdf-1a541027e8d9-trusted-ca-bundle\") pod \"console-c6655d68d-wsh5g\" (UID: \"a363e903-30ca-4436-bcdf-1a541027e8d9\") " pod="openshift-console/console-c6655d68d-wsh5g"
May 25 11:20:36.569825 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:20:36.569798    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a363e903-30ca-4436-bcdf-1a541027e8d9-console-oauth-config\") pod \"console-c6655d68d-wsh5g\" (UID: \"a363e903-30ca-4436-bcdf-1a541027e8d9\") " pod="openshift-console/console-c6655d68d-wsh5g"
May 25 11:20:36.569923 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:20:36.569872    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a363e903-30ca-4436-bcdf-1a541027e8d9-console-serving-cert\") pod \"console-c6655d68d-wsh5g\" (UID: \"a363e903-30ca-4436-bcdf-1a541027e8d9\") " pod="openshift-console/console-c6655d68d-wsh5g"
May 25 11:20:36.576867 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:20:36.576849    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fh6l4\" (UniqueName: \"kubernetes.io/projected/a363e903-30ca-4436-bcdf-1a541027e8d9-kube-api-access-fh6l4\") pod \"console-c6655d68d-wsh5g\" (UID: \"a363e903-30ca-4436-bcdf-1a541027e8d9\") " pod="openshift-console/console-c6655d68d-wsh5g"
May 25 11:20:36.662095 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:20:36.662015    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c6655d68d-wsh5g"
May 25 11:20:36.789988 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:20:36.789959    2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-c6655d68d-wsh5g"]
May 25 11:20:36.792476 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:20:36.792448    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda363e903_30ca_4436_bcdf_1a541027e8d9.slice/crio-c2fa5f24098af7802d39a7c2580f5c7e12e1c3b5e72e995582b282b76874543a WatchSource:0}: Error finding container c2fa5f24098af7802d39a7c2580f5c7e12e1c3b5e72e995582b282b76874543a: Status 404 returned error can't find the container with id c2fa5f24098af7802d39a7c2580f5c7e12e1c3b5e72e995582b282b76874543a
May 25 11:20:37.622497 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:20:37.622462    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c6655d68d-wsh5g" event={"ID":"a363e903-30ca-4436-bcdf-1a541027e8d9","Type":"ContainerStarted","Data":"900852ae85869490d81f1a0db0ca767f95795594d3453709f8abcac5669b9b95"}
May 25 11:20:37.622497 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:20:37.622502    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c6655d68d-wsh5g" event={"ID":"a363e903-30ca-4436-bcdf-1a541027e8d9","Type":"ContainerStarted","Data":"c2fa5f24098af7802d39a7c2580f5c7e12e1c3b5e72e995582b282b76874543a"}
May 25 11:20:37.642080 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:20:37.642029    2560 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console/console-c6655d68d-wsh5g" podStartSLOduration=1.642012123 podStartE2EDuration="1.642012123s" podCreationTimestamp="2026-05-25 11:20:36 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 11:20:37.640390905 +0000 UTC m=+179.321368889" watchObservedRunningTime="2026-05-25 11:20:37.642012123 +0000 UTC m=+179.322990098"
May 25 11:20:42.741182 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:20:42.741148    2560 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-c6655d68d-wsh5g"]
May 25 11:20:42.790674 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:20:42.790645    2560 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console/console-7c4bbdcc86-bnkkf"]
May 25 11:20:42.795413 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:20:42.795391    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c4bbdcc86-bnkkf"
May 25 11:20:42.807710 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:20:42.807684    2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7c4bbdcc86-bnkkf"]
May 25 11:20:42.920433 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:20:42.920401    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/af689d73-9b4a-4020-b811-b5930019e908-console-config\") pod \"console-7c4bbdcc86-bnkkf\" (UID: \"af689d73-9b4a-4020-b811-b5930019e908\") " pod="openshift-console/console-7c4bbdcc86-bnkkf"
May 25 11:20:42.920433 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:20:42.920435    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/af689d73-9b4a-4020-b811-b5930019e908-trusted-ca-bundle\") pod \"console-7c4bbdcc86-bnkkf\" (UID: \"af689d73-9b4a-4020-b811-b5930019e908\") " pod="openshift-console/console-7c4bbdcc86-bnkkf"
May 25 11:20:42.920640 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:20:42.920472    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/af689d73-9b4a-4020-b811-b5930019e908-console-oauth-config\") pod \"console-7c4bbdcc86-bnkkf\" (UID: \"af689d73-9b4a-4020-b811-b5930019e908\") " pod="openshift-console/console-7c4bbdcc86-bnkkf"
May 25 11:20:42.920640 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:20:42.920520    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/af689d73-9b4a-4020-b811-b5930019e908-service-ca\") pod \"console-7c4bbdcc86-bnkkf\" (UID: \"af689d73-9b4a-4020-b811-b5930019e908\") " pod="openshift-console/console-7c4bbdcc86-bnkkf"
May 25 11:20:42.920640 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:20:42.920554    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/af689d73-9b4a-4020-b811-b5930019e908-oauth-serving-cert\") pod \"console-7c4bbdcc86-bnkkf\" (UID: \"af689d73-9b4a-4020-b811-b5930019e908\") " pod="openshift-console/console-7c4bbdcc86-bnkkf"
May 25 11:20:42.920640 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:20:42.920576    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92xgz\" (UniqueName: \"kubernetes.io/projected/af689d73-9b4a-4020-b811-b5930019e908-kube-api-access-92xgz\") pod \"console-7c4bbdcc86-bnkkf\" (UID: \"af689d73-9b4a-4020-b811-b5930019e908\") " pod="openshift-console/console-7c4bbdcc86-bnkkf"
May 25 11:20:42.920640 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:20:42.920615    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/af689d73-9b4a-4020-b811-b5930019e908-console-serving-cert\") pod \"console-7c4bbdcc86-bnkkf\" (UID: \"af689d73-9b4a-4020-b811-b5930019e908\") " pod="openshift-console/console-7c4bbdcc86-bnkkf"
May 25 11:20:43.021501 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:20:43.021404    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/af689d73-9b4a-4020-b811-b5930019e908-console-serving-cert\") pod \"console-7c4bbdcc86-bnkkf\" (UID: \"af689d73-9b4a-4020-b811-b5930019e908\") " pod="openshift-console/console-7c4bbdcc86-bnkkf"
May 25 11:20:43.021501 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:20:43.021457    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/af689d73-9b4a-4020-b811-b5930019e908-console-config\") pod \"console-7c4bbdcc86-bnkkf\" (UID: \"af689d73-9b4a-4020-b811-b5930019e908\") " pod="openshift-console/console-7c4bbdcc86-bnkkf"
May 25 11:20:43.021730 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:20:43.021636    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/af689d73-9b4a-4020-b811-b5930019e908-trusted-ca-bundle\") pod \"console-7c4bbdcc86-bnkkf\" (UID: \"af689d73-9b4a-4020-b811-b5930019e908\") " pod="openshift-console/console-7c4bbdcc86-bnkkf"
May 25 11:20:43.021730 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:20:43.021723    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/af689d73-9b4a-4020-b811-b5930019e908-console-oauth-config\") pod \"console-7c4bbdcc86-bnkkf\" (UID: \"af689d73-9b4a-4020-b811-b5930019e908\") " pod="openshift-console/console-7c4bbdcc86-bnkkf"
May 25 11:20:43.021834 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:20:43.021771    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/af689d73-9b4a-4020-b811-b5930019e908-service-ca\") pod \"console-7c4bbdcc86-bnkkf\" (UID: \"af689d73-9b4a-4020-b811-b5930019e908\") " pod="openshift-console/console-7c4bbdcc86-bnkkf"
May 25 11:20:43.021834 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:20:43.021802    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/af689d73-9b4a-4020-b811-b5930019e908-oauth-serving-cert\") pod \"console-7c4bbdcc86-bnkkf\" (UID: \"af689d73-9b4a-4020-b811-b5930019e908\") " pod="openshift-console/console-7c4bbdcc86-bnkkf"
May 25 11:20:43.021834 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:20:43.021828    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-92xgz\" (UniqueName: \"kubernetes.io/projected/af689d73-9b4a-4020-b811-b5930019e908-kube-api-access-92xgz\") pod \"console-7c4bbdcc86-bnkkf\" (UID: \"af689d73-9b4a-4020-b811-b5930019e908\") " pod="openshift-console/console-7c4bbdcc86-bnkkf"
May 25 11:20:43.022231 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:20:43.022178    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/af689d73-9b4a-4020-b811-b5930019e908-console-config\") pod \"console-7c4bbdcc86-bnkkf\" (UID: \"af689d73-9b4a-4020-b811-b5930019e908\") " pod="openshift-console/console-7c4bbdcc86-bnkkf"
May 25 11:20:43.022425 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:20:43.022405    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/af689d73-9b4a-4020-b811-b5930019e908-trusted-ca-bundle\") pod \"console-7c4bbdcc86-bnkkf\" (UID: \"af689d73-9b4a-4020-b811-b5930019e908\") " pod="openshift-console/console-7c4bbdcc86-bnkkf"
May 25 11:20:43.022550 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:20:43.022530    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/af689d73-9b4a-4020-b811-b5930019e908-service-ca\") pod \"console-7c4bbdcc86-bnkkf\" (UID: \"af689d73-9b4a-4020-b811-b5930019e908\") " pod="openshift-console/console-7c4bbdcc86-bnkkf"
May 25 11:20:43.022719 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:20:43.022692    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/af689d73-9b4a-4020-b811-b5930019e908-oauth-serving-cert\") pod \"console-7c4bbdcc86-bnkkf\" (UID: \"af689d73-9b4a-4020-b811-b5930019e908\") " pod="openshift-console/console-7c4bbdcc86-bnkkf"
May 25 11:20:43.024176 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:20:43.024155    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/af689d73-9b4a-4020-b811-b5930019e908-console-oauth-config\") pod \"console-7c4bbdcc86-bnkkf\" (UID: \"af689d73-9b4a-4020-b811-b5930019e908\") " pod="openshift-console/console-7c4bbdcc86-bnkkf"
May 25 11:20:43.024286 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:20:43.024263    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/af689d73-9b4a-4020-b811-b5930019e908-console-serving-cert\") pod \"console-7c4bbdcc86-bnkkf\" (UID: \"af689d73-9b4a-4020-b811-b5930019e908\") " pod="openshift-console/console-7c4bbdcc86-bnkkf"
May 25 11:20:43.031441 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:20:43.031415    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-92xgz\" (UniqueName: \"kubernetes.io/projected/af689d73-9b4a-4020-b811-b5930019e908-kube-api-access-92xgz\") pod \"console-7c4bbdcc86-bnkkf\" (UID: \"af689d73-9b4a-4020-b811-b5930019e908\") " pod="openshift-console/console-7c4bbdcc86-bnkkf"
May 25 11:20:43.105441 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:20:43.105404    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c4bbdcc86-bnkkf"
May 25 11:20:43.237675 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:20:43.237651    2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7c4bbdcc86-bnkkf"]
May 25 11:20:43.239868 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:20:43.239818    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf689d73_9b4a_4020_b811_b5930019e908.slice/crio-34d147d663bf688e352efad432b30a628ec10297f3e529784b4334c6601aeb6e WatchSource:0}: Error finding container 34d147d663bf688e352efad432b30a628ec10297f3e529784b4334c6601aeb6e: Status 404 returned error can't find the container with id 34d147d663bf688e352efad432b30a628ec10297f3e529784b4334c6601aeb6e
May 25 11:20:43.640836 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:20:43.640802    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c4bbdcc86-bnkkf" event={"ID":"af689d73-9b4a-4020-b811-b5930019e908","Type":"ContainerStarted","Data":"3915b38dd7ce9b01f0e9960a7c82d4cb3e330fd2741d320f570c962615cf441c"}
May 25 11:20:43.641003 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:20:43.640842    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c4bbdcc86-bnkkf" event={"ID":"af689d73-9b4a-4020-b811-b5930019e908","Type":"ContainerStarted","Data":"34d147d663bf688e352efad432b30a628ec10297f3e529784b4334c6601aeb6e"}
May 25 11:20:43.662466 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:20:43.662417    2560 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console/console-7c4bbdcc86-bnkkf" podStartSLOduration=1.662402847 podStartE2EDuration="1.662402847s" podCreationTimestamp="2026-05-25 11:20:42 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 11:20:43.660949009 +0000 UTC m=+185.341926981" watchObservedRunningTime="2026-05-25 11:20:43.662402847 +0000 UTC m=+185.343380814"
May 25 11:20:46.662661 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:20:46.662632    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-c6655d68d-wsh5g"
May 25 11:20:53.106380 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:20:53.106341    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7c4bbdcc86-bnkkf"
May 25 11:20:53.106380 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:20:53.106390    2560 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7c4bbdcc86-bnkkf"
May 25 11:20:53.111071 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:20:53.111049    2560 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7c4bbdcc86-bnkkf"
May 25 11:20:53.671567 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:20:53.671537    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7c4bbdcc86-bnkkf"
May 25 11:20:53.719233 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:20:53.719186    2560 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-768d57dcfd-m4gtl"]
May 25 11:21:07.760964 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:21:07.760902    2560 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-console/console-c6655d68d-wsh5g" podUID="a363e903-30ca-4436-bcdf-1a541027e8d9" containerName="console" containerID="cri-o://900852ae85869490d81f1a0db0ca767f95795594d3453709f8abcac5669b9b95" gracePeriod=15
May 25 11:21:08.008515 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:21:08.008488    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-c6655d68d-wsh5g_a363e903-30ca-4436-bcdf-1a541027e8d9/console/0.log"
May 25 11:21:08.008652 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:21:08.008548    2560 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c6655d68d-wsh5g"
May 25 11:21:08.143584 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:21:08.143547    2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a363e903-30ca-4436-bcdf-1a541027e8d9-console-serving-cert\") pod \"a363e903-30ca-4436-bcdf-1a541027e8d9\" (UID: \"a363e903-30ca-4436-bcdf-1a541027e8d9\") "
May 25 11:21:08.143747 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:21:08.143596    2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a363e903-30ca-4436-bcdf-1a541027e8d9-oauth-serving-cert\") pod \"a363e903-30ca-4436-bcdf-1a541027e8d9\" (UID: \"a363e903-30ca-4436-bcdf-1a541027e8d9\") "
May 25 11:21:08.143747 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:21:08.143635    2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fh6l4\" (UniqueName: \"kubernetes.io/projected/a363e903-30ca-4436-bcdf-1a541027e8d9-kube-api-access-fh6l4\") pod \"a363e903-30ca-4436-bcdf-1a541027e8d9\" (UID: \"a363e903-30ca-4436-bcdf-1a541027e8d9\") "
May 25 11:21:08.143747 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:21:08.143662    2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a363e903-30ca-4436-bcdf-1a541027e8d9-service-ca\") pod \"a363e903-30ca-4436-bcdf-1a541027e8d9\" (UID: \"a363e903-30ca-4436-bcdf-1a541027e8d9\") "
May 25 11:21:08.143747 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:21:08.143689    2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a363e903-30ca-4436-bcdf-1a541027e8d9-console-config\") pod \"a363e903-30ca-4436-bcdf-1a541027e8d9\" (UID: \"a363e903-30ca-4436-bcdf-1a541027e8d9\") "
May 25 11:21:08.143941 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:21:08.143751    2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a363e903-30ca-4436-bcdf-1a541027e8d9-trusted-ca-bundle\") pod \"a363e903-30ca-4436-bcdf-1a541027e8d9\" (UID: \"a363e903-30ca-4436-bcdf-1a541027e8d9\") "
May 25 11:21:08.143941 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:21:08.143783    2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a363e903-30ca-4436-bcdf-1a541027e8d9-console-oauth-config\") pod \"a363e903-30ca-4436-bcdf-1a541027e8d9\" (UID: \"a363e903-30ca-4436-bcdf-1a541027e8d9\") "
May 25 11:21:08.144189 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:21:08.144155    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a363e903-30ca-4436-bcdf-1a541027e8d9-service-ca" (OuterVolumeSpecName: "service-ca") pod "a363e903-30ca-4436-bcdf-1a541027e8d9" (UID: "a363e903-30ca-4436-bcdf-1a541027e8d9"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 11:21:08.144316 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:21:08.144191    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a363e903-30ca-4436-bcdf-1a541027e8d9-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "a363e903-30ca-4436-bcdf-1a541027e8d9" (UID: "a363e903-30ca-4436-bcdf-1a541027e8d9"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 11:21:08.144316 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:21:08.144198    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a363e903-30ca-4436-bcdf-1a541027e8d9-console-config" (OuterVolumeSpecName: "console-config") pod "a363e903-30ca-4436-bcdf-1a541027e8d9" (UID: "a363e903-30ca-4436-bcdf-1a541027e8d9"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 11:21:08.144316 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:21:08.144290    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a363e903-30ca-4436-bcdf-1a541027e8d9-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "a363e903-30ca-4436-bcdf-1a541027e8d9" (UID: "a363e903-30ca-4436-bcdf-1a541027e8d9"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 11:21:08.145831 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:21:08.145803    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a363e903-30ca-4436-bcdf-1a541027e8d9-kube-api-access-fh6l4" (OuterVolumeSpecName: "kube-api-access-fh6l4") pod "a363e903-30ca-4436-bcdf-1a541027e8d9" (UID: "a363e903-30ca-4436-bcdf-1a541027e8d9"). InnerVolumeSpecName "kube-api-access-fh6l4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 11:21:08.145909 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:21:08.145832    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a363e903-30ca-4436-bcdf-1a541027e8d9-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "a363e903-30ca-4436-bcdf-1a541027e8d9" (UID: "a363e903-30ca-4436-bcdf-1a541027e8d9"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 11:21:08.145909 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:21:08.145863    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a363e903-30ca-4436-bcdf-1a541027e8d9-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "a363e903-30ca-4436-bcdf-1a541027e8d9" (UID: "a363e903-30ca-4436-bcdf-1a541027e8d9"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 11:21:08.244498 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:21:08.244466    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fh6l4\" (UniqueName: \"kubernetes.io/projected/a363e903-30ca-4436-bcdf-1a541027e8d9-kube-api-access-fh6l4\") on node \"ip-10-0-128-146.ec2.internal\" DevicePath \"\""
May 25 11:21:08.244498 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:21:08.244495    2560 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a363e903-30ca-4436-bcdf-1a541027e8d9-service-ca\") on node \"ip-10-0-128-146.ec2.internal\" DevicePath \"\""
May 25 11:21:08.244498 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:21:08.244505    2560 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a363e903-30ca-4436-bcdf-1a541027e8d9-console-config\") on node \"ip-10-0-128-146.ec2.internal\" DevicePath \"\""
May 25 11:21:08.244723 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:21:08.244514    2560 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a363e903-30ca-4436-bcdf-1a541027e8d9-trusted-ca-bundle\") on node \"ip-10-0-128-146.ec2.internal\" DevicePath \"\""
May 25 11:21:08.244723 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:21:08.244523    2560 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a363e903-30ca-4436-bcdf-1a541027e8d9-console-oauth-config\") on node \"ip-10-0-128-146.ec2.internal\" DevicePath \"\""
May 25 11:21:08.244723 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:21:08.244533    2560 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a363e903-30ca-4436-bcdf-1a541027e8d9-console-serving-cert\") on node \"ip-10-0-128-146.ec2.internal\" DevicePath \"\""
May 25 11:21:08.244723 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:21:08.244542    2560 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a363e903-30ca-4436-bcdf-1a541027e8d9-oauth-serving-cert\") on node \"ip-10-0-128-146.ec2.internal\" DevicePath \"\""
May 25 11:21:08.709641 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:21:08.709616    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-c6655d68d-wsh5g_a363e903-30ca-4436-bcdf-1a541027e8d9/console/0.log"
May 25 11:21:08.709839 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:21:08.709652    2560 generic.go:358] "Generic (PLEG): container finished" podID="a363e903-30ca-4436-bcdf-1a541027e8d9" containerID="900852ae85869490d81f1a0db0ca767f95795594d3453709f8abcac5669b9b95" exitCode=2
May 25 11:21:08.709839 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:21:08.709719    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c6655d68d-wsh5g" event={"ID":"a363e903-30ca-4436-bcdf-1a541027e8d9","Type":"ContainerDied","Data":"900852ae85869490d81f1a0db0ca767f95795594d3453709f8abcac5669b9b95"}
May 25 11:21:08.709839 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:21:08.709732    2560 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c6655d68d-wsh5g"
May 25 11:21:08.709839 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:21:08.709745    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c6655d68d-wsh5g" event={"ID":"a363e903-30ca-4436-bcdf-1a541027e8d9","Type":"ContainerDied","Data":"c2fa5f24098af7802d39a7c2580f5c7e12e1c3b5e72e995582b282b76874543a"}
May 25 11:21:08.709839 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:21:08.709760    2560 scope.go:117] "RemoveContainer" containerID="900852ae85869490d81f1a0db0ca767f95795594d3453709f8abcac5669b9b95"
May 25 11:21:08.718396 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:21:08.718377    2560 scope.go:117] "RemoveContainer" containerID="900852ae85869490d81f1a0db0ca767f95795594d3453709f8abcac5669b9b95"
May 25 11:21:08.718658 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:21:08.718637    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"900852ae85869490d81f1a0db0ca767f95795594d3453709f8abcac5669b9b95\": container with ID starting with 900852ae85869490d81f1a0db0ca767f95795594d3453709f8abcac5669b9b95 not found: ID does not exist" containerID="900852ae85869490d81f1a0db0ca767f95795594d3453709f8abcac5669b9b95"
May 25 11:21:08.718717 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:21:08.718667    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"900852ae85869490d81f1a0db0ca767f95795594d3453709f8abcac5669b9b95"} err="failed to get container status \"900852ae85869490d81f1a0db0ca767f95795594d3453709f8abcac5669b9b95\": rpc error: code = NotFound desc = could not find container \"900852ae85869490d81f1a0db0ca767f95795594d3453709f8abcac5669b9b95\": container with ID starting with 900852ae85869490d81f1a0db0ca767f95795594d3453709f8abcac5669b9b95 not found: ID does not exist"
May 25 11:21:08.732499 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:21:08.732475    2560 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-c6655d68d-wsh5g"]
May 25 11:21:08.737826 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:21:08.737804    2560 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-c6655d68d-wsh5g"]
May 25 11:21:08.933884 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:21:08.933853    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a363e903-30ca-4436-bcdf-1a541027e8d9" path="/var/lib/kubelet/pods/a363e903-30ca-4436-bcdf-1a541027e8d9/volumes"
May 25 11:21:18.742228 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:21:18.742174    2560 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-console/console-768d57dcfd-m4gtl" podUID="8bb8e0f0-519e-49bd-a871-e5c13624fd0a" containerName="console" containerID="cri-o://ae662f2d9abff807a13021aa500788d9ec847dc6df7428eccec6f97230aabe93" gracePeriod=15
May 25 11:21:18.977329 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:21:18.977305    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-768d57dcfd-m4gtl_8bb8e0f0-519e-49bd-a871-e5c13624fd0a/console/0.log"
May 25 11:21:18.977443 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:21:18.977369    2560 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-768d57dcfd-m4gtl"
May 25 11:21:19.028870 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:21:19.028796    2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8bb8e0f0-519e-49bd-a871-e5c13624fd0a-console-serving-cert\") pod \"8bb8e0f0-519e-49bd-a871-e5c13624fd0a\" (UID: \"8bb8e0f0-519e-49bd-a871-e5c13624fd0a\") "
May 25 11:21:19.028870 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:21:19.028839    2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8bb8e0f0-519e-49bd-a871-e5c13624fd0a-oauth-serving-cert\") pod \"8bb8e0f0-519e-49bd-a871-e5c13624fd0a\" (UID: \"8bb8e0f0-519e-49bd-a871-e5c13624fd0a\") "
May 25 11:21:19.028870 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:21:19.028862    2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8bb8e0f0-519e-49bd-a871-e5c13624fd0a-trusted-ca-bundle\") pod \"8bb8e0f0-519e-49bd-a871-e5c13624fd0a\" (UID: \"8bb8e0f0-519e-49bd-a871-e5c13624fd0a\") "
May 25 11:21:19.029123 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:21:19.028880    2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8bb8e0f0-519e-49bd-a871-e5c13624fd0a-console-config\") pod \"8bb8e0f0-519e-49bd-a871-e5c13624fd0a\" (UID: \"8bb8e0f0-519e-49bd-a871-e5c13624fd0a\") "
May 25 11:21:19.029123 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:21:19.028903    2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8bb8e0f0-519e-49bd-a871-e5c13624fd0a-service-ca\") pod \"8bb8e0f0-519e-49bd-a871-e5c13624fd0a\" (UID: \"8bb8e0f0-519e-49bd-a871-e5c13624fd0a\") "
May 25 11:21:19.029123 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:21:19.028923    2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r26kt\" (UniqueName: \"kubernetes.io/projected/8bb8e0f0-519e-49bd-a871-e5c13624fd0a-kube-api-access-r26kt\") pod \"8bb8e0f0-519e-49bd-a871-e5c13624fd0a\" (UID: \"8bb8e0f0-519e-49bd-a871-e5c13624fd0a\") "
May 25 11:21:19.029387 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:21:19.029358    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bb8e0f0-519e-49bd-a871-e5c13624fd0a-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "8bb8e0f0-519e-49bd-a871-e5c13624fd0a" (UID: "8bb8e0f0-519e-49bd-a871-e5c13624fd0a"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 11:21:19.029446 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:21:19.029382    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bb8e0f0-519e-49bd-a871-e5c13624fd0a-console-config" (OuterVolumeSpecName: "console-config") pod "8bb8e0f0-519e-49bd-a871-e5c13624fd0a" (UID: "8bb8e0f0-519e-49bd-a871-e5c13624fd0a"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 11:21:19.029446 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:21:19.029379    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bb8e0f0-519e-49bd-a871-e5c13624fd0a-service-ca" (OuterVolumeSpecName: "service-ca") pod "8bb8e0f0-519e-49bd-a871-e5c13624fd0a" (UID: "8bb8e0f0-519e-49bd-a871-e5c13624fd0a"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 11:21:19.029446 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:21:19.029391    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bb8e0f0-519e-49bd-a871-e5c13624fd0a-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "8bb8e0f0-519e-49bd-a871-e5c13624fd0a" (UID: "8bb8e0f0-519e-49bd-a871-e5c13624fd0a"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 11:21:19.030959 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:21:19.030938    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8bb8e0f0-519e-49bd-a871-e5c13624fd0a-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "8bb8e0f0-519e-49bd-a871-e5c13624fd0a" (UID: "8bb8e0f0-519e-49bd-a871-e5c13624fd0a"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 11:21:19.031047 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:21:19.030999    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bb8e0f0-519e-49bd-a871-e5c13624fd0a-kube-api-access-r26kt" (OuterVolumeSpecName: "kube-api-access-r26kt") pod "8bb8e0f0-519e-49bd-a871-e5c13624fd0a" (UID: "8bb8e0f0-519e-49bd-a871-e5c13624fd0a"). InnerVolumeSpecName "kube-api-access-r26kt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 11:21:19.129637 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:21:19.129600    2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8bb8e0f0-519e-49bd-a871-e5c13624fd0a-console-oauth-config\") pod \"8bb8e0f0-519e-49bd-a871-e5c13624fd0a\" (UID: \"8bb8e0f0-519e-49bd-a871-e5c13624fd0a\") "
May 25 11:21:19.129806 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:21:19.129783    2560 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8bb8e0f0-519e-49bd-a871-e5c13624fd0a-console-serving-cert\") on node \"ip-10-0-128-146.ec2.internal\" DevicePath \"\""
May 25 11:21:19.129806 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:21:19.129802    2560 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8bb8e0f0-519e-49bd-a871-e5c13624fd0a-oauth-serving-cert\") on node \"ip-10-0-128-146.ec2.internal\" DevicePath \"\""
May 25 11:21:19.129876 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:21:19.129818    2560 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8bb8e0f0-519e-49bd-a871-e5c13624fd0a-trusted-ca-bundle\") on node \"ip-10-0-128-146.ec2.internal\" DevicePath \"\""
May 25 11:21:19.129876 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:21:19.129832    2560 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8bb8e0f0-519e-49bd-a871-e5c13624fd0a-console-config\") on node \"ip-10-0-128-146.ec2.internal\" DevicePath \"\""
May 25 11:21:19.129876 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:21:19.129845    2560 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8bb8e0f0-519e-49bd-a871-e5c13624fd0a-service-ca\") on node \"ip-10-0-128-146.ec2.internal\" DevicePath \"\""
May 25 11:21:19.129876 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:21:19.129857    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r26kt\" (UniqueName: \"kubernetes.io/projected/8bb8e0f0-519e-49bd-a871-e5c13624fd0a-kube-api-access-r26kt\") on node \"ip-10-0-128-146.ec2.internal\" DevicePath \"\""
May 25 11:21:19.131639 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:21:19.131617    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8bb8e0f0-519e-49bd-a871-e5c13624fd0a-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "8bb8e0f0-519e-49bd-a871-e5c13624fd0a" (UID: "8bb8e0f0-519e-49bd-a871-e5c13624fd0a"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 11:21:19.230176 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:21:19.230126    2560 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8bb8e0f0-519e-49bd-a871-e5c13624fd0a-console-oauth-config\") on node \"ip-10-0-128-146.ec2.internal\" DevicePath \"\""
May 25 11:21:19.743432 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:21:19.743407    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-768d57dcfd-m4gtl_8bb8e0f0-519e-49bd-a871-e5c13624fd0a/console/0.log"
May 25 11:21:19.743829 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:21:19.743443    2560 generic.go:358] "Generic (PLEG): container finished" podID="8bb8e0f0-519e-49bd-a871-e5c13624fd0a" containerID="ae662f2d9abff807a13021aa500788d9ec847dc6df7428eccec6f97230aabe93" exitCode=2
May 25 11:21:19.743829 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:21:19.743497    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-768d57dcfd-m4gtl" event={"ID":"8bb8e0f0-519e-49bd-a871-e5c13624fd0a","Type":"ContainerDied","Data":"ae662f2d9abff807a13021aa500788d9ec847dc6df7428eccec6f97230aabe93"}
May 25 11:21:19.743829 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:21:19.743514    2560 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-768d57dcfd-m4gtl"
May 25 11:21:19.743829 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:21:19.743527    2560 scope.go:117] "RemoveContainer" containerID="ae662f2d9abff807a13021aa500788d9ec847dc6df7428eccec6f97230aabe93"
May 25 11:21:19.743829 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:21:19.743518    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-768d57dcfd-m4gtl" event={"ID":"8bb8e0f0-519e-49bd-a871-e5c13624fd0a","Type":"ContainerDied","Data":"a5e87bb4fc5ba480dce1308d092b35a4af103d9e93279c54a00d43635139d490"}
May 25 11:21:19.750967 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:21:19.750947    2560 scope.go:117] "RemoveContainer" containerID="ae662f2d9abff807a13021aa500788d9ec847dc6df7428eccec6f97230aabe93"
May 25 11:21:19.751195 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:21:19.751175    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae662f2d9abff807a13021aa500788d9ec847dc6df7428eccec6f97230aabe93\": container with ID starting with ae662f2d9abff807a13021aa500788d9ec847dc6df7428eccec6f97230aabe93 not found: ID does not exist" containerID="ae662f2d9abff807a13021aa500788d9ec847dc6df7428eccec6f97230aabe93"
May 25 11:21:19.751281 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:21:19.751202    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae662f2d9abff807a13021aa500788d9ec847dc6df7428eccec6f97230aabe93"} err="failed to get container status \"ae662f2d9abff807a13021aa500788d9ec847dc6df7428eccec6f97230aabe93\": rpc error: code = NotFound desc = could not find container \"ae662f2d9abff807a13021aa500788d9ec847dc6df7428eccec6f97230aabe93\": container with ID starting with ae662f2d9abff807a13021aa500788d9ec847dc6df7428eccec6f97230aabe93 not found: ID does not exist"
May 25 11:21:19.768419 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:21:19.768394    2560 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-768d57dcfd-m4gtl"]
May 25 11:21:19.771811 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:21:19.771791    2560 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-768d57dcfd-m4gtl"]
May 25 11:21:20.933740 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:21:20.933709    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8bb8e0f0-519e-49bd-a871-e5c13624fd0a" path="/var/lib/kubelet/pods/8bb8e0f0-519e-49bd-a871-e5c13624fd0a/volumes"
May 25 11:21:35.138015 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:21:35.137985    2560 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c9m2qd"]
May 25 11:21:35.138510 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:21:35.138280    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8bb8e0f0-519e-49bd-a871-e5c13624fd0a" containerName="console"
May 25 11:21:35.138510 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:21:35.138291    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="8bb8e0f0-519e-49bd-a871-e5c13624fd0a" containerName="console"
May 25 11:21:35.138510 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:21:35.138301    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a363e903-30ca-4436-bcdf-1a541027e8d9" containerName="console"
May 25 11:21:35.138510 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:21:35.138306    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="a363e903-30ca-4436-bcdf-1a541027e8d9" containerName="console"
May 25 11:21:35.138510 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:21:35.138374    2560 memory_manager.go:356] "RemoveStaleState removing state" podUID="8bb8e0f0-519e-49bd-a871-e5c13624fd0a" containerName="console"
May 25 11:21:35.138510 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:21:35.138381    2560 memory_manager.go:356] "RemoveStaleState removing state" podUID="a363e903-30ca-4436-bcdf-1a541027e8d9" containerName="console"
May 25 11:21:35.141175 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:21:35.141158    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c9m2qd"
May 25 11:21:35.143963 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:21:35.143938    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-4z78r\""
May 25 11:21:35.144098 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:21:35.143991    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
May 25 11:21:35.145042 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:21:35.145025    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
May 25 11:21:35.149419 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:21:35.149395    2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c9m2qd"]
May 25 11:21:35.248984 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:21:35.248952    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3442d020-91dc-4674-ae3a-669407a861df-bundle\") pod \"59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c9m2qd\" (UID: \"3442d020-91dc-4674-ae3a-669407a861df\") " pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c9m2qd"
May 25 11:21:35.249184 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:21:35.249014    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3442d020-91dc-4674-ae3a-669407a861df-util\") pod \"59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c9m2qd\" (UID: \"3442d020-91dc-4674-ae3a-669407a861df\") " pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c9m2qd"
May 25 11:21:35.249184 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:21:35.249107    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxlmf\" (UniqueName: \"kubernetes.io/projected/3442d020-91dc-4674-ae3a-669407a861df-kube-api-access-rxlmf\") pod \"59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c9m2qd\" (UID: \"3442d020-91dc-4674-ae3a-669407a861df\") " pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c9m2qd"
May 25 11:21:35.349544 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:21:35.349505    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-rxlmf\" (UniqueName: \"kubernetes.io/projected/3442d020-91dc-4674-ae3a-669407a861df-kube-api-access-rxlmf\") pod \"59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c9m2qd\" (UID: \"3442d020-91dc-4674-ae3a-669407a861df\") " pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c9m2qd"
May 25 11:21:35.349710 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:21:35.349551    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3442d020-91dc-4674-ae3a-669407a861df-bundle\") pod \"59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c9m2qd\" (UID: \"3442d020-91dc-4674-ae3a-669407a861df\") " pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c9m2qd"
May 25 11:21:35.349710 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:21:35.349601    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3442d020-91dc-4674-ae3a-669407a861df-util\") pod \"59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c9m2qd\" (UID: \"3442d020-91dc-4674-ae3a-669407a861df\") " pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c9m2qd"
May 25 11:21:35.349905 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:21:35.349884    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3442d020-91dc-4674-ae3a-669407a861df-bundle\") pod \"59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c9m2qd\" (UID: \"3442d020-91dc-4674-ae3a-669407a861df\") " pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c9m2qd"
May 25 11:21:35.349965 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:21:35.349912    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3442d020-91dc-4674-ae3a-669407a861df-util\") pod \"59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c9m2qd\" (UID: \"3442d020-91dc-4674-ae3a-669407a861df\") " pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c9m2qd"
May 25 11:21:35.358610 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:21:35.358587    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxlmf\" (UniqueName: \"kubernetes.io/projected/3442d020-91dc-4674-ae3a-669407a861df-kube-api-access-rxlmf\") pod \"59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c9m2qd\" (UID: \"3442d020-91dc-4674-ae3a-669407a861df\") " pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c9m2qd"
May 25 11:21:35.450925 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:21:35.450846    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c9m2qd"
May 25 11:21:35.572825 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:21:35.572796    2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c9m2qd"]
May 25 11:21:35.576046 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:21:35.576001    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3442d020_91dc_4674_ae3a_669407a861df.slice/crio-170cefeb7b5ee20e93d11fd9e577c1584c68a8df007bc0fd9e2650b9e8961bd9 WatchSource:0}: Error finding container 170cefeb7b5ee20e93d11fd9e577c1584c68a8df007bc0fd9e2650b9e8961bd9: Status 404 returned error can't find the container with id 170cefeb7b5ee20e93d11fd9e577c1584c68a8df007bc0fd9e2650b9e8961bd9
May 25 11:21:35.787664 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:21:35.787581    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c9m2qd" event={"ID":"3442d020-91dc-4674-ae3a-669407a861df","Type":"ContainerStarted","Data":"170cefeb7b5ee20e93d11fd9e577c1584c68a8df007bc0fd9e2650b9e8961bd9"}
May 25 11:21:40.806614 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:21:40.806574    2560 generic.go:358] "Generic (PLEG): container finished" podID="3442d020-91dc-4674-ae3a-669407a861df" containerID="0aaf3298e7912ae26a0b891d2c91895fa5fca78f893aea57c9cb329d0afe750a" exitCode=0
May 25 11:21:40.807010 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:21:40.806663    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c9m2qd" event={"ID":"3442d020-91dc-4674-ae3a-669407a861df","Type":"ContainerDied","Data":"0aaf3298e7912ae26a0b891d2c91895fa5fca78f893aea57c9cb329d0afe750a"}
May 25 11:21:43.819600 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:21:43.819566    2560 generic.go:358] "Generic (PLEG): container finished" podID="3442d020-91dc-4674-ae3a-669407a861df" containerID="0c85fe2c22f596fc55edbae8c6ee0ca01a7442238d54c0e14199c7d89d000d0a" exitCode=0
May 25 11:21:43.819979 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:21:43.819644    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c9m2qd" event={"ID":"3442d020-91dc-4674-ae3a-669407a861df","Type":"ContainerDied","Data":"0c85fe2c22f596fc55edbae8c6ee0ca01a7442238d54c0e14199c7d89d000d0a"}
May 25 11:21:50.844152 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:21:50.844116    2560 generic.go:358] "Generic (PLEG): container finished" podID="3442d020-91dc-4674-ae3a-669407a861df" containerID="0c002a698de70a1b82a69289674cd4603b6af9882d0bc3129c1772474c43340e" exitCode=0
May 25 11:21:50.844599 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:21:50.844199    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c9m2qd" event={"ID":"3442d020-91dc-4674-ae3a-669407a861df","Type":"ContainerDied","Data":"0c002a698de70a1b82a69289674cd4603b6af9882d0bc3129c1772474c43340e"}
May 25 11:21:51.964438 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:21:51.964416    2560 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c9m2qd"
May 25 11:21:52.092498 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:21:52.092463    2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rxlmf\" (UniqueName: \"kubernetes.io/projected/3442d020-91dc-4674-ae3a-669407a861df-kube-api-access-rxlmf\") pod \"3442d020-91dc-4674-ae3a-669407a861df\" (UID: \"3442d020-91dc-4674-ae3a-669407a861df\") "
May 25 11:21:52.092658 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:21:52.092531    2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3442d020-91dc-4674-ae3a-669407a861df-bundle\") pod \"3442d020-91dc-4674-ae3a-669407a861df\" (UID: \"3442d020-91dc-4674-ae3a-669407a861df\") "
May 25 11:21:52.092658 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:21:52.092551    2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3442d020-91dc-4674-ae3a-669407a861df-util\") pod \"3442d020-91dc-4674-ae3a-669407a861df\" (UID: \"3442d020-91dc-4674-ae3a-669407a861df\") "
May 25 11:21:52.093116 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:21:52.093089    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3442d020-91dc-4674-ae3a-669407a861df-bundle" (OuterVolumeSpecName: "bundle") pod "3442d020-91dc-4674-ae3a-669407a861df" (UID: "3442d020-91dc-4674-ae3a-669407a861df"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:21:52.094810 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:21:52.094778    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3442d020-91dc-4674-ae3a-669407a861df-kube-api-access-rxlmf" (OuterVolumeSpecName: "kube-api-access-rxlmf") pod "3442d020-91dc-4674-ae3a-669407a861df" (UID: "3442d020-91dc-4674-ae3a-669407a861df"). InnerVolumeSpecName "kube-api-access-rxlmf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 11:21:52.096765 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:21:52.096742    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3442d020-91dc-4674-ae3a-669407a861df-util" (OuterVolumeSpecName: "util") pod "3442d020-91dc-4674-ae3a-669407a861df" (UID: "3442d020-91dc-4674-ae3a-669407a861df"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:21:52.193691 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:21:52.193653    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rxlmf\" (UniqueName: \"kubernetes.io/projected/3442d020-91dc-4674-ae3a-669407a861df-kube-api-access-rxlmf\") on node \"ip-10-0-128-146.ec2.internal\" DevicePath \"\""
May 25 11:21:52.193691 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:21:52.193687    2560 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3442d020-91dc-4674-ae3a-669407a861df-bundle\") on node \"ip-10-0-128-146.ec2.internal\" DevicePath \"\""
May 25 11:21:52.193691 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:21:52.193697    2560 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3442d020-91dc-4674-ae3a-669407a861df-util\") on node \"ip-10-0-128-146.ec2.internal\" DevicePath \"\""
May 25 11:21:52.850618 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:21:52.850583    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c9m2qd" event={"ID":"3442d020-91dc-4674-ae3a-669407a861df","Type":"ContainerDied","Data":"170cefeb7b5ee20e93d11fd9e577c1584c68a8df007bc0fd9e2650b9e8961bd9"}
May 25 11:21:52.850618 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:21:52.850614    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="170cefeb7b5ee20e93d11fd9e577c1584c68a8df007bc0fd9e2650b9e8961bd9"
May 25 11:21:52.850858 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:21:52.850625    2560 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c9m2qd"
May 25 11:21:57.008294 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:21:57.008259    2560 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-w9cfr"]
May 25 11:21:57.008726 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:21:57.008537    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3442d020-91dc-4674-ae3a-669407a861df" containerName="util"
May 25 11:21:57.008726 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:21:57.008547    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="3442d020-91dc-4674-ae3a-669407a861df" containerName="util"
May 25 11:21:57.008726 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:21:57.008554    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3442d020-91dc-4674-ae3a-669407a861df" containerName="pull"
May 25 11:21:57.008726 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:21:57.008561    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="3442d020-91dc-4674-ae3a-669407a861df" containerName="pull"
May 25 11:21:57.008726 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:21:57.008568    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3442d020-91dc-4674-ae3a-669407a861df" containerName="extract"
May 25 11:21:57.008726 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:21:57.008573    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="3442d020-91dc-4674-ae3a-669407a861df" containerName="extract"
May 25 11:21:57.008726 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:21:57.008617    2560 memory_manager.go:356] "RemoveStaleState removing state" podUID="3442d020-91dc-4674-ae3a-669407a861df" containerName="extract"
May 25 11:21:57.052826 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:21:57.052796    2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-w9cfr"]
May 25 11:21:57.052978 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:21:57.052907    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-w9cfr"
May 25 11:21:57.055789 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:21:57.055767    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
May 25 11:21:57.055916 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:21:57.055897    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"custom-metrics-autoscaler-operator-dockercfg-sp8rx\""
May 25 11:21:57.056175 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:21:57.056160    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
May 25 11:21:57.056501 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:21:57.056485    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
May 25 11:21:57.138765 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:21:57.138727    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8tpqn\" (UniqueName: \"kubernetes.io/projected/5f2090fc-ccef-4e5c-a732-285db3b3725e-kube-api-access-8tpqn\") pod \"custom-metrics-autoscaler-operator-bbf89fd5d-w9cfr\" (UID: \"5f2090fc-ccef-4e5c-a732-285db3b3725e\") " pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-w9cfr"
May 25 11:21:57.138925 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:21:57.138844    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/5f2090fc-ccef-4e5c-a732-285db3b3725e-certificates\") pod \"custom-metrics-autoscaler-operator-bbf89fd5d-w9cfr\" (UID: \"5f2090fc-ccef-4e5c-a732-285db3b3725e\") " pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-w9cfr"
May 25 11:21:57.239826 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:21:57.239795    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/5f2090fc-ccef-4e5c-a732-285db3b3725e-certificates\") pod \"custom-metrics-autoscaler-operator-bbf89fd5d-w9cfr\" (UID: \"5f2090fc-ccef-4e5c-a732-285db3b3725e\") " pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-w9cfr"
May 25 11:21:57.239979 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:21:57.239842    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-8tpqn\" (UniqueName: \"kubernetes.io/projected/5f2090fc-ccef-4e5c-a732-285db3b3725e-kube-api-access-8tpqn\") pod \"custom-metrics-autoscaler-operator-bbf89fd5d-w9cfr\" (UID: \"5f2090fc-ccef-4e5c-a732-285db3b3725e\") " pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-w9cfr"
May 25 11:21:57.242062 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:21:57.242043    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/5f2090fc-ccef-4e5c-a732-285db3b3725e-certificates\") pod \"custom-metrics-autoscaler-operator-bbf89fd5d-w9cfr\" (UID: \"5f2090fc-ccef-4e5c-a732-285db3b3725e\") " pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-w9cfr"
May 25 11:21:57.248708 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:21:57.248685    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tpqn\" (UniqueName: \"kubernetes.io/projected/5f2090fc-ccef-4e5c-a732-285db3b3725e-kube-api-access-8tpqn\") pod \"custom-metrics-autoscaler-operator-bbf89fd5d-w9cfr\" (UID: \"5f2090fc-ccef-4e5c-a732-285db3b3725e\") " pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-w9cfr"
May 25 11:21:57.363244 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:21:57.363184    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-w9cfr"
May 25 11:21:57.495249 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:21:57.495224    2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-w9cfr"]
May 25 11:21:57.497517 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:21:57.497485    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f2090fc_ccef_4e5c_a732_285db3b3725e.slice/crio-c04e350baace62a273c85724dddfdf1a80ab31ba2a906055631566f66b658697 WatchSource:0}: Error finding container c04e350baace62a273c85724dddfdf1a80ab31ba2a906055631566f66b658697: Status 404 returned error can't find the container with id c04e350baace62a273c85724dddfdf1a80ab31ba2a906055631566f66b658697
May 25 11:21:57.866427 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:21:57.866396    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-w9cfr" event={"ID":"5f2090fc-ccef-4e5c-a732-285db3b3725e","Type":"ContainerStarted","Data":"c04e350baace62a273c85724dddfdf1a80ab31ba2a906055631566f66b658697"}
May 25 11:22:03.888990 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:22:03.888951    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-w9cfr" event={"ID":"5f2090fc-ccef-4e5c-a732-285db3b3725e","Type":"ContainerStarted","Data":"31bdc0e43542631ce50368669010a75cb7d3f318fa38a62bbe362321aefd60db"}
May 25 11:22:03.889388 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:22:03.889052    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-w9cfr"
May 25 11:22:03.913737 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:22:03.913678    2560 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-w9cfr" podStartSLOduration=1.857119355 podStartE2EDuration="7.913660151s" podCreationTimestamp="2026-05-25 11:21:56 +0000 UTC" firstStartedPulling="2026-05-25 11:21:57.499379882 +0000 UTC m=+259.180357833" lastFinishedPulling="2026-05-25 11:22:03.555920677 +0000 UTC m=+265.236898629" observedRunningTime="2026-05-25 11:22:03.912251192 +0000 UTC m=+265.593229162" watchObservedRunningTime="2026-05-25 11:22:03.913660151 +0000 UTC m=+265.594638127"
May 25 11:22:04.057750 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:22:04.057718    2560 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-operator-ffbb595cb-jr2sr"]
May 25 11:22:04.059756 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:22:04.059740    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-ffbb595cb-jr2sr"
May 25 11:22:04.062313 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:22:04.062285    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
May 25 11:22:04.062447 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:22:04.062352    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-certs\""
May 25 11:22:04.062447 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:22:04.062360    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-542pd\""
May 25 11:22:04.069733 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:22:04.069713    2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-ffbb595cb-jr2sr"]
May 25 11:22:04.197363 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:22:04.197283    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qdtq\" (UniqueName: \"kubernetes.io/projected/eb555e14-6401-44c5-84f9-196553b6d851-kube-api-access-6qdtq\") pod \"keda-operator-ffbb595cb-jr2sr\" (UID: \"eb555e14-6401-44c5-84f9-196553b6d851\") " pod="openshift-keda/keda-operator-ffbb595cb-jr2sr"
May 25 11:22:04.197505 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:22:04.197391    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/eb555e14-6401-44c5-84f9-196553b6d851-certificates\") pod \"keda-operator-ffbb595cb-jr2sr\" (UID: \"eb555e14-6401-44c5-84f9-196553b6d851\") " pod="openshift-keda/keda-operator-ffbb595cb-jr2sr"
May 25 11:22:04.197505 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:22:04.197418    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/eb555e14-6401-44c5-84f9-196553b6d851-cabundle0\") pod \"keda-operator-ffbb595cb-jr2sr\" (UID: \"eb555e14-6401-44c5-84f9-196553b6d851\") " pod="openshift-keda/keda-operator-ffbb595cb-jr2sr"
May 25 11:22:04.298446 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:22:04.298407    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/eb555e14-6401-44c5-84f9-196553b6d851-certificates\") pod \"keda-operator-ffbb595cb-jr2sr\" (UID: \"eb555e14-6401-44c5-84f9-196553b6d851\") " pod="openshift-keda/keda-operator-ffbb595cb-jr2sr"
May 25 11:22:04.298446 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:22:04.298452    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/eb555e14-6401-44c5-84f9-196553b6d851-cabundle0\") pod \"keda-operator-ffbb595cb-jr2sr\" (UID: \"eb555e14-6401-44c5-84f9-196553b6d851\") " pod="openshift-keda/keda-operator-ffbb595cb-jr2sr"
May 25 11:22:04.298703 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:22:04.298486    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-6qdtq\" (UniqueName: \"kubernetes.io/projected/eb555e14-6401-44c5-84f9-196553b6d851-kube-api-access-6qdtq\") pod \"keda-operator-ffbb595cb-jr2sr\" (UID: \"eb555e14-6401-44c5-84f9-196553b6d851\") " pod="openshift-keda/keda-operator-ffbb595cb-jr2sr"
May 25 11:22:04.298703 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:22:04.298581    2560 projected.go:264] Couldn't get secret openshift-keda/keda-operator-certs: secret "keda-operator-certs" not found
May 25 11:22:04.298703 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:22:04.298608    2560 secret.go:281] references non-existent secret key: ca.crt
May 25 11:22:04.298703 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:22:04.298617    2560 projected.go:277] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
May 25 11:22:04.298703 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:22:04.298633    2560 projected.go:194] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-ffbb595cb-jr2sr: [secret "keda-operator-certs" not found, references non-existent secret key: ca.crt]
May 25 11:22:04.298958 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:22:04.298708    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/eb555e14-6401-44c5-84f9-196553b6d851-certificates podName:eb555e14-6401-44c5-84f9-196553b6d851 nodeName:}" failed. No retries permitted until 2026-05-25 11:22:04.798687163 +0000 UTC m=+266.479665128 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/eb555e14-6401-44c5-84f9-196553b6d851-certificates") pod "keda-operator-ffbb595cb-jr2sr" (UID: "eb555e14-6401-44c5-84f9-196553b6d851") : [secret "keda-operator-certs" not found, references non-existent secret key: ca.crt]
May 25 11:22:04.299002 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:22:04.298989    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/eb555e14-6401-44c5-84f9-196553b6d851-cabundle0\") pod \"keda-operator-ffbb595cb-jr2sr\" (UID: \"eb555e14-6401-44c5-84f9-196553b6d851\") " pod="openshift-keda/keda-operator-ffbb595cb-jr2sr"
May 25 11:22:04.307489 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:22:04.307468    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qdtq\" (UniqueName: \"kubernetes.io/projected/eb555e14-6401-44c5-84f9-196553b6d851-kube-api-access-6qdtq\") pod \"keda-operator-ffbb595cb-jr2sr\" (UID: \"eb555e14-6401-44c5-84f9-196553b6d851\") " pod="openshift-keda/keda-operator-ffbb595cb-jr2sr"
May 25 11:22:04.423712 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:22:04.423684    2560 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-metrics-apiserver-7c9f485588-5s8kl"]
May 25 11:22:04.425888 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:22:04.425873    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-7c9f485588-5s8kl"
May 25 11:22:04.428396 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:22:04.428371    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-metrics-apiserver-certs\""
May 25 11:22:04.434947 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:22:04.434924    2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-7c9f485588-5s8kl"]
May 25 11:22:04.500824 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:22:04.500752    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hjjl\" (UniqueName: \"kubernetes.io/projected/b4c9b5cc-e244-49ad-ab64-f3e5775162e8-kube-api-access-6hjjl\") pod \"keda-metrics-apiserver-7c9f485588-5s8kl\" (UID: \"b4c9b5cc-e244-49ad-ab64-f3e5775162e8\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-5s8kl"
May 25 11:22:04.500824 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:22:04.500797    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/b4c9b5cc-e244-49ad-ab64-f3e5775162e8-certificates\") pod \"keda-metrics-apiserver-7c9f485588-5s8kl\" (UID: \"b4c9b5cc-e244-49ad-ab64-f3e5775162e8\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-5s8kl"
May 25 11:22:04.500824 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:22:04.500817    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/b4c9b5cc-e244-49ad-ab64-f3e5775162e8-temp-vol\") pod \"keda-metrics-apiserver-7c9f485588-5s8kl\" (UID: \"b4c9b5cc-e244-49ad-ab64-f3e5775162e8\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-5s8kl"
May 25 11:22:04.601725 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:22:04.601693    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-6hjjl\" (UniqueName: \"kubernetes.io/projected/b4c9b5cc-e244-49ad-ab64-f3e5775162e8-kube-api-access-6hjjl\") pod \"keda-metrics-apiserver-7c9f485588-5s8kl\" (UID: \"b4c9b5cc-e244-49ad-ab64-f3e5775162e8\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-5s8kl"
May 25 11:22:04.601878 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:22:04.601740    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/b4c9b5cc-e244-49ad-ab64-f3e5775162e8-certificates\") pod \"keda-metrics-apiserver-7c9f485588-5s8kl\" (UID: \"b4c9b5cc-e244-49ad-ab64-f3e5775162e8\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-5s8kl"
May 25 11:22:04.601878 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:22:04.601820    2560 secret.go:281] references non-existent secret key: tls.crt
May 25 11:22:04.601878 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:22:04.601831    2560 projected.go:277] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: tls.crt
May 25 11:22:04.601878 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:22:04.601850    2560 projected.go:194] Error preparing data for projected volume certificates for pod openshift-keda/keda-metrics-apiserver-7c9f485588-5s8kl: references non-existent secret key: tls.crt
May 25 11:22:04.602007 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:22:04.601890    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b4c9b5cc-e244-49ad-ab64-f3e5775162e8-certificates podName:b4c9b5cc-e244-49ad-ab64-f3e5775162e8 nodeName:}" failed. No retries permitted until 2026-05-25 11:22:05.101877237 +0000 UTC m=+266.782855188 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/b4c9b5cc-e244-49ad-ab64-f3e5775162e8-certificates") pod "keda-metrics-apiserver-7c9f485588-5s8kl" (UID: "b4c9b5cc-e244-49ad-ab64-f3e5775162e8") : references non-existent secret key: tls.crt
May 25 11:22:04.602007 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:22:04.601902    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/b4c9b5cc-e244-49ad-ab64-f3e5775162e8-temp-vol\") pod \"keda-metrics-apiserver-7c9f485588-5s8kl\" (UID: \"b4c9b5cc-e244-49ad-ab64-f3e5775162e8\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-5s8kl"
May 25 11:22:04.602318 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:22:04.602301    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/b4c9b5cc-e244-49ad-ab64-f3e5775162e8-temp-vol\") pod \"keda-metrics-apiserver-7c9f485588-5s8kl\" (UID: \"b4c9b5cc-e244-49ad-ab64-f3e5775162e8\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-5s8kl"
May 25 11:22:04.614492 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:22:04.614471    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hjjl\" (UniqueName: \"kubernetes.io/projected/b4c9b5cc-e244-49ad-ab64-f3e5775162e8-kube-api-access-6hjjl\") pod \"keda-metrics-apiserver-7c9f485588-5s8kl\" (UID: \"b4c9b5cc-e244-49ad-ab64-f3e5775162e8\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-5s8kl"
May 25 11:22:04.635191 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:22:04.635166    2560 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-admission-cf49989db-qf5kj"]
May 25 11:22:04.637272 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:22:04.637257    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-cf49989db-qf5kj"
May 25 11:22:04.640168 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:22:04.640152    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-admission-webhooks-certs\""
May 25 11:22:04.653093 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:22:04.653048    2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-cf49989db-qf5kj"]
May 25 11:22:04.803768 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:22:04.803684    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/510f6e0c-58bd-4ce8-ac94-9059d7e7d3b1-certificates\") pod \"keda-admission-cf49989db-qf5kj\" (UID: \"510f6e0c-58bd-4ce8-ac94-9059d7e7d3b1\") " pod="openshift-keda/keda-admission-cf49989db-qf5kj"
May 25 11:22:04.803768 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:22:04.803756    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/eb555e14-6401-44c5-84f9-196553b6d851-certificates\") pod \"keda-operator-ffbb595cb-jr2sr\" (UID: \"eb555e14-6401-44c5-84f9-196553b6d851\") " pod="openshift-keda/keda-operator-ffbb595cb-jr2sr"
May 25 11:22:04.803947 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:22:04.803822    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6mcw\" (UniqueName: \"kubernetes.io/projected/510f6e0c-58bd-4ce8-ac94-9059d7e7d3b1-kube-api-access-g6mcw\") pod \"keda-admission-cf49989db-qf5kj\" (UID: \"510f6e0c-58bd-4ce8-ac94-9059d7e7d3b1\") " pod="openshift-keda/keda-admission-cf49989db-qf5kj"
May 25 11:22:04.803947 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:22:04.803850    2560 secret.go:281] references non-existent secret key: ca.crt
May 25 11:22:04.803947 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:22:04.803863    2560 projected.go:277] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
May 25 11:22:04.803947 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:22:04.803871    2560 projected.go:194] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-ffbb595cb-jr2sr: references non-existent secret key: ca.crt
May 25 11:22:04.803947 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:22:04.803921    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/eb555e14-6401-44c5-84f9-196553b6d851-certificates podName:eb555e14-6401-44c5-84f9-196553b6d851 nodeName:}" failed. No retries permitted until 2026-05-25 11:22:05.803908286 +0000 UTC m=+267.484886237 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/eb555e14-6401-44c5-84f9-196553b6d851-certificates") pod "keda-operator-ffbb595cb-jr2sr" (UID: "eb555e14-6401-44c5-84f9-196553b6d851") : references non-existent secret key: ca.crt
May 25 11:22:04.905203 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:22:04.905173    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-g6mcw\" (UniqueName: \"kubernetes.io/projected/510f6e0c-58bd-4ce8-ac94-9059d7e7d3b1-kube-api-access-g6mcw\") pod \"keda-admission-cf49989db-qf5kj\" (UID: \"510f6e0c-58bd-4ce8-ac94-9059d7e7d3b1\") " pod="openshift-keda/keda-admission-cf49989db-qf5kj"
May 25 11:22:04.905583 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:22:04.905270    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/510f6e0c-58bd-4ce8-ac94-9059d7e7d3b1-certificates\") pod \"keda-admission-cf49989db-qf5kj\" (UID: \"510f6e0c-58bd-4ce8-ac94-9059d7e7d3b1\") " pod="openshift-keda/keda-admission-cf49989db-qf5kj"
May 25 11:22:04.907723 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:22:04.907703    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/510f6e0c-58bd-4ce8-ac94-9059d7e7d3b1-certificates\") pod \"keda-admission-cf49989db-qf5kj\" (UID: \"510f6e0c-58bd-4ce8-ac94-9059d7e7d3b1\") " pod="openshift-keda/keda-admission-cf49989db-qf5kj"
May 25 11:22:04.915026 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:22:04.915003    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6mcw\" (UniqueName: \"kubernetes.io/projected/510f6e0c-58bd-4ce8-ac94-9059d7e7d3b1-kube-api-access-g6mcw\") pod \"keda-admission-cf49989db-qf5kj\" (UID: \"510f6e0c-58bd-4ce8-ac94-9059d7e7d3b1\") " pod="openshift-keda/keda-admission-cf49989db-qf5kj"
May 25 11:22:04.955796 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:22:04.955762    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-cf49989db-qf5kj"
May 25 11:22:05.084452 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:22:05.084347    2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-cf49989db-qf5kj"]
May 25 11:22:05.089514 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:22:05.089469    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod510f6e0c_58bd_4ce8_ac94_9059d7e7d3b1.slice/crio-aa3807043665531111c960b44aa4972ccbde2cb52b08b596f1af28b0c654152f WatchSource:0}: Error finding container aa3807043665531111c960b44aa4972ccbde2cb52b08b596f1af28b0c654152f: Status 404 returned error can't find the container with id aa3807043665531111c960b44aa4972ccbde2cb52b08b596f1af28b0c654152f
May 25 11:22:05.107985 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:22:05.107962    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/b4c9b5cc-e244-49ad-ab64-f3e5775162e8-certificates\") pod \"keda-metrics-apiserver-7c9f485588-5s8kl\" (UID: \"b4c9b5cc-e244-49ad-ab64-f3e5775162e8\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-5s8kl"
May 25 11:22:05.108104 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:22:05.108082    2560 secret.go:281] references non-existent secret key: tls.crt
May 25 11:22:05.108104 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:22:05.108095    2560 projected.go:277] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: tls.crt
May 25 11:22:05.108170 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:22:05.108111    2560 projected.go:194] Error preparing data for projected volume certificates for pod openshift-keda/keda-metrics-apiserver-7c9f485588-5s8kl: references non-existent secret key: tls.crt
May 25 11:22:05.108242 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:22:05.108169    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b4c9b5cc-e244-49ad-ab64-f3e5775162e8-certificates podName:b4c9b5cc-e244-49ad-ab64-f3e5775162e8 nodeName:}" failed. No retries permitted until 2026-05-25 11:22:06.108156375 +0000 UTC m=+267.789134326 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/b4c9b5cc-e244-49ad-ab64-f3e5775162e8-certificates") pod "keda-metrics-apiserver-7c9f485588-5s8kl" (UID: "b4c9b5cc-e244-49ad-ab64-f3e5775162e8") : references non-existent secret key: tls.crt
May 25 11:22:05.815680 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:22:05.815648    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/eb555e14-6401-44c5-84f9-196553b6d851-certificates\") pod \"keda-operator-ffbb595cb-jr2sr\" (UID: \"eb555e14-6401-44c5-84f9-196553b6d851\") " pod="openshift-keda/keda-operator-ffbb595cb-jr2sr"
May 25 11:22:05.815869 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:22:05.815814    2560 secret.go:281] references non-existent secret key: ca.crt
May 25 11:22:05.815869 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:22:05.815837    2560 projected.go:277] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
May 25 11:22:05.815869 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:22:05.815847    2560 projected.go:194] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-ffbb595cb-jr2sr: references non-existent secret key: ca.crt
May 25 11:22:05.816039 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:22:05.815904    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/eb555e14-6401-44c5-84f9-196553b6d851-certificates podName:eb555e14-6401-44c5-84f9-196553b6d851 nodeName:}" failed. No retries permitted until 2026-05-25 11:22:07.815890217 +0000 UTC m=+269.496868172 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/eb555e14-6401-44c5-84f9-196553b6d851-certificates") pod "keda-operator-ffbb595cb-jr2sr" (UID: "eb555e14-6401-44c5-84f9-196553b6d851") : references non-existent secret key: ca.crt
May 25 11:22:05.896792 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:22:05.896755    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-cf49989db-qf5kj" event={"ID":"510f6e0c-58bd-4ce8-ac94-9059d7e7d3b1","Type":"ContainerStarted","Data":"aa3807043665531111c960b44aa4972ccbde2cb52b08b596f1af28b0c654152f"}
May 25 11:22:06.118643 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:22:06.118603    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/b4c9b5cc-e244-49ad-ab64-f3e5775162e8-certificates\") pod \"keda-metrics-apiserver-7c9f485588-5s8kl\" (UID: \"b4c9b5cc-e244-49ad-ab64-f3e5775162e8\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-5s8kl"
May 25 11:22:06.119111 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:22:06.118734    2560 secret.go:281] references non-existent secret key: tls.crt
May 25 11:22:06.119111 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:22:06.118748    2560 projected.go:277] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: tls.crt
May 25 11:22:06.119111 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:22:06.118763    2560 projected.go:194] Error preparing data for projected volume certificates for pod openshift-keda/keda-metrics-apiserver-7c9f485588-5s8kl: references non-existent secret key: tls.crt
May 25 11:22:06.119111 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:22:06.118813    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b4c9b5cc-e244-49ad-ab64-f3e5775162e8-certificates podName:b4c9b5cc-e244-49ad-ab64-f3e5775162e8 nodeName:}" failed. No retries permitted until 2026-05-25 11:22:08.118799259 +0000 UTC m=+269.799777210 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/b4c9b5cc-e244-49ad-ab64-f3e5775162e8-certificates") pod "keda-metrics-apiserver-7c9f485588-5s8kl" (UID: "b4c9b5cc-e244-49ad-ab64-f3e5775162e8") : references non-existent secret key: tls.crt
May 25 11:22:06.901188 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:22:06.901148    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-cf49989db-qf5kj" event={"ID":"510f6e0c-58bd-4ce8-ac94-9059d7e7d3b1","Type":"ContainerStarted","Data":"df89a192fabb0cbee3da317523a140d14381cd26cd61ccde8d4b96fe35a9ec68"}
May 25 11:22:06.901373 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:22:06.901257    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-admission-cf49989db-qf5kj"
May 25 11:22:06.920439 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:22:06.920395    2560 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-keda/keda-admission-cf49989db-qf5kj" podStartSLOduration=1.753900764 podStartE2EDuration="2.920380056s" podCreationTimestamp="2026-05-25 11:22:04 +0000 UTC" firstStartedPulling="2026-05-25 11:22:05.090743444 +0000 UTC m=+266.771721395" lastFinishedPulling="2026-05-25 11:22:06.257222722 +0000 UTC m=+267.938200687" observedRunningTime="2026-05-25 11:22:06.919025868 +0000 UTC m=+268.600003858" watchObservedRunningTime="2026-05-25 11:22:06.920380056 +0000 UTC m=+268.601358033"
May 25 11:22:07.833396 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:22:07.833353    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/eb555e14-6401-44c5-84f9-196553b6d851-certificates\") pod \"keda-operator-ffbb595cb-jr2sr\" (UID: \"eb555e14-6401-44c5-84f9-196553b6d851\") " pod="openshift-keda/keda-operator-ffbb595cb-jr2sr"
May 25 11:22:07.833878 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:22:07.833494    2560 secret.go:281] references non-existent secret key: ca.crt
May 25 11:22:07.833878 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:22:07.833513    2560 projected.go:277] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
May 25 11:22:07.833878 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:22:07.833522    2560 projected.go:194] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-ffbb595cb-jr2sr: references non-existent secret key: ca.crt
May 25 11:22:07.833878 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:22:07.833570    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/eb555e14-6401-44c5-84f9-196553b6d851-certificates podName:eb555e14-6401-44c5-84f9-196553b6d851 nodeName:}" failed. No retries permitted until 2026-05-25 11:22:11.833556613 +0000 UTC m=+273.514534565 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/eb555e14-6401-44c5-84f9-196553b6d851-certificates") pod "keda-operator-ffbb595cb-jr2sr" (UID: "eb555e14-6401-44c5-84f9-196553b6d851") : references non-existent secret key: ca.crt
May 25 11:22:08.135571 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:22:08.135537    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/b4c9b5cc-e244-49ad-ab64-f3e5775162e8-certificates\") pod \"keda-metrics-apiserver-7c9f485588-5s8kl\" (UID: \"b4c9b5cc-e244-49ad-ab64-f3e5775162e8\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-5s8kl"
May 25 11:22:08.135758 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:22:08.135698    2560 secret.go:281] references non-existent secret key: tls.crt
May 25 11:22:08.135758 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:22:08.135721    2560 projected.go:277] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: tls.crt
May 25 11:22:08.135758 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:22:08.135743    2560 projected.go:194] Error preparing data for projected volume certificates for pod openshift-keda/keda-metrics-apiserver-7c9f485588-5s8kl: references non-existent secret key: tls.crt
May 25 11:22:08.135913 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:22:08.135810    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b4c9b5cc-e244-49ad-ab64-f3e5775162e8-certificates podName:b4c9b5cc-e244-49ad-ab64-f3e5775162e8 nodeName:}" failed. No retries permitted until 2026-05-25 11:22:12.135789769 +0000 UTC m=+273.816767741 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/b4c9b5cc-e244-49ad-ab64-f3e5775162e8-certificates") pod "keda-metrics-apiserver-7c9f485588-5s8kl" (UID: "b4c9b5cc-e244-49ad-ab64-f3e5775162e8") : references non-existent secret key: tls.crt
May 25 11:22:11.868824 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:22:11.868777    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/eb555e14-6401-44c5-84f9-196553b6d851-certificates\") pod \"keda-operator-ffbb595cb-jr2sr\" (UID: \"eb555e14-6401-44c5-84f9-196553b6d851\") " pod="openshift-keda/keda-operator-ffbb595cb-jr2sr"
May 25 11:22:11.871255 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:22:11.871231    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/eb555e14-6401-44c5-84f9-196553b6d851-certificates\") pod \"keda-operator-ffbb595cb-jr2sr\" (UID: \"eb555e14-6401-44c5-84f9-196553b6d851\") " pod="openshift-keda/keda-operator-ffbb595cb-jr2sr"
May 25 11:22:12.170550 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:22:12.170454    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-ffbb595cb-jr2sr"
May 25 11:22:12.171051 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:22:12.171029    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/b4c9b5cc-e244-49ad-ab64-f3e5775162e8-certificates\") pod \"keda-metrics-apiserver-7c9f485588-5s8kl\" (UID: \"b4c9b5cc-e244-49ad-ab64-f3e5775162e8\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-5s8kl"
May 25 11:22:12.173481 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:22:12.173458    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/b4c9b5cc-e244-49ad-ab64-f3e5775162e8-certificates\") pod \"keda-metrics-apiserver-7c9f485588-5s8kl\" (UID: \"b4c9b5cc-e244-49ad-ab64-f3e5775162e8\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-5s8kl"
May 25 11:22:12.237505 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:22:12.237468    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-7c9f485588-5s8kl"
May 25 11:22:12.306907 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:22:12.306874    2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-ffbb595cb-jr2sr"]
May 25 11:22:12.308359 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:22:12.308333    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb555e14_6401_44c5_84f9_196553b6d851.slice/crio-ab3b27743d608db9486c54cfc5eee45a66c8aa3e3111f09cdb0613123ccc6ee7 WatchSource:0}: Error finding container ab3b27743d608db9486c54cfc5eee45a66c8aa3e3111f09cdb0613123ccc6ee7: Status 404 returned error can't find the container with id ab3b27743d608db9486c54cfc5eee45a66c8aa3e3111f09cdb0613123ccc6ee7
May 25 11:22:12.365658 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:22:12.365628    2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-7c9f485588-5s8kl"]
May 25 11:22:12.367988 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:22:12.367958    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4c9b5cc_e244_49ad_ab64_f3e5775162e8.slice/crio-b0e6dce6b88b61c4d8b45b01348ddf71be73a9edd6761abc0cf4b9134fae8544 WatchSource:0}: Error finding container b0e6dce6b88b61c4d8b45b01348ddf71be73a9edd6761abc0cf4b9134fae8544: Status 404 returned error can't find the container with id b0e6dce6b88b61c4d8b45b01348ddf71be73a9edd6761abc0cf4b9134fae8544
May 25 11:22:12.919741 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:22:12.919701    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-7c9f485588-5s8kl" event={"ID":"b4c9b5cc-e244-49ad-ab64-f3e5775162e8","Type":"ContainerStarted","Data":"b0e6dce6b88b61c4d8b45b01348ddf71be73a9edd6761abc0cf4b9134fae8544"}
May 25 11:22:12.920683 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:22:12.920659    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-ffbb595cb-jr2sr" event={"ID":"eb555e14-6401-44c5-84f9-196553b6d851","Type":"ContainerStarted","Data":"ab3b27743d608db9486c54cfc5eee45a66c8aa3e3111f09cdb0613123ccc6ee7"}
May 25 11:22:15.932437 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:22:15.932404    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-ffbb595cb-jr2sr" event={"ID":"eb555e14-6401-44c5-84f9-196553b6d851","Type":"ContainerStarted","Data":"12fed159f349427da46ee26d546413c2087ff8811a388553312b07f3b28a6199"}
May 25 11:22:15.932860 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:22:15.932563    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-operator-ffbb595cb-jr2sr"
May 25 11:22:15.950881 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:22:15.950836    2560 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-keda/keda-operator-ffbb595cb-jr2sr" podStartSLOduration=8.853233877 podStartE2EDuration="11.950822535s" podCreationTimestamp="2026-05-25 11:22:04 +0000 UTC" firstStartedPulling="2026-05-25 11:22:12.309574396 +0000 UTC m=+273.990552346" lastFinishedPulling="2026-05-25 11:22:15.407163042 +0000 UTC m=+277.088141004" observedRunningTime="2026-05-25 11:22:15.949071025 +0000 UTC m=+277.630048999" watchObservedRunningTime="2026-05-25 11:22:15.950822535 +0000 UTC m=+277.631800508"
May 25 11:22:16.942017 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:22:16.941979    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-7c9f485588-5s8kl" event={"ID":"b4c9b5cc-e244-49ad-ab64-f3e5775162e8","Type":"ContainerStarted","Data":"ad9b67e3581466c5917e1a5b5e55a83c08cd15f39c4ddae025d4c94938861e00"}
May 25 11:22:16.942428 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:22:16.942233    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-metrics-apiserver-7c9f485588-5s8kl"
May 25 11:22:16.960669 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:22:16.960625    2560 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-keda/keda-metrics-apiserver-7c9f485588-5s8kl" podStartSLOduration=8.861568933000001 podStartE2EDuration="12.960612653s" podCreationTimestamp="2026-05-25 11:22:04 +0000 UTC" firstStartedPulling="2026-05-25 11:22:12.369336278 +0000 UTC m=+274.050314229" lastFinishedPulling="2026-05-25 11:22:16.468379995 +0000 UTC m=+278.149357949" observedRunningTime="2026-05-25 11:22:16.958871368 +0000 UTC m=+278.639849357" watchObservedRunningTime="2026-05-25 11:22:16.960612653 +0000 UTC m=+278.641590626"
May 25 11:22:24.894132 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:22:24.894099    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-w9cfr"
May 25 11:22:27.906951 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:22:27.906921    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-admission-cf49989db-qf5kj"
May 25 11:22:27.951041 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:22:27.951013    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-metrics-apiserver-7c9f485588-5s8kl"
May 25 11:22:36.944099 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:22:36.944073    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-operator-ffbb595cb-jr2sr"
May 25 11:22:38.827241 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:22:38.827190    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-zd54m_2be32661-9288-4902-9d67-1a5e8f74670d/console-operator/1.log"
May 25 11:22:38.830259 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:22:38.830235    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-zd54m_2be32661-9288-4902-9d67-1a5e8f74670d/console-operator/1.log"
May 25 11:22:38.834272 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:22:38.834255    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m55v2_b93f3e9c-f4f9-49dc-bfba-6d0f80c0971d/ovn-acl-logging/0.log"
May 25 11:22:38.837339 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:22:38.837322    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m55v2_b93f3e9c-f4f9-49dc-bfba-6d0f80c0971d/ovn-acl-logging/0.log"
May 25 11:22:38.837666 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:22:38.837651    2560 kubelet.go:1628] "Image garbage collection succeeded"
May 25 11:23:10.103589 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:23:10.103552    2560 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-559cc6dd6f-rn22d"]
May 25 11:23:10.108583 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:23:10.108555    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-559cc6dd6f-rn22d"
May 25 11:23:10.110018 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:23:10.109994    2560 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve/llmisvc-controller-manager-6b94ff949c-hrsrt"]
May 25 11:23:10.112137 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:23:10.112118    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-6b94ff949c-hrsrt"
May 25 11:23:10.112247 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:23:10.112125    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
May 25 11:23:10.112854 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:23:10.112835    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-w5fxw\""
May 25 11:23:10.112854 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:23:10.112850    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
May 25 11:23:10.112965 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:23:10.112855    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\""
May 25 11:23:10.114452 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:23:10.114435    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-webhook-server-cert\""
May 25 11:23:10.115270 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:23:10.115254    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-controller-manager-dockercfg-6zlcz\""
May 25 11:23:10.119469 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:23:10.119449    2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-559cc6dd6f-rn22d"]
May 25 11:23:10.125394 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:23:10.125376    2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-6b94ff949c-hrsrt"]
May 25 11:23:10.240222 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:23:10.240175    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6539f1f2-5123-4750-ba1b-3e6159bdb1a6-cert\") pod \"llmisvc-controller-manager-6b94ff949c-hrsrt\" (UID: \"6539f1f2-5123-4750-ba1b-3e6159bdb1a6\") " pod="kserve/llmisvc-controller-manager-6b94ff949c-hrsrt"
May 25 11:23:10.240409 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:23:10.240229    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7tkjm\" (UniqueName: \"kubernetes.io/projected/6539f1f2-5123-4750-ba1b-3e6159bdb1a6-kube-api-access-7tkjm\") pod \"llmisvc-controller-manager-6b94ff949c-hrsrt\" (UID: \"6539f1f2-5123-4750-ba1b-3e6159bdb1a6\") " pod="kserve/llmisvc-controller-manager-6b94ff949c-hrsrt"
May 25 11:23:10.240409 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:23:10.240356    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d8dc6d2a-9fc3-4af1-9e64-1566efc8da3c-cert\") pod \"kserve-controller-manager-559cc6dd6f-rn22d\" (UID: \"d8dc6d2a-9fc3-4af1-9e64-1566efc8da3c\") " pod="kserve/kserve-controller-manager-559cc6dd6f-rn22d"
May 25 11:23:10.240409 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:23:10.240403    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szk28\" (UniqueName: \"kubernetes.io/projected/d8dc6d2a-9fc3-4af1-9e64-1566efc8da3c-kube-api-access-szk28\") pod \"kserve-controller-manager-559cc6dd6f-rn22d\" (UID: \"d8dc6d2a-9fc3-4af1-9e64-1566efc8da3c\") " pod="kserve/kserve-controller-manager-559cc6dd6f-rn22d"
May 25 11:23:10.341878 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:23:10.341848    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6539f1f2-5123-4750-ba1b-3e6159bdb1a6-cert\") pod \"llmisvc-controller-manager-6b94ff949c-hrsrt\" (UID: \"6539f1f2-5123-4750-ba1b-3e6159bdb1a6\") " pod="kserve/llmisvc-controller-manager-6b94ff949c-hrsrt"
May 25 11:23:10.342059 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:23:10.341884    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-7tkjm\" (UniqueName: \"kubernetes.io/projected/6539f1f2-5123-4750-ba1b-3e6159bdb1a6-kube-api-access-7tkjm\") pod \"llmisvc-controller-manager-6b94ff949c-hrsrt\" (UID: \"6539f1f2-5123-4750-ba1b-3e6159bdb1a6\") " pod="kserve/llmisvc-controller-manager-6b94ff949c-hrsrt"
May 25 11:23:10.342059 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:23:10.341935    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d8dc6d2a-9fc3-4af1-9e64-1566efc8da3c-cert\") pod \"kserve-controller-manager-559cc6dd6f-rn22d\" (UID: \"d8dc6d2a-9fc3-4af1-9e64-1566efc8da3c\") " pod="kserve/kserve-controller-manager-559cc6dd6f-rn22d"
May 25 11:23:10.342059 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:23:10.341957    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-szk28\" (UniqueName: \"kubernetes.io/projected/d8dc6d2a-9fc3-4af1-9e64-1566efc8da3c-kube-api-access-szk28\") pod \"kserve-controller-manager-559cc6dd6f-rn22d\" (UID: \"d8dc6d2a-9fc3-4af1-9e64-1566efc8da3c\") " pod="kserve/kserve-controller-manager-559cc6dd6f-rn22d"
May 25 11:23:10.344502 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:23:10.344477    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6539f1f2-5123-4750-ba1b-3e6159bdb1a6-cert\") pod \"llmisvc-controller-manager-6b94ff949c-hrsrt\" (UID: \"6539f1f2-5123-4750-ba1b-3e6159bdb1a6\") " pod="kserve/llmisvc-controller-manager-6b94ff949c-hrsrt"
May 25 11:23:10.344626 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:23:10.344480    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d8dc6d2a-9fc3-4af1-9e64-1566efc8da3c-cert\") pod \"kserve-controller-manager-559cc6dd6f-rn22d\" (UID: \"d8dc6d2a-9fc3-4af1-9e64-1566efc8da3c\") " pod="kserve/kserve-controller-manager-559cc6dd6f-rn22d"
May 25 11:23:10.351265 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:23:10.351242    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-szk28\" (UniqueName: \"kubernetes.io/projected/d8dc6d2a-9fc3-4af1-9e64-1566efc8da3c-kube-api-access-szk28\") pod \"kserve-controller-manager-559cc6dd6f-rn22d\" (UID: \"d8dc6d2a-9fc3-4af1-9e64-1566efc8da3c\") " pod="kserve/kserve-controller-manager-559cc6dd6f-rn22d"
May 25 11:23:10.351886 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:23:10.351864    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7tkjm\" (UniqueName: \"kubernetes.io/projected/6539f1f2-5123-4750-ba1b-3e6159bdb1a6-kube-api-access-7tkjm\") pod \"llmisvc-controller-manager-6b94ff949c-hrsrt\" (UID: \"6539f1f2-5123-4750-ba1b-3e6159bdb1a6\") " pod="kserve/llmisvc-controller-manager-6b94ff949c-hrsrt"
May 25 11:23:10.420942 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:23:10.420861    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-559cc6dd6f-rn22d"
May 25 11:23:10.426705 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:23:10.426680    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-6b94ff949c-hrsrt"
May 25 11:23:10.576775 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:23:10.576743    2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-559cc6dd6f-rn22d"]
May 25 11:23:10.578357 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:23:10.578329    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8dc6d2a_9fc3_4af1_9e64_1566efc8da3c.slice/crio-8a6c9386d8a852aa7af415fe8d6a3a0672eda2f41e1397a72399812024f4d7e2 WatchSource:0}: Error finding container 8a6c9386d8a852aa7af415fe8d6a3a0672eda2f41e1397a72399812024f4d7e2: Status 404 returned error can't find the container with id 8a6c9386d8a852aa7af415fe8d6a3a0672eda2f41e1397a72399812024f4d7e2
May 25 11:23:10.579835 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:23:10.579817    2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 25 11:23:10.601912 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:23:10.601889    2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-6b94ff949c-hrsrt"]
May 25 11:23:10.604041 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:23:10.604010    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod6539f1f2_5123_4750_ba1b_3e6159bdb1a6.slice/crio-1fff2fe3b201b573b16f0ba72fb10cad603dd1be56396fdcb9b138a9e89d18b1 WatchSource:0}: Error finding container 1fff2fe3b201b573b16f0ba72fb10cad603dd1be56396fdcb9b138a9e89d18b1: Status 404 returned error can't find the container with id 1fff2fe3b201b573b16f0ba72fb10cad603dd1be56396fdcb9b138a9e89d18b1
May 25 11:23:11.114876 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:23:11.114818    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-6b94ff949c-hrsrt" event={"ID":"6539f1f2-5123-4750-ba1b-3e6159bdb1a6","Type":"ContainerStarted","Data":"1fff2fe3b201b573b16f0ba72fb10cad603dd1be56396fdcb9b138a9e89d18b1"}
May 25 11:23:11.116712 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:23:11.116677    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-559cc6dd6f-rn22d" event={"ID":"d8dc6d2a-9fc3-4af1-9e64-1566efc8da3c","Type":"ContainerStarted","Data":"8a6c9386d8a852aa7af415fe8d6a3a0672eda2f41e1397a72399812024f4d7e2"}
May 25 11:23:14.127580 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:23:14.127548    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-6b94ff949c-hrsrt" event={"ID":"6539f1f2-5123-4750-ba1b-3e6159bdb1a6","Type":"ContainerStarted","Data":"a4709ef780524907eb416ce77bd28bb2d85a6bd74685a35639de14e010452d0e"}
May 25 11:23:14.128008 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:23:14.127643    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-6b94ff949c-hrsrt"
May 25 11:23:14.128945 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:23:14.128924    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-559cc6dd6f-rn22d" event={"ID":"d8dc6d2a-9fc3-4af1-9e64-1566efc8da3c","Type":"ContainerStarted","Data":"6cc59012889b765ba07d7cb0cd608f8b8d8d8caf0ecd88f44b9f537d2dbeeac4"}
May 25 11:23:14.129072 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:23:14.129059    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-559cc6dd6f-rn22d"
May 25 11:23:14.159815 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:23:14.159768    2560 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve/kserve-controller-manager-559cc6dd6f-rn22d" podStartSLOduration=1.077766723 podStartE2EDuration="4.159756335s" podCreationTimestamp="2026-05-25 11:23:10 +0000 UTC" firstStartedPulling="2026-05-25 11:23:10.579932929 +0000 UTC m=+332.260910880" lastFinishedPulling="2026-05-25 11:23:13.661922527 +0000 UTC m=+335.342900492" observedRunningTime="2026-05-25 11:23:14.158177179 +0000 UTC m=+335.839155162" watchObservedRunningTime="2026-05-25 11:23:14.159756335 +0000 UTC m=+335.840734354"
May 25 11:23:14.160418 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:23:14.160391    2560 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve/llmisvc-controller-manager-6b94ff949c-hrsrt" podStartSLOduration=1.105856236 podStartE2EDuration="4.160383456s" podCreationTimestamp="2026-05-25 11:23:10 +0000 UTC" firstStartedPulling="2026-05-25 11:23:10.605378885 +0000 UTC m=+332.286356835" lastFinishedPulling="2026-05-25 11:23:13.659906091 +0000 UTC m=+335.340884055" observedRunningTime="2026-05-25 11:23:14.142781588 +0000 UTC m=+335.823759552" watchObservedRunningTime="2026-05-25 11:23:14.160383456 +0000 UTC m=+335.841361431"
May 25 11:23:45.134928 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:23:45.134853    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-6b94ff949c-hrsrt"
May 25 11:23:45.137876 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:23:45.137858    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-559cc6dd6f-rn22d"
May 25 11:23:46.735321 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:23:46.735289    2560 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-559cc6dd6f-rn22d"]
May 25 11:23:46.735711 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:23:46.735500    2560 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve/kserve-controller-manager-559cc6dd6f-rn22d" podUID="d8dc6d2a-9fc3-4af1-9e64-1566efc8da3c" containerName="manager" containerID="cri-o://6cc59012889b765ba07d7cb0cd608f8b8d8d8caf0ecd88f44b9f537d2dbeeac4" gracePeriod=10
May 25 11:23:46.759063 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:23:46.759042    2560 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-559cc6dd6f-qxcs2"]
May 25 11:23:46.761291 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:23:46.761278    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-559cc6dd6f-qxcs2"
May 25 11:23:46.769465 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:23:46.769435    2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-559cc6dd6f-qxcs2"]
May 25 11:23:46.840711 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:23:46.840685    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f2b8ad4e-8b6d-494d-88f6-f1e516d7f58b-cert\") pod \"kserve-controller-manager-559cc6dd6f-qxcs2\" (UID: \"f2b8ad4e-8b6d-494d-88f6-f1e516d7f58b\") " pod="kserve/kserve-controller-manager-559cc6dd6f-qxcs2"
May 25 11:23:46.840869 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:23:46.840721    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fp5v9\" (UniqueName: \"kubernetes.io/projected/f2b8ad4e-8b6d-494d-88f6-f1e516d7f58b-kube-api-access-fp5v9\") pod \"kserve-controller-manager-559cc6dd6f-qxcs2\" (UID: \"f2b8ad4e-8b6d-494d-88f6-f1e516d7f58b\") " pod="kserve/kserve-controller-manager-559cc6dd6f-qxcs2"
May 25 11:23:46.941806 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:23:46.941771    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-fp5v9\" (UniqueName: \"kubernetes.io/projected/f2b8ad4e-8b6d-494d-88f6-f1e516d7f58b-kube-api-access-fp5v9\") pod \"kserve-controller-manager-559cc6dd6f-qxcs2\" (UID: \"f2b8ad4e-8b6d-494d-88f6-f1e516d7f58b\") " pod="kserve/kserve-controller-manager-559cc6dd6f-qxcs2"
May 25 11:23:46.941950 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:23:46.941867    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f2b8ad4e-8b6d-494d-88f6-f1e516d7f58b-cert\") pod \"kserve-controller-manager-559cc6dd6f-qxcs2\" (UID: \"f2b8ad4e-8b6d-494d-88f6-f1e516d7f58b\") " pod="kserve/kserve-controller-manager-559cc6dd6f-qxcs2"
May 25 11:23:46.944310 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:23:46.944287    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f2b8ad4e-8b6d-494d-88f6-f1e516d7f58b-cert\") pod \"kserve-controller-manager-559cc6dd6f-qxcs2\" (UID: \"f2b8ad4e-8b6d-494d-88f6-f1e516d7f58b\") " pod="kserve/kserve-controller-manager-559cc6dd6f-qxcs2"
May 25 11:23:46.950227 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:23:46.950193    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fp5v9\" (UniqueName: \"kubernetes.io/projected/f2b8ad4e-8b6d-494d-88f6-f1e516d7f58b-kube-api-access-fp5v9\") pod \"kserve-controller-manager-559cc6dd6f-qxcs2\" (UID: \"f2b8ad4e-8b6d-494d-88f6-f1e516d7f58b\") " pod="kserve/kserve-controller-manager-559cc6dd6f-qxcs2"
May 25 11:23:46.973263 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:23:46.973237    2560 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-559cc6dd6f-rn22d"
May 25 11:23:47.042864 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:23:47.042792    2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d8dc6d2a-9fc3-4af1-9e64-1566efc8da3c-cert\") pod \"d8dc6d2a-9fc3-4af1-9e64-1566efc8da3c\" (UID: \"d8dc6d2a-9fc3-4af1-9e64-1566efc8da3c\") "
May 25 11:23:47.042992 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:23:47.042885    2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-szk28\" (UniqueName: \"kubernetes.io/projected/d8dc6d2a-9fc3-4af1-9e64-1566efc8da3c-kube-api-access-szk28\") pod \"d8dc6d2a-9fc3-4af1-9e64-1566efc8da3c\" (UID: \"d8dc6d2a-9fc3-4af1-9e64-1566efc8da3c\") "
May 25 11:23:47.044874 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:23:47.044848    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8dc6d2a-9fc3-4af1-9e64-1566efc8da3c-cert" (OuterVolumeSpecName: "cert") pod "d8dc6d2a-9fc3-4af1-9e64-1566efc8da3c" (UID: "d8dc6d2a-9fc3-4af1-9e64-1566efc8da3c"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 11:23:47.044988 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:23:47.044914    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8dc6d2a-9fc3-4af1-9e64-1566efc8da3c-kube-api-access-szk28" (OuterVolumeSpecName: "kube-api-access-szk28") pod "d8dc6d2a-9fc3-4af1-9e64-1566efc8da3c" (UID: "d8dc6d2a-9fc3-4af1-9e64-1566efc8da3c"). InnerVolumeSpecName "kube-api-access-szk28". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 11:23:47.114028 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:23:47.113985    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-559cc6dd6f-qxcs2"
May 25 11:23:47.143759 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:23:47.143731    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-szk28\" (UniqueName: \"kubernetes.io/projected/d8dc6d2a-9fc3-4af1-9e64-1566efc8da3c-kube-api-access-szk28\") on node \"ip-10-0-128-146.ec2.internal\" DevicePath \"\""
May 25 11:23:47.143759 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:23:47.143757    2560 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d8dc6d2a-9fc3-4af1-9e64-1566efc8da3c-cert\") on node \"ip-10-0-128-146.ec2.internal\" DevicePath \"\""
May 25 11:23:47.232474 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:23:47.232442    2560 generic.go:358] "Generic (PLEG): container finished" podID="d8dc6d2a-9fc3-4af1-9e64-1566efc8da3c" containerID="6cc59012889b765ba07d7cb0cd608f8b8d8d8caf0ecd88f44b9f537d2dbeeac4" exitCode=0
May 25 11:23:47.232615 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:23:47.232497    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-559cc6dd6f-rn22d" event={"ID":"d8dc6d2a-9fc3-4af1-9e64-1566efc8da3c","Type":"ContainerDied","Data":"6cc59012889b765ba07d7cb0cd608f8b8d8d8caf0ecd88f44b9f537d2dbeeac4"}
May 25 11:23:47.232615 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:23:47.232512    2560 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-559cc6dd6f-rn22d"
May 25 11:23:47.232615 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:23:47.232537    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-559cc6dd6f-rn22d" event={"ID":"d8dc6d2a-9fc3-4af1-9e64-1566efc8da3c","Type":"ContainerDied","Data":"8a6c9386d8a852aa7af415fe8d6a3a0672eda2f41e1397a72399812024f4d7e2"}
May 25 11:23:47.232615 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:23:47.232554    2560 scope.go:117] "RemoveContainer" containerID="6cc59012889b765ba07d7cb0cd608f8b8d8d8caf0ecd88f44b9f537d2dbeeac4"
May 25 11:23:47.240427 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:23:47.240407    2560 scope.go:117] "RemoveContainer" containerID="6cc59012889b765ba07d7cb0cd608f8b8d8d8caf0ecd88f44b9f537d2dbeeac4"
May 25 11:23:47.240685 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:23:47.240659    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6cc59012889b765ba07d7cb0cd608f8b8d8d8caf0ecd88f44b9f537d2dbeeac4\": container with ID starting with 6cc59012889b765ba07d7cb0cd608f8b8d8d8caf0ecd88f44b9f537d2dbeeac4 not found: ID does not exist" containerID="6cc59012889b765ba07d7cb0cd608f8b8d8d8caf0ecd88f44b9f537d2dbeeac4"
May 25 11:23:47.240787 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:23:47.240694    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6cc59012889b765ba07d7cb0cd608f8b8d8d8caf0ecd88f44b9f537d2dbeeac4"} err="failed to get container status \"6cc59012889b765ba07d7cb0cd608f8b8d8d8caf0ecd88f44b9f537d2dbeeac4\": rpc error: code = NotFound desc = could not find container \"6cc59012889b765ba07d7cb0cd608f8b8d8d8caf0ecd88f44b9f537d2dbeeac4\": container with ID starting with 6cc59012889b765ba07d7cb0cd608f8b8d8d8caf0ecd88f44b9f537d2dbeeac4 not found: ID does not exist"
May 25 11:23:47.259888 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:23:47.259858    2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-559cc6dd6f-qxcs2"]
May 25 11:23:47.262713 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:23:47.262693    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2b8ad4e_8b6d_494d_88f6_f1e516d7f58b.slice/crio-50ca61b7e72affca4bbe0da3ebb32ae54d81e7fd028f5225c4950b31852e5ff8 WatchSource:0}: Error finding container 50ca61b7e72affca4bbe0da3ebb32ae54d81e7fd028f5225c4950b31852e5ff8: Status 404 returned error can't find the container with id 50ca61b7e72affca4bbe0da3ebb32ae54d81e7fd028f5225c4950b31852e5ff8
May 25 11:23:47.271106 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:23:47.271081    2560 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-559cc6dd6f-rn22d"]
May 25 11:23:47.277299 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:23:47.277270    2560 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve/kserve-controller-manager-559cc6dd6f-rn22d"]
May 25 11:23:48.237199 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:23:48.237159    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-559cc6dd6f-qxcs2" event={"ID":"f2b8ad4e-8b6d-494d-88f6-f1e516d7f58b","Type":"ContainerStarted","Data":"a9d59e5548f167fd3ceac19bbcf7ab0192a40a9b52a36077c80fe0601f3e81e5"}
May 25 11:23:48.237199 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:23:48.237199    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-559cc6dd6f-qxcs2" event={"ID":"f2b8ad4e-8b6d-494d-88f6-f1e516d7f58b","Type":"ContainerStarted","Data":"50ca61b7e72affca4bbe0da3ebb32ae54d81e7fd028f5225c4950b31852e5ff8"}
May 25 11:23:48.237652 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:23:48.237266    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-559cc6dd6f-qxcs2"
May 25 11:23:48.256551 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:23:48.256509    2560 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve/kserve-controller-manager-559cc6dd6f-qxcs2" podStartSLOduration=1.9034160610000002 podStartE2EDuration="2.256497178s" podCreationTimestamp="2026-05-25 11:23:46 +0000 UTC" firstStartedPulling="2026-05-25 11:23:47.264247254 +0000 UTC m=+368.945225205" lastFinishedPulling="2026-05-25 11:23:47.617328367 +0000 UTC m=+369.298306322" observedRunningTime="2026-05-25 11:23:48.256246093 +0000 UTC m=+369.937224066" watchObservedRunningTime="2026-05-25 11:23:48.256497178 +0000 UTC m=+369.937475151"
May 25 11:23:48.935034 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:23:48.935000    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d8dc6d2a-9fc3-4af1-9e64-1566efc8da3c" path="/var/lib/kubelet/pods/d8dc6d2a-9fc3-4af1-9e64-1566efc8da3c/volumes"
May 25 11:24:19.245107 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:24:19.245075    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-559cc6dd6f-qxcs2"
May 25 11:24:20.173343 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:24:20.173307    2560 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-86f7b4b499-68fvm"]
May 25 11:24:20.173639 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:24:20.173627    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d8dc6d2a-9fc3-4af1-9e64-1566efc8da3c" containerName="manager"
May 25 11:24:20.173683 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:24:20.173641    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="d8dc6d2a-9fc3-4af1-9e64-1566efc8da3c" containerName="manager"
May 25 11:24:20.173716 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:24:20.173702    2560 memory_manager.go:356] "RemoveStaleState removing state" podUID="d8dc6d2a-9fc3-4af1-9e64-1566efc8da3c" containerName="manager"
May 25 11:24:20.175375 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:24:20.175359    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-86f7b4b499-68fvm"
May 25 11:24:20.179384 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:24:20.179360    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
May 25 11:24:20.179495 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:24:20.179381    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-vfxbk\""
May 25 11:24:20.190399 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:24:20.190377    2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-86f7b4b499-68fvm"]
May 25 11:24:20.217306 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:24:20.217280    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4652z\" (UniqueName: \"kubernetes.io/projected/555cfdd5-04ae-4933-8b5b-24597359444b-kube-api-access-4652z\") pod \"model-serving-api-86f7b4b499-68fvm\" (UID: \"555cfdd5-04ae-4933-8b5b-24597359444b\") " pod="kserve/model-serving-api-86f7b4b499-68fvm"
May 25 11:24:20.217454 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:24:20.217322    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/555cfdd5-04ae-4933-8b5b-24597359444b-tls-certs\") pod \"model-serving-api-86f7b4b499-68fvm\" (UID: \"555cfdd5-04ae-4933-8b5b-24597359444b\") " pod="kserve/model-serving-api-86f7b4b499-68fvm"
May 25 11:24:20.317765 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:24:20.317733    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-4652z\" (UniqueName: \"kubernetes.io/projected/555cfdd5-04ae-4933-8b5b-24597359444b-kube-api-access-4652z\") pod \"model-serving-api-86f7b4b499-68fvm\" (UID: \"555cfdd5-04ae-4933-8b5b-24597359444b\") " pod="kserve/model-serving-api-86f7b4b499-68fvm"
May 25 11:24:20.318159 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:24:20.317776    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/555cfdd5-04ae-4933-8b5b-24597359444b-tls-certs\") pod \"model-serving-api-86f7b4b499-68fvm\" (UID: \"555cfdd5-04ae-4933-8b5b-24597359444b\") " pod="kserve/model-serving-api-86f7b4b499-68fvm"
May 25 11:24:20.320147 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:24:20.320128    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/555cfdd5-04ae-4933-8b5b-24597359444b-tls-certs\") pod \"model-serving-api-86f7b4b499-68fvm\" (UID: \"555cfdd5-04ae-4933-8b5b-24597359444b\") " pod="kserve/model-serving-api-86f7b4b499-68fvm"
May 25 11:24:20.326943 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:24:20.326917    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4652z\" (UniqueName: \"kubernetes.io/projected/555cfdd5-04ae-4933-8b5b-24597359444b-kube-api-access-4652z\") pod \"model-serving-api-86f7b4b499-68fvm\" (UID: \"555cfdd5-04ae-4933-8b5b-24597359444b\") " pod="kserve/model-serving-api-86f7b4b499-68fvm"
May 25 11:24:20.485517 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:24:20.485436    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-86f7b4b499-68fvm"
May 25 11:24:20.610049 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:24:20.610010    2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-86f7b4b499-68fvm"]
May 25 11:24:20.613673 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:24:20.613646    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod555cfdd5_04ae_4933_8b5b_24597359444b.slice/crio-941c073297fbc6aaf3180a7e68ff523c0f858b186233d81297f06c05ca465326 WatchSource:0}: Error finding container 941c073297fbc6aaf3180a7e68ff523c0f858b186233d81297f06c05ca465326: Status 404 returned error can't find the container with id 941c073297fbc6aaf3180a7e68ff523c0f858b186233d81297f06c05ca465326
May 25 11:24:21.350518 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:24:21.350457    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-86f7b4b499-68fvm" event={"ID":"555cfdd5-04ae-4933-8b5b-24597359444b","Type":"ContainerStarted","Data":"941c073297fbc6aaf3180a7e68ff523c0f858b186233d81297f06c05ca465326"}
May 25 11:24:23.358312 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:24:23.358278    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-86f7b4b499-68fvm" event={"ID":"555cfdd5-04ae-4933-8b5b-24597359444b","Type":"ContainerStarted","Data":"c0318b63c8efd4366629750b176d5b15ec903f2fb2e774d11ef5c7e441512406"}
May 25 11:24:23.358691 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:24:23.358339    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-86f7b4b499-68fvm"
May 25 11:24:23.376690 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:24:23.376647    2560 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve/model-serving-api-86f7b4b499-68fvm" podStartSLOduration=1.247647382 podStartE2EDuration="3.376634609s" podCreationTimestamp="2026-05-25 11:24:20 +0000 UTC" firstStartedPulling="2026-05-25 11:24:20.615381655 +0000 UTC m=+402.296359605" lastFinishedPulling="2026-05-25 11:24:22.744368882 +0000 UTC m=+404.425346832" observedRunningTime="2026-05-25 11:24:23.374409201 +0000 UTC m=+405.055387184" watchObservedRunningTime="2026-05-25 11:24:23.376634609 +0000 UTC m=+405.057612581"
May 25 11:24:34.366158 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:24:34.366131    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-86f7b4b499-68fvm"
May 25 11:24:55.393420 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:24:55.393336    2560 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-21751-predictor-6888887675-wtlk4"]
May 25 11:24:55.396121 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:24:55.396105    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-21751-predictor-6888887675-wtlk4"
May 25 11:24:55.398891 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:24:55.398867    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-21751-predictor-serving-cert\""
May 25 11:24:55.399458 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:24:55.399438    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-21751-kube-rbac-proxy-sar-config\""
May 25 11:24:55.400098 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:24:55.400025    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
May 25 11:24:55.400098 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:24:55.400063    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
May 25 11:24:55.400098 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:24:55.400089    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-chxn5\""
May 25 11:24:55.406865 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:24:55.406839    2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-21751-predictor-6888887675-wtlk4"]
May 25 11:24:55.512341 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:24:55.512284    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"error-404-isvc-21751-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c63d88a9-9acf-41e8-a675-83742a1f2064-error-404-isvc-21751-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-21751-predictor-6888887675-wtlk4\" (UID: \"c63d88a9-9acf-41e8-a675-83742a1f2064\") " pod="kserve-ci-e2e-test/error-404-isvc-21751-predictor-6888887675-wtlk4"
May 25 11:24:55.512516 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:24:55.512347    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c63d88a9-9acf-41e8-a675-83742a1f2064-proxy-tls\") pod \"error-404-isvc-21751-predictor-6888887675-wtlk4\" (UID: \"c63d88a9-9acf-41e8-a675-83742a1f2064\") " pod="kserve-ci-e2e-test/error-404-isvc-21751-predictor-6888887675-wtlk4"
May 25 11:24:55.512516 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:24:55.512440    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t96n6\" (UniqueName: \"kubernetes.io/projected/c63d88a9-9acf-41e8-a675-83742a1f2064-kube-api-access-t96n6\") pod \"error-404-isvc-21751-predictor-6888887675-wtlk4\" (UID: \"c63d88a9-9acf-41e8-a675-83742a1f2064\") " pod="kserve-ci-e2e-test/error-404-isvc-21751-predictor-6888887675-wtlk4"
May 25 11:24:55.613320 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:24:55.613284    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-t96n6\" (UniqueName: \"kubernetes.io/projected/c63d88a9-9acf-41e8-a675-83742a1f2064-kube-api-access-t96n6\") pod \"error-404-isvc-21751-predictor-6888887675-wtlk4\" (UID: \"c63d88a9-9acf-41e8-a675-83742a1f2064\") " pod="kserve-ci-e2e-test/error-404-isvc-21751-predictor-6888887675-wtlk4"
May 25 11:24:55.613560 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:24:55.613358    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"error-404-isvc-21751-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c63d88a9-9acf-41e8-a675-83742a1f2064-error-404-isvc-21751-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-21751-predictor-6888887675-wtlk4\" (UID: \"c63d88a9-9acf-41e8-a675-83742a1f2064\") " pod="kserve-ci-e2e-test/error-404-isvc-21751-predictor-6888887675-wtlk4"
May 25 11:24:55.613560 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:24:55.613385    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c63d88a9-9acf-41e8-a675-83742a1f2064-proxy-tls\") pod \"error-404-isvc-21751-predictor-6888887675-wtlk4\" (UID: \"c63d88a9-9acf-41e8-a675-83742a1f2064\") " pod="kserve-ci-e2e-test/error-404-isvc-21751-predictor-6888887675-wtlk4"
May 25 11:24:55.614086 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:24:55.614065    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"error-404-isvc-21751-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c63d88a9-9acf-41e8-a675-83742a1f2064-error-404-isvc-21751-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-21751-predictor-6888887675-wtlk4\" (UID: \"c63d88a9-9acf-41e8-a675-83742a1f2064\") " pod="kserve-ci-e2e-test/error-404-isvc-21751-predictor-6888887675-wtlk4"
May 25 11:24:55.615951 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:24:55.615924    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c63d88a9-9acf-41e8-a675-83742a1f2064-proxy-tls\") pod \"error-404-isvc-21751-predictor-6888887675-wtlk4\" (UID: \"c63d88a9-9acf-41e8-a675-83742a1f2064\") " pod="kserve-ci-e2e-test/error-404-isvc-21751-predictor-6888887675-wtlk4"
May 25 11:24:55.622151 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:24:55.622128    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t96n6\" (UniqueName: \"kubernetes.io/projected/c63d88a9-9acf-41e8-a675-83742a1f2064-kube-api-access-t96n6\") pod \"error-404-isvc-21751-predictor-6888887675-wtlk4\" (UID: \"c63d88a9-9acf-41e8-a675-83742a1f2064\") " pod="kserve-ci-e2e-test/error-404-isvc-21751-predictor-6888887675-wtlk4"
May 25 11:24:55.709447 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:24:55.709357    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-21751-predictor-6888887675-wtlk4"
May 25 11:24:55.836470 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:24:55.836438    2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-21751-predictor-6888887675-wtlk4"]
May 25 11:24:55.839570 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:24:55.839535    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc63d88a9_9acf_41e8_a675_83742a1f2064.slice/crio-3d3ee3ae843275b8284a9354551cdec7e219d38bdae7889773080fcbdab06836 WatchSource:0}: Error finding container 3d3ee3ae843275b8284a9354551cdec7e219d38bdae7889773080fcbdab06836: Status 404 returned error can't find the container with id 3d3ee3ae843275b8284a9354551cdec7e219d38bdae7889773080fcbdab06836
May 25 11:24:56.476473 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:24:56.476416    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-21751-predictor-6888887675-wtlk4" event={"ID":"c63d88a9-9acf-41e8-a675-83742a1f2064","Type":"ContainerStarted","Data":"3d3ee3ae843275b8284a9354551cdec7e219d38bdae7889773080fcbdab06836"}
May 25 11:25:07.519413 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:25:07.519380    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-21751-predictor-6888887675-wtlk4" event={"ID":"c63d88a9-9acf-41e8-a675-83742a1f2064","Type":"ContainerStarted","Data":"386599237f9be94f9d703a8267d5269cb7c8b418c225c0519423fe967626b832"}
May 25 11:25:10.533629 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:25:10.533589    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-21751-predictor-6888887675-wtlk4" event={"ID":"c63d88a9-9acf-41e8-a675-83742a1f2064","Type":"ContainerStarted","Data":"d88858e36cd7609adf271c5910b8d1dcb047f6696a3f44a86bb31269060dd803"}
May 25 11:25:10.534029 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:25:10.533641    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-21751-predictor-6888887675-wtlk4"
May 25 11:25:10.534029 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:25:10.533695    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-21751-predictor-6888887675-wtlk4"
May 25 11:25:10.535196 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:25:10.535154    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-21751-predictor-6888887675-wtlk4" podUID="c63d88a9-9acf-41e8-a675-83742a1f2064" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
May 25 11:25:10.552936 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:25:10.552875    2560 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-21751-predictor-6888887675-wtlk4" podStartSLOduration=1.608253213 podStartE2EDuration="15.552861182s" podCreationTimestamp="2026-05-25 11:24:55 +0000 UTC" firstStartedPulling="2026-05-25 11:24:55.841743946 +0000 UTC m=+437.522721897" lastFinishedPulling="2026-05-25 11:25:09.786351916 +0000 UTC m=+451.467329866" observedRunningTime="2026-05-25 11:25:10.552089433 +0000 UTC m=+452.233067421" watchObservedRunningTime="2026-05-25 11:25:10.552861182 +0000 UTC m=+452.233839156"
May 25 11:25:11.536575 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:25:11.536534    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-21751-predictor-6888887675-wtlk4" podUID="c63d88a9-9acf-41e8-a675-83742a1f2064" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
May 25 11:25:16.540883 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:25:16.540848    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-21751-predictor-6888887675-wtlk4"
May 25 11:25:16.541337 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:25:16.541308    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-21751-predictor-6888887675-wtlk4" podUID="c63d88a9-9acf-41e8-a675-83742a1f2064" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
May 25 11:25:26.541417 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:25:26.541371    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-21751-predictor-6888887675-wtlk4" podUID="c63d88a9-9acf-41e8-a675-83742a1f2064" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
May 25 11:25:36.541376 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:25:36.541333    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-21751-predictor-6888887675-wtlk4" podUID="c63d88a9-9acf-41e8-a675-83742a1f2064" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
May 25 11:25:46.541415 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:25:46.541378    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-21751-predictor-6888887675-wtlk4" podUID="c63d88a9-9acf-41e8-a675-83742a1f2064" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
May 25 11:25:56.542438 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:25:56.542403    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-21751-predictor-6888887675-wtlk4"
May 25 11:26:25.365125 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:26:25.365053    2560 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-21751-predictor-6888887675-wtlk4"]
May 25 11:26:25.365605 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:26:25.365348    2560 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-21751-predictor-6888887675-wtlk4" podUID="c63d88a9-9acf-41e8-a675-83742a1f2064" containerName="kserve-container" containerID="cri-o://386599237f9be94f9d703a8267d5269cb7c8b418c225c0519423fe967626b832" gracePeriod=30
May 25 11:26:25.365605 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:26:25.365398    2560 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-21751-predictor-6888887675-wtlk4" podUID="c63d88a9-9acf-41e8-a675-83742a1f2064" containerName="kube-rbac-proxy" containerID="cri-o://d88858e36cd7609adf271c5910b8d1dcb047f6696a3f44a86bb31269060dd803" gracePeriod=30
May 25 11:26:25.620508 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:26:25.620426    2560 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-55329-predictor-689f64f8cf-7x2km"]
May 25 11:26:25.622828 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:26:25.622814    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-55329-predictor-689f64f8cf-7x2km"
May 25 11:26:25.625462 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:26:25.625445    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-55329-kube-rbac-proxy-sar-config\""
May 25 11:26:25.625543 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:26:25.625494    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-55329-predictor-serving-cert\""
May 25 11:26:25.631954 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:26:25.631935    2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-55329-predictor-689f64f8cf-7x2km"]
May 25 11:26:25.683535 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:26:25.683506    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n57zw\" (UniqueName: \"kubernetes.io/projected/47d7ff78-3506-4b71-931a-1b9788def94b-kube-api-access-n57zw\") pod \"error-404-isvc-55329-predictor-689f64f8cf-7x2km\" (UID: \"47d7ff78-3506-4b71-931a-1b9788def94b\") " pod="kserve-ci-e2e-test/error-404-isvc-55329-predictor-689f64f8cf-7x2km"
May 25 11:26:25.683718 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:26:25.683580    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"error-404-isvc-55329-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/47d7ff78-3506-4b71-931a-1b9788def94b-error-404-isvc-55329-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-55329-predictor-689f64f8cf-7x2km\" (UID: \"47d7ff78-3506-4b71-931a-1b9788def94b\") " pod="kserve-ci-e2e-test/error-404-isvc-55329-predictor-689f64f8cf-7x2km"
May 25 11:26:25.683718 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:26:25.683645    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/47d7ff78-3506-4b71-931a-1b9788def94b-proxy-tls\") pod \"error-404-isvc-55329-predictor-689f64f8cf-7x2km\" (UID: \"47d7ff78-3506-4b71-931a-1b9788def94b\") " pod="kserve-ci-e2e-test/error-404-isvc-55329-predictor-689f64f8cf-7x2km"
May 25 11:26:25.781786 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:26:25.781749    2560 generic.go:358] "Generic (PLEG): container finished" podID="c63d88a9-9acf-41e8-a675-83742a1f2064" containerID="d88858e36cd7609adf271c5910b8d1dcb047f6696a3f44a86bb31269060dd803" exitCode=2
May 25 11:26:25.781945 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:26:25.781824    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-21751-predictor-6888887675-wtlk4" event={"ID":"c63d88a9-9acf-41e8-a675-83742a1f2064","Type":"ContainerDied","Data":"d88858e36cd7609adf271c5910b8d1dcb047f6696a3f44a86bb31269060dd803"}
May 25 11:26:25.784179 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:26:25.784162    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-n57zw\" (UniqueName: \"kubernetes.io/projected/47d7ff78-3506-4b71-931a-1b9788def94b-kube-api-access-n57zw\") pod \"error-404-isvc-55329-predictor-689f64f8cf-7x2km\" (UID: \"47d7ff78-3506-4b71-931a-1b9788def94b\") " pod="kserve-ci-e2e-test/error-404-isvc-55329-predictor-689f64f8cf-7x2km"
May 25 11:26:25.784258 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:26:25.784244    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"error-404-isvc-55329-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/47d7ff78-3506-4b71-931a-1b9788def94b-error-404-isvc-55329-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-55329-predictor-689f64f8cf-7x2km\" (UID: \"47d7ff78-3506-4b71-931a-1b9788def94b\") " pod="kserve-ci-e2e-test/error-404-isvc-55329-predictor-689f64f8cf-7x2km"
May 25 11:26:25.784303 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:26:25.784293    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/47d7ff78-3506-4b71-931a-1b9788def94b-proxy-tls\") pod \"error-404-isvc-55329-predictor-689f64f8cf-7x2km\" (UID: \"47d7ff78-3506-4b71-931a-1b9788def94b\") " pod="kserve-ci-e2e-test/error-404-isvc-55329-predictor-689f64f8cf-7x2km"
May 25 11:26:25.784902 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:26:25.784875    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"error-404-isvc-55329-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/47d7ff78-3506-4b71-931a-1b9788def94b-error-404-isvc-55329-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-55329-predictor-689f64f8cf-7x2km\" (UID: \"47d7ff78-3506-4b71-931a-1b9788def94b\") " pod="kserve-ci-e2e-test/error-404-isvc-55329-predictor-689f64f8cf-7x2km"
May 25 11:26:25.786738 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:26:25.786722    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/47d7ff78-3506-4b71-931a-1b9788def94b-proxy-tls\") pod \"error-404-isvc-55329-predictor-689f64f8cf-7x2km\" (UID: \"47d7ff78-3506-4b71-931a-1b9788def94b\") " pod="kserve-ci-e2e-test/error-404-isvc-55329-predictor-689f64f8cf-7x2km"
May 25 11:26:25.792273 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:26:25.792249    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n57zw\" (UniqueName: \"kubernetes.io/projected/47d7ff78-3506-4b71-931a-1b9788def94b-kube-api-access-n57zw\") pod \"error-404-isvc-55329-predictor-689f64f8cf-7x2km\" (UID: \"47d7ff78-3506-4b71-931a-1b9788def94b\") " pod="kserve-ci-e2e-test/error-404-isvc-55329-predictor-689f64f8cf-7x2km"
May 25 11:26:25.932847 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:26:25.932773    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-55329-predictor-689f64f8cf-7x2km"
May 25 11:26:26.061823 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:26:26.061798    2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-55329-predictor-689f64f8cf-7x2km"]
May 25 11:26:26.063749 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:26:26.063722    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47d7ff78_3506_4b71_931a_1b9788def94b.slice/crio-aab39d974191ed711a2074a9bb85eb2219d666099c871bd611f57617af0c667a WatchSource:0}: Error finding container aab39d974191ed711a2074a9bb85eb2219d666099c871bd611f57617af0c667a: Status 404 returned error can't find the container with id aab39d974191ed711a2074a9bb85eb2219d666099c871bd611f57617af0c667a
May 25 11:26:26.537323 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:26:26.537285    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-21751-predictor-6888887675-wtlk4" podUID="c63d88a9-9acf-41e8-a675-83742a1f2064" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.33:8643/healthz\": dial tcp 10.132.0.33:8643: connect: connection refused"
May 25 11:26:26.541502 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:26:26.541473    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-21751-predictor-6888887675-wtlk4" podUID="c63d88a9-9acf-41e8-a675-83742a1f2064" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
May 25 11:26:26.786776 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:26:26.786742    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-55329-predictor-689f64f8cf-7x2km" event={"ID":"47d7ff78-3506-4b71-931a-1b9788def94b","Type":"ContainerStarted","Data":"6619a5a125d477b88b4460fe8c614d1ce709ea3c55170c9f0e780f6753c0fcd4"}
May 25 11:26:26.786776 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:26:26.786780    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-55329-predictor-689f64f8cf-7x2km" event={"ID":"47d7ff78-3506-4b71-931a-1b9788def94b","Type":"ContainerStarted","Data":"62db5f94662ecd3c08a5f905da0188d689618825d5957a79bc232612f479fc59"}
May 25 11:26:26.787036 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:26:26.786801    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-55329-predictor-689f64f8cf-7x2km"
May 25 11:26:26.787036 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:26:26.786816    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-55329-predictor-689f64f8cf-7x2km"
May 25 11:26:26.787036 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:26:26.786826    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-55329-predictor-689f64f8cf-7x2km" event={"ID":"47d7ff78-3506-4b71-931a-1b9788def94b","Type":"ContainerStarted","Data":"aab39d974191ed711a2074a9bb85eb2219d666099c871bd611f57617af0c667a"}
May 25 11:26:26.788166 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:26:26.788096    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-55329-predictor-689f64f8cf-7x2km" podUID="47d7ff78-3506-4b71-931a-1b9788def94b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
May 25 11:26:26.806021 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:26:26.805973    2560 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-55329-predictor-689f64f8cf-7x2km" podStartSLOduration=1.8059597429999998 podStartE2EDuration="1.805959743s" podCreationTimestamp="2026-05-25 11:26:25 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 11:26:26.803851238 +0000 UTC m=+528.484829210" watchObservedRunningTime="2026-05-25 11:26:26.805959743 +0000 UTC m=+528.486937716"
May 25 11:26:27.790153 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:26:27.790114    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-55329-predictor-689f64f8cf-7x2km" podUID="47d7ff78-3506-4b71-931a-1b9788def94b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
May 25 11:26:28.504640 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:26:28.504610    2560 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-21751-predictor-6888887675-wtlk4"
May 25 11:26:28.607057 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:26:28.606960    2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t96n6\" (UniqueName: \"kubernetes.io/projected/c63d88a9-9acf-41e8-a675-83742a1f2064-kube-api-access-t96n6\") pod \"c63d88a9-9acf-41e8-a675-83742a1f2064\" (UID: \"c63d88a9-9acf-41e8-a675-83742a1f2064\") "
May 25 11:26:28.607057 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:26:28.607027    2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c63d88a9-9acf-41e8-a675-83742a1f2064-proxy-tls\") pod \"c63d88a9-9acf-41e8-a675-83742a1f2064\" (UID: \"c63d88a9-9acf-41e8-a675-83742a1f2064\") "
May 25 11:26:28.607317 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:26:28.607074    2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"error-404-isvc-21751-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c63d88a9-9acf-41e8-a675-83742a1f2064-error-404-isvc-21751-kube-rbac-proxy-sar-config\") pod \"c63d88a9-9acf-41e8-a675-83742a1f2064\" (UID: \"c63d88a9-9acf-41e8-a675-83742a1f2064\") "
May 25 11:26:28.607459 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:26:28.607435    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c63d88a9-9acf-41e8-a675-83742a1f2064-error-404-isvc-21751-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "error-404-isvc-21751-kube-rbac-proxy-sar-config") pod "c63d88a9-9acf-41e8-a675-83742a1f2064" (UID: "c63d88a9-9acf-41e8-a675-83742a1f2064"). InnerVolumeSpecName "error-404-isvc-21751-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 11:26:28.609326 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:26:28.609295    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c63d88a9-9acf-41e8-a675-83742a1f2064-kube-api-access-t96n6" (OuterVolumeSpecName: "kube-api-access-t96n6") pod "c63d88a9-9acf-41e8-a675-83742a1f2064" (UID: "c63d88a9-9acf-41e8-a675-83742a1f2064"). InnerVolumeSpecName "kube-api-access-t96n6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 11:26:28.609430 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:26:28.609300    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c63d88a9-9acf-41e8-a675-83742a1f2064-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c63d88a9-9acf-41e8-a675-83742a1f2064" (UID: "c63d88a9-9acf-41e8-a675-83742a1f2064"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 11:26:28.708109 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:26:28.708076    2560 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c63d88a9-9acf-41e8-a675-83742a1f2064-proxy-tls\") on node \"ip-10-0-128-146.ec2.internal\" DevicePath \"\""
May 25 11:26:28.708109 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:26:28.708105    2560 reconciler_common.go:299] "Volume detached for volume \"error-404-isvc-21751-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c63d88a9-9acf-41e8-a675-83742a1f2064-error-404-isvc-21751-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-146.ec2.internal\" DevicePath \"\""
May 25 11:26:28.708109 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:26:28.708116    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t96n6\" (UniqueName: \"kubernetes.io/projected/c63d88a9-9acf-41e8-a675-83742a1f2064-kube-api-access-t96n6\") on node \"ip-10-0-128-146.ec2.internal\" DevicePath \"\""
May 25 11:26:28.794728 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:26:28.794689    2560 generic.go:358] "Generic (PLEG): container finished" podID="c63d88a9-9acf-41e8-a675-83742a1f2064" containerID="386599237f9be94f9d703a8267d5269cb7c8b418c225c0519423fe967626b832" exitCode=0
May 25 11:26:28.795090 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:26:28.794751    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-21751-predictor-6888887675-wtlk4" event={"ID":"c63d88a9-9acf-41e8-a675-83742a1f2064","Type":"ContainerDied","Data":"386599237f9be94f9d703a8267d5269cb7c8b418c225c0519423fe967626b832"}
May 25 11:26:28.795090 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:26:28.794770    2560 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-21751-predictor-6888887675-wtlk4"
May 25 11:26:28.795090 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:26:28.794786    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-21751-predictor-6888887675-wtlk4" event={"ID":"c63d88a9-9acf-41e8-a675-83742a1f2064","Type":"ContainerDied","Data":"3d3ee3ae843275b8284a9354551cdec7e219d38bdae7889773080fcbdab06836"}
May 25 11:26:28.795090 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:26:28.794805    2560 scope.go:117] "RemoveContainer" containerID="d88858e36cd7609adf271c5910b8d1dcb047f6696a3f44a86bb31269060dd803"
May 25 11:26:28.803182 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:26:28.803153    2560 scope.go:117] "RemoveContainer" containerID="386599237f9be94f9d703a8267d5269cb7c8b418c225c0519423fe967626b832"
May 25 11:26:28.810026 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:26:28.809997    2560 scope.go:117] "RemoveContainer" containerID="d88858e36cd7609adf271c5910b8d1dcb047f6696a3f44a86bb31269060dd803"
May 25 11:26:28.810283 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:26:28.810261    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d88858e36cd7609adf271c5910b8d1dcb047f6696a3f44a86bb31269060dd803\": container with ID starting with d88858e36cd7609adf271c5910b8d1dcb047f6696a3f44a86bb31269060dd803 not found: ID does not exist" containerID="d88858e36cd7609adf271c5910b8d1dcb047f6696a3f44a86bb31269060dd803"
May 25 11:26:28.810422 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:26:28.810289    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d88858e36cd7609adf271c5910b8d1dcb047f6696a3f44a86bb31269060dd803"} err="failed to get container status \"d88858e36cd7609adf271c5910b8d1dcb047f6696a3f44a86bb31269060dd803\": rpc error: code = NotFound desc = could not find container \"d88858e36cd7609adf271c5910b8d1dcb047f6696a3f44a86bb31269060dd803\": container with ID starting with d88858e36cd7609adf271c5910b8d1dcb047f6696a3f44a86bb31269060dd803 not found: ID does not exist"
May 25 11:26:28.810422 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:26:28.810307    2560 scope.go:117] "RemoveContainer" containerID="386599237f9be94f9d703a8267d5269cb7c8b418c225c0519423fe967626b832"
May 25 11:26:28.810549 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:26:28.810533    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"386599237f9be94f9d703a8267d5269cb7c8b418c225c0519423fe967626b832\": container with ID starting with 386599237f9be94f9d703a8267d5269cb7c8b418c225c0519423fe967626b832 not found: ID does not exist" containerID="386599237f9be94f9d703a8267d5269cb7c8b418c225c0519423fe967626b832"
May 25 11:26:28.810588 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:26:28.810554    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"386599237f9be94f9d703a8267d5269cb7c8b418c225c0519423fe967626b832"} err="failed to get container status \"386599237f9be94f9d703a8267d5269cb7c8b418c225c0519423fe967626b832\": rpc error: code = NotFound desc = could not find container \"386599237f9be94f9d703a8267d5269cb7c8b418c225c0519423fe967626b832\": container with ID starting with 386599237f9be94f9d703a8267d5269cb7c8b418c225c0519423fe967626b832 not found: ID does not exist"
May 25 11:26:28.839592 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:26:28.836984    2560 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-21751-predictor-6888887675-wtlk4"]
May 25 11:26:28.840539 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:26:28.840517    2560 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-21751-predictor-6888887675-wtlk4"]
May 25 11:26:28.935673 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:26:28.935593    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c63d88a9-9acf-41e8-a675-83742a1f2064" path="/var/lib/kubelet/pods/c63d88a9-9acf-41e8-a675-83742a1f2064/volumes"
May 25 11:26:32.796439 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:26:32.796414    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-55329-predictor-689f64f8cf-7x2km"
May 25 11:26:32.797021 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:26:32.796992    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-55329-predictor-689f64f8cf-7x2km" podUID="47d7ff78-3506-4b71-931a-1b9788def94b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
May 25 11:26:42.797254 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:26:42.797189    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-55329-predictor-689f64f8cf-7x2km" podUID="47d7ff78-3506-4b71-931a-1b9788def94b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
May 25 11:26:52.797326 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:26:52.797279    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-55329-predictor-689f64f8cf-7x2km" podUID="47d7ff78-3506-4b71-931a-1b9788def94b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
May 25 11:26:55.288366 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:26:55.288326    2560 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-f16f5-predictor-d9979f5f9-2b89m"]
May 25 11:26:55.288859 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:26:55.288840    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c63d88a9-9acf-41e8-a675-83742a1f2064" containerName="kserve-container"
May 25 11:26:55.288930 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:26:55.288861    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="c63d88a9-9acf-41e8-a675-83742a1f2064" containerName="kserve-container"
May 25 11:26:55.288930 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:26:55.288881    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c63d88a9-9acf-41e8-a675-83742a1f2064" containerName="kube-rbac-proxy"
May 25 11:26:55.288930 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:26:55.288890    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="c63d88a9-9acf-41e8-a675-83742a1f2064" containerName="kube-rbac-proxy"
May 25 11:26:55.289077 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:26:55.288975    2560 memory_manager.go:356] "RemoveStaleState removing state" podUID="c63d88a9-9acf-41e8-a675-83742a1f2064" containerName="kube-rbac-proxy"
May 25 11:26:55.289077 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:26:55.288993    2560 memory_manager.go:356] "RemoveStaleState removing state" podUID="c63d88a9-9acf-41e8-a675-83742a1f2064" containerName="kserve-container"
May 25 11:26:55.290982 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:26:55.290961    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-f16f5-predictor-d9979f5f9-2b89m"
May 25 11:26:55.293447 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:26:55.293421    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-f16f5-kube-rbac-proxy-sar-config\""
May 25 11:26:55.293539 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:26:55.293447    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-f16f5-predictor-serving-cert\""
May 25 11:26:55.302581 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:26:55.302559    2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-f16f5-predictor-d9979f5f9-2b89m"]
May 25 11:26:55.431226 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:26:55.431182    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"error-404-isvc-f16f5-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/646904be-ae32-46bb-8ccc-b3b1e359cf48-error-404-isvc-f16f5-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-f16f5-predictor-d9979f5f9-2b89m\" (UID: \"646904be-ae32-46bb-8ccc-b3b1e359cf48\") " pod="kserve-ci-e2e-test/error-404-isvc-f16f5-predictor-d9979f5f9-2b89m"
May 25 11:26:55.431391 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:26:55.431267    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/646904be-ae32-46bb-8ccc-b3b1e359cf48-proxy-tls\") pod \"error-404-isvc-f16f5-predictor-d9979f5f9-2b89m\" (UID: \"646904be-ae32-46bb-8ccc-b3b1e359cf48\") " pod="kserve-ci-e2e-test/error-404-isvc-f16f5-predictor-d9979f5f9-2b89m"
May 25 11:26:55.431391 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:26:55.431327    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vsk8s\" (UniqueName: \"kubernetes.io/projected/646904be-ae32-46bb-8ccc-b3b1e359cf48-kube-api-access-vsk8s\") pod \"error-404-isvc-f16f5-predictor-d9979f5f9-2b89m\" (UID: \"646904be-ae32-46bb-8ccc-b3b1e359cf48\") " pod="kserve-ci-e2e-test/error-404-isvc-f16f5-predictor-d9979f5f9-2b89m"
May 25 11:26:55.531833 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:26:55.531795    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-vsk8s\" (UniqueName: \"kubernetes.io/projected/646904be-ae32-46bb-8ccc-b3b1e359cf48-kube-api-access-vsk8s\") pod \"error-404-isvc-f16f5-predictor-d9979f5f9-2b89m\" (UID: \"646904be-ae32-46bb-8ccc-b3b1e359cf48\") " pod="kserve-ci-e2e-test/error-404-isvc-f16f5-predictor-d9979f5f9-2b89m"
May 25 11:26:55.532000 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:26:55.531858    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"error-404-isvc-f16f5-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/646904be-ae32-46bb-8ccc-b3b1e359cf48-error-404-isvc-f16f5-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-f16f5-predictor-d9979f5f9-2b89m\" (UID: \"646904be-ae32-46bb-8ccc-b3b1e359cf48\") " pod="kserve-ci-e2e-test/error-404-isvc-f16f5-predictor-d9979f5f9-2b89m"
May 25 11:26:55.532000 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:26:55.531898    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/646904be-ae32-46bb-8ccc-b3b1e359cf48-proxy-tls\") pod \"error-404-isvc-f16f5-predictor-d9979f5f9-2b89m\" (UID: \"646904be-ae32-46bb-8ccc-b3b1e359cf48\") " pod="kserve-ci-e2e-test/error-404-isvc-f16f5-predictor-d9979f5f9-2b89m"
May 25 11:26:55.532552 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:26:55.532527    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"error-404-isvc-f16f5-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/646904be-ae32-46bb-8ccc-b3b1e359cf48-error-404-isvc-f16f5-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-f16f5-predictor-d9979f5f9-2b89m\" (UID: \"646904be-ae32-46bb-8ccc-b3b1e359cf48\") " pod="kserve-ci-e2e-test/error-404-isvc-f16f5-predictor-d9979f5f9-2b89m"
May 25 11:26:55.534349 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:26:55.534333    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/646904be-ae32-46bb-8ccc-b3b1e359cf48-proxy-tls\") pod \"error-404-isvc-f16f5-predictor-d9979f5f9-2b89m\" (UID: \"646904be-ae32-46bb-8ccc-b3b1e359cf48\") " pod="kserve-ci-e2e-test/error-404-isvc-f16f5-predictor-d9979f5f9-2b89m"
May 25 11:26:55.541164 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:26:55.541113    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsk8s\" (UniqueName: \"kubernetes.io/projected/646904be-ae32-46bb-8ccc-b3b1e359cf48-kube-api-access-vsk8s\") pod \"error-404-isvc-f16f5-predictor-d9979f5f9-2b89m\" (UID: \"646904be-ae32-46bb-8ccc-b3b1e359cf48\") " pod="kserve-ci-e2e-test/error-404-isvc-f16f5-predictor-d9979f5f9-2b89m"
May 25 11:26:55.601151 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:26:55.601118    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-f16f5-predictor-d9979f5f9-2b89m"
May 25 11:26:55.732512 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:26:55.732488    2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-f16f5-predictor-d9979f5f9-2b89m"]
May 25 11:26:55.734659 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:26:55.734635    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod646904be_ae32_46bb_8ccc_b3b1e359cf48.slice/crio-31e8d224c626d00ed34ef0f73a226b354513336c468e1318b5998df930226b4c WatchSource:0}: Error finding container 31e8d224c626d00ed34ef0f73a226b354513336c468e1318b5998df930226b4c: Status 404 returned error can't find the container with id 31e8d224c626d00ed34ef0f73a226b354513336c468e1318b5998df930226b4c
May 25 11:26:55.886223 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:26:55.886185    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-f16f5-predictor-d9979f5f9-2b89m" event={"ID":"646904be-ae32-46bb-8ccc-b3b1e359cf48","Type":"ContainerStarted","Data":"7fbe27669a9fc577f088bede4effb007340de7ee0a088d4bf1d2923fd5aa7aff"}
May 25 11:26:55.886382 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:26:55.886243    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-f16f5-predictor-d9979f5f9-2b89m" event={"ID":"646904be-ae32-46bb-8ccc-b3b1e359cf48","Type":"ContainerStarted","Data":"f4b5631a275acf81755d466317ce62d0ed4dc1fd8a8eb95d4177da14a4e529e0"}
May 25 11:26:55.886382 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:26:55.886253    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-f16f5-predictor-d9979f5f9-2b89m" event={"ID":"646904be-ae32-46bb-8ccc-b3b1e359cf48","Type":"ContainerStarted","Data":"31e8d224c626d00ed34ef0f73a226b354513336c468e1318b5998df930226b4c"}
May 25 11:26:55.886482 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:26:55.886470    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-f16f5-predictor-d9979f5f9-2b89m"
May 25 11:26:55.886599 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:26:55.886582    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-f16f5-predictor-d9979f5f9-2b89m"
May 25 11:26:55.887825 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:26:55.887799    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-f16f5-predictor-d9979f5f9-2b89m" podUID="646904be-ae32-46bb-8ccc-b3b1e359cf48" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
May 25 11:26:55.905251 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:26:55.905195    2560 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-f16f5-predictor-d9979f5f9-2b89m" podStartSLOduration=0.905184637 podStartE2EDuration="905.184637ms" podCreationTimestamp="2026-05-25 11:26:55 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 11:26:55.903912252 +0000 UTC m=+557.584890225" watchObservedRunningTime="2026-05-25 11:26:55.905184637 +0000 UTC m=+557.586162611"
May 25 11:26:56.889714 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:26:56.889674    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-f16f5-predictor-d9979f5f9-2b89m" podUID="646904be-ae32-46bb-8ccc-b3b1e359cf48" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
May 25 11:27:01.893709 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:27:01.893680    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-f16f5-predictor-d9979f5f9-2b89m"
May 25 11:27:01.894120 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:27:01.894093    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-f16f5-predictor-d9979f5f9-2b89m" podUID="646904be-ae32-46bb-8ccc-b3b1e359cf48" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
May 25 11:27:02.797164 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:27:02.797130    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-55329-predictor-689f64f8cf-7x2km" podUID="47d7ff78-3506-4b71-931a-1b9788def94b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
May 25 11:27:11.894197 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:27:11.894157    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-f16f5-predictor-d9979f5f9-2b89m" podUID="646904be-ae32-46bb-8ccc-b3b1e359cf48" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
May 25 11:27:12.797964 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:27:12.797932    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-55329-predictor-689f64f8cf-7x2km"
May 25 11:27:21.894614 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:27:21.894577    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-f16f5-predictor-d9979f5f9-2b89m" podUID="646904be-ae32-46bb-8ccc-b3b1e359cf48" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
May 25 11:27:31.894907 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:27:31.894861    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-f16f5-predictor-d9979f5f9-2b89m" podUID="646904be-ae32-46bb-8ccc-b3b1e359cf48" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
May 25 11:27:38.854079 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:27:38.854053    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-zd54m_2be32661-9288-4902-9d67-1a5e8f74670d/console-operator/1.log"
May 25 11:27:38.854544 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:27:38.854232    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-zd54m_2be32661-9288-4902-9d67-1a5e8f74670d/console-operator/1.log"
May 25 11:27:38.860620 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:27:38.860601    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m55v2_b93f3e9c-f4f9-49dc-bfba-6d0f80c0971d/ovn-acl-logging/0.log"
May 25 11:27:38.860620 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:27:38.860617    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m55v2_b93f3e9c-f4f9-49dc-bfba-6d0f80c0971d/ovn-acl-logging/0.log"
May 25 11:27:41.895380 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:27:41.895350    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-f16f5-predictor-d9979f5f9-2b89m"
May 25 11:32:38.878260 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:32:38.878154    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-zd54m_2be32661-9288-4902-9d67-1a5e8f74670d/console-operator/1.log"
May 25 11:32:38.880536 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:32:38.880516    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-zd54m_2be32661-9288-4902-9d67-1a5e8f74670d/console-operator/1.log"
May 25 11:32:38.884610 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:32:38.884590    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m55v2_b93f3e9c-f4f9-49dc-bfba-6d0f80c0971d/ovn-acl-logging/0.log"
May 25 11:32:38.886714 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:32:38.886697    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m55v2_b93f3e9c-f4f9-49dc-bfba-6d0f80c0971d/ovn-acl-logging/0.log"
May 25 11:35:41.087312 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:35:41.087226    2560 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-49543-predictor-585f55b477-zzzff"]
May 25 11:35:41.090901 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:35:41.090883    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-49543-predictor-585f55b477-zzzff"
May 25 11:35:41.096161 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:35:41.096135    2560 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-55329-predictor-689f64f8cf-7x2km"]
May 25 11:35:41.096505 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:35:41.096447    2560 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-55329-predictor-689f64f8cf-7x2km" podUID="47d7ff78-3506-4b71-931a-1b9788def94b" containerName="kserve-container" containerID="cri-o://62db5f94662ecd3c08a5f905da0188d689618825d5957a79bc232612f479fc59" gracePeriod=30
May 25 11:35:41.096623 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:35:41.096501    2560 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-55329-predictor-689f64f8cf-7x2km" podUID="47d7ff78-3506-4b71-931a-1b9788def94b" containerName="kube-rbac-proxy" containerID="cri-o://6619a5a125d477b88b4460fe8c614d1ce709ea3c55170c9f0e780f6753c0fcd4" gracePeriod=30
May 25 11:35:41.106777 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:35:41.106752    2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-49543-predictor-585f55b477-zzzff"]
May 25 11:35:41.111629 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:35:41.111609    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-49543-kube-rbac-proxy-sar-config\""
May 25 11:35:41.111796 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:35:41.111745    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-49543-predictor-serving-cert\""
May 25 11:35:41.138335 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:35:41.138299    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"error-404-isvc-49543-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a8bf0f51-c69e-4b1d-bcd9-9f5a4bd9a28a-error-404-isvc-49543-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-49543-predictor-585f55b477-zzzff\" (UID: \"a8bf0f51-c69e-4b1d-bcd9-9f5a4bd9a28a\") " pod="kserve-ci-e2e-test/error-404-isvc-49543-predictor-585f55b477-zzzff"
May 25 11:35:41.138508 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:35:41.138357    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vmdj\" (UniqueName: \"kubernetes.io/projected/a8bf0f51-c69e-4b1d-bcd9-9f5a4bd9a28a-kube-api-access-5vmdj\") pod \"error-404-isvc-49543-predictor-585f55b477-zzzff\" (UID: \"a8bf0f51-c69e-4b1d-bcd9-9f5a4bd9a28a\") " pod="kserve-ci-e2e-test/error-404-isvc-49543-predictor-585f55b477-zzzff"
May 25 11:35:41.138508 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:35:41.138410    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a8bf0f51-c69e-4b1d-bcd9-9f5a4bd9a28a-proxy-tls\") pod \"error-404-isvc-49543-predictor-585f55b477-zzzff\" (UID: \"a8bf0f51-c69e-4b1d-bcd9-9f5a4bd9a28a\") " pod="kserve-ci-e2e-test/error-404-isvc-49543-predictor-585f55b477-zzzff"
May 25 11:35:41.239003 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:35:41.238971    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-5vmdj\" (UniqueName: \"kubernetes.io/projected/a8bf0f51-c69e-4b1d-bcd9-9f5a4bd9a28a-kube-api-access-5vmdj\") pod \"error-404-isvc-49543-predictor-585f55b477-zzzff\" (UID: \"a8bf0f51-c69e-4b1d-bcd9-9f5a4bd9a28a\") " pod="kserve-ci-e2e-test/error-404-isvc-49543-predictor-585f55b477-zzzff"
May 25 11:35:41.239003 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:35:41.239008    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a8bf0f51-c69e-4b1d-bcd9-9f5a4bd9a28a-proxy-tls\") pod \"error-404-isvc-49543-predictor-585f55b477-zzzff\" (UID: \"a8bf0f51-c69e-4b1d-bcd9-9f5a4bd9a28a\") " pod="kserve-ci-e2e-test/error-404-isvc-49543-predictor-585f55b477-zzzff"
May 25 11:35:41.239197 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:35:41.239058    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"error-404-isvc-49543-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a8bf0f51-c69e-4b1d-bcd9-9f5a4bd9a28a-error-404-isvc-49543-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-49543-predictor-585f55b477-zzzff\" (UID: \"a8bf0f51-c69e-4b1d-bcd9-9f5a4bd9a28a\") " pod="kserve-ci-e2e-test/error-404-isvc-49543-predictor-585f55b477-zzzff"
May 25 11:35:41.239668 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:35:41.239642    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"error-404-isvc-49543-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a8bf0f51-c69e-4b1d-bcd9-9f5a4bd9a28a-error-404-isvc-49543-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-49543-predictor-585f55b477-zzzff\" (UID: \"a8bf0f51-c69e-4b1d-bcd9-9f5a4bd9a28a\") " pod="kserve-ci-e2e-test/error-404-isvc-49543-predictor-585f55b477-zzzff"
May 25 11:35:41.241522 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:35:41.241500    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a8bf0f51-c69e-4b1d-bcd9-9f5a4bd9a28a-proxy-tls\") pod \"error-404-isvc-49543-predictor-585f55b477-zzzff\" (UID: \"a8bf0f51-c69e-4b1d-bcd9-9f5a4bd9a28a\") " pod="kserve-ci-e2e-test/error-404-isvc-49543-predictor-585f55b477-zzzff"
May 25 11:35:41.249931 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:35:41.249908    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vmdj\" (UniqueName: \"kubernetes.io/projected/a8bf0f51-c69e-4b1d-bcd9-9f5a4bd9a28a-kube-api-access-5vmdj\") pod \"error-404-isvc-49543-predictor-585f55b477-zzzff\" (UID: \"a8bf0f51-c69e-4b1d-bcd9-9f5a4bd9a28a\") " pod="kserve-ci-e2e-test/error-404-isvc-49543-predictor-585f55b477-zzzff"
May 25 11:35:41.403142 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:35:41.403048    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-49543-predictor-585f55b477-zzzff"
May 25 11:35:41.541577 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:35:41.541538    2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-49543-predictor-585f55b477-zzzff"]
May 25 11:35:41.548297 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:35:41.548268    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8bf0f51_c69e_4b1d_bcd9_9f5a4bd9a28a.slice/crio-c7b7a1476cc996cc666d068fee917838e354ac7583ff18521d881bcff7f1a4f4 WatchSource:0}: Error finding container c7b7a1476cc996cc666d068fee917838e354ac7583ff18521d881bcff7f1a4f4: Status 404 returned error can't find the container with id c7b7a1476cc996cc666d068fee917838e354ac7583ff18521d881bcff7f1a4f4
May 25 11:35:41.550052 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:35:41.550037    2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 25 11:35:41.623751 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:35:41.623725    2560 generic.go:358] "Generic (PLEG): container finished" podID="47d7ff78-3506-4b71-931a-1b9788def94b" containerID="6619a5a125d477b88b4460fe8c614d1ce709ea3c55170c9f0e780f6753c0fcd4" exitCode=2
May 25 11:35:41.623876 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:35:41.623793    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-55329-predictor-689f64f8cf-7x2km" event={"ID":"47d7ff78-3506-4b71-931a-1b9788def94b","Type":"ContainerDied","Data":"6619a5a125d477b88b4460fe8c614d1ce709ea3c55170c9f0e780f6753c0fcd4"}
May 25 11:35:41.624938 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:35:41.624917    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-49543-predictor-585f55b477-zzzff" event={"ID":"a8bf0f51-c69e-4b1d-bcd9-9f5a4bd9a28a","Type":"ContainerStarted","Data":"36b4111be561d4fdabce87b623e6565d7f959b0b7bff7913bd41dfe042e4f7ac"}
May 25 11:35:41.625019 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:35:41.624948    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-49543-predictor-585f55b477-zzzff" event={"ID":"a8bf0f51-c69e-4b1d-bcd9-9f5a4bd9a28a","Type":"ContainerStarted","Data":"c7b7a1476cc996cc666d068fee917838e354ac7583ff18521d881bcff7f1a4f4"}
May 25 11:35:42.629971 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:35:42.629933    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-49543-predictor-585f55b477-zzzff" event={"ID":"a8bf0f51-c69e-4b1d-bcd9-9f5a4bd9a28a","Type":"ContainerStarted","Data":"e2b00a61abe0d5fb156a7b668c5f5fd6dbd66594800a78387facc5c1fbb361c6"}
May 25 11:35:42.630376 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:35:42.630004    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-49543-predictor-585f55b477-zzzff"
May 25 11:35:42.671913 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:35:42.671862    2560 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-49543-predictor-585f55b477-zzzff" podStartSLOduration=1.671847798 podStartE2EDuration="1.671847798s" podCreationTimestamp="2026-05-25 11:35:41 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 11:35:42.6709405 +0000 UTC m=+1084.351918474" watchObservedRunningTime="2026-05-25 11:35:42.671847798 +0000 UTC m=+1084.352825771"
May 25 11:35:42.790594 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:35:42.790547    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-55329-predictor-689f64f8cf-7x2km" podUID="47d7ff78-3506-4b71-931a-1b9788def94b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.34:8643/healthz\": dial tcp 10.132.0.34:8643: connect: connection refused"
May 25 11:35:42.796912 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:35:42.796885    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-55329-predictor-689f64f8cf-7x2km" podUID="47d7ff78-3506-4b71-931a-1b9788def94b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
May 25 11:35:43.633879 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:35:43.633850    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-49543-predictor-585f55b477-zzzff"
May 25 11:35:43.635221 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:35:43.635177    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-49543-predictor-585f55b477-zzzff" podUID="a8bf0f51-c69e-4b1d-bcd9-9f5a4bd9a28a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
May 25 11:35:44.246179 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:35:44.246157    2560 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-55329-predictor-689f64f8cf-7x2km"
May 25 11:35:44.268705 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:35:44.268675    2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/47d7ff78-3506-4b71-931a-1b9788def94b-proxy-tls\") pod \"47d7ff78-3506-4b71-931a-1b9788def94b\" (UID: \"47d7ff78-3506-4b71-931a-1b9788def94b\") "
May 25 11:35:44.268841 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:35:44.268747    2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"error-404-isvc-55329-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/47d7ff78-3506-4b71-931a-1b9788def94b-error-404-isvc-55329-kube-rbac-proxy-sar-config\") pod \"47d7ff78-3506-4b71-931a-1b9788def94b\" (UID: \"47d7ff78-3506-4b71-931a-1b9788def94b\") "
May 25 11:35:44.268841 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:35:44.268772    2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n57zw\" (UniqueName: \"kubernetes.io/projected/47d7ff78-3506-4b71-931a-1b9788def94b-kube-api-access-n57zw\") pod \"47d7ff78-3506-4b71-931a-1b9788def94b\" (UID: \"47d7ff78-3506-4b71-931a-1b9788def94b\") "
May 25 11:35:44.269114 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:35:44.269083    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47d7ff78-3506-4b71-931a-1b9788def94b-error-404-isvc-55329-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "error-404-isvc-55329-kube-rbac-proxy-sar-config") pod "47d7ff78-3506-4b71-931a-1b9788def94b" (UID: "47d7ff78-3506-4b71-931a-1b9788def94b"). InnerVolumeSpecName "error-404-isvc-55329-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 11:35:44.270786 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:35:44.270760    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47d7ff78-3506-4b71-931a-1b9788def94b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "47d7ff78-3506-4b71-931a-1b9788def94b" (UID: "47d7ff78-3506-4b71-931a-1b9788def94b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 11:35:44.270786 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:35:44.270764    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47d7ff78-3506-4b71-931a-1b9788def94b-kube-api-access-n57zw" (OuterVolumeSpecName: "kube-api-access-n57zw") pod "47d7ff78-3506-4b71-931a-1b9788def94b" (UID: "47d7ff78-3506-4b71-931a-1b9788def94b"). InnerVolumeSpecName "kube-api-access-n57zw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 11:35:44.370406 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:35:44.370378    2560 reconciler_common.go:299] "Volume detached for volume \"error-404-isvc-55329-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/47d7ff78-3506-4b71-931a-1b9788def94b-error-404-isvc-55329-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-146.ec2.internal\" DevicePath \"\""
May 25 11:35:44.370406 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:35:44.370405    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n57zw\" (UniqueName: \"kubernetes.io/projected/47d7ff78-3506-4b71-931a-1b9788def94b-kube-api-access-n57zw\") on node \"ip-10-0-128-146.ec2.internal\" DevicePath \"\""
May 25 11:35:44.370595 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:35:44.370415    2560 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/47d7ff78-3506-4b71-931a-1b9788def94b-proxy-tls\") on node \"ip-10-0-128-146.ec2.internal\" DevicePath \"\""
May 25 11:35:44.638131 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:35:44.638040    2560 generic.go:358] "Generic (PLEG): container finished" podID="47d7ff78-3506-4b71-931a-1b9788def94b" containerID="62db5f94662ecd3c08a5f905da0188d689618825d5957a79bc232612f479fc59" exitCode=0
May 25 11:35:44.638131 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:35:44.638111    2560 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-55329-predictor-689f64f8cf-7x2km"
May 25 11:35:44.638591 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:35:44.638127    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-55329-predictor-689f64f8cf-7x2km" event={"ID":"47d7ff78-3506-4b71-931a-1b9788def94b","Type":"ContainerDied","Data":"62db5f94662ecd3c08a5f905da0188d689618825d5957a79bc232612f479fc59"}
May 25 11:35:44.638591 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:35:44.638167    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-55329-predictor-689f64f8cf-7x2km" event={"ID":"47d7ff78-3506-4b71-931a-1b9788def94b","Type":"ContainerDied","Data":"aab39d974191ed711a2074a9bb85eb2219d666099c871bd611f57617af0c667a"}
May 25 11:35:44.638591 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:35:44.638185    2560 scope.go:117] "RemoveContainer" containerID="6619a5a125d477b88b4460fe8c614d1ce709ea3c55170c9f0e780f6753c0fcd4"
May 25 11:35:44.638724 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:35:44.638660    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-49543-predictor-585f55b477-zzzff" podUID="a8bf0f51-c69e-4b1d-bcd9-9f5a4bd9a28a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
May 25 11:35:44.646862 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:35:44.646842    2560 scope.go:117] "RemoveContainer" containerID="62db5f94662ecd3c08a5f905da0188d689618825d5957a79bc232612f479fc59"
May 25 11:35:44.654167 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:35:44.654149    2560 scope.go:117] "RemoveContainer" containerID="6619a5a125d477b88b4460fe8c614d1ce709ea3c55170c9f0e780f6753c0fcd4"
May 25 11:35:44.654411 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:35:44.654390    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6619a5a125d477b88b4460fe8c614d1ce709ea3c55170c9f0e780f6753c0fcd4\": container with ID starting with 6619a5a125d477b88b4460fe8c614d1ce709ea3c55170c9f0e780f6753c0fcd4 not found: ID does not exist" containerID="6619a5a125d477b88b4460fe8c614d1ce709ea3c55170c9f0e780f6753c0fcd4"
May 25 11:35:44.654459 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:35:44.654418    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6619a5a125d477b88b4460fe8c614d1ce709ea3c55170c9f0e780f6753c0fcd4"} err="failed to get container status \"6619a5a125d477b88b4460fe8c614d1ce709ea3c55170c9f0e780f6753c0fcd4\": rpc error: code = NotFound desc = could not find container \"6619a5a125d477b88b4460fe8c614d1ce709ea3c55170c9f0e780f6753c0fcd4\": container with ID starting with 6619a5a125d477b88b4460fe8c614d1ce709ea3c55170c9f0e780f6753c0fcd4 not found: ID does not exist"
May 25 11:35:44.654459 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:35:44.654435    2560 scope.go:117] "RemoveContainer" containerID="62db5f94662ecd3c08a5f905da0188d689618825d5957a79bc232612f479fc59"
May 25 11:35:44.654658 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:35:44.654644    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62db5f94662ecd3c08a5f905da0188d689618825d5957a79bc232612f479fc59\": container with ID starting with 62db5f94662ecd3c08a5f905da0188d689618825d5957a79bc232612f479fc59 not found: ID does not exist" containerID="62db5f94662ecd3c08a5f905da0188d689618825d5957a79bc232612f479fc59"
May 25 11:35:44.654702 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:35:44.654660    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62db5f94662ecd3c08a5f905da0188d689618825d5957a79bc232612f479fc59"} err="failed to get container status \"62db5f94662ecd3c08a5f905da0188d689618825d5957a79bc232612f479fc59\": rpc error: code = NotFound desc = could not find container \"62db5f94662ecd3c08a5f905da0188d689618825d5957a79bc232612f479fc59\": container with ID starting with 62db5f94662ecd3c08a5f905da0188d689618825d5957a79bc232612f479fc59 not found: ID does not exist"
May 25 11:35:44.664004 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:35:44.663975    2560 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-55329-predictor-689f64f8cf-7x2km"]
May 25 11:35:44.673199 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:35:44.673179    2560 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-55329-predictor-689f64f8cf-7x2km"]
May 25 11:35:44.935328 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:35:44.935244    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="47d7ff78-3506-4b71-931a-1b9788def94b" path="/var/lib/kubelet/pods/47d7ff78-3506-4b71-931a-1b9788def94b/volumes"
May 25 11:35:49.642964 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:35:49.642939    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-49543-predictor-585f55b477-zzzff"
May 25 11:35:49.643507 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:35:49.643482    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-49543-predictor-585f55b477-zzzff" podUID="a8bf0f51-c69e-4b1d-bcd9-9f5a4bd9a28a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
May 25 11:35:59.643519 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:35:59.643482    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-49543-predictor-585f55b477-zzzff" podUID="a8bf0f51-c69e-4b1d-bcd9-9f5a4bd9a28a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
May 25 11:36:09.643864 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:36:09.643824    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-49543-predictor-585f55b477-zzzff" podUID="a8bf0f51-c69e-4b1d-bcd9-9f5a4bd9a28a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
May 25 11:36:10.191888 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:36:10.191855    2560 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-f16f5-predictor-d9979f5f9-2b89m"]
May 25 11:36:10.192148 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:36:10.192107    2560 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-f16f5-predictor-d9979f5f9-2b89m" podUID="646904be-ae32-46bb-8ccc-b3b1e359cf48" containerName="kserve-container" containerID="cri-o://f4b5631a275acf81755d466317ce62d0ed4dc1fd8a8eb95d4177da14a4e529e0" gracePeriod=30
May 25 11:36:10.192245 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:36:10.192153    2560 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-f16f5-predictor-d9979f5f9-2b89m" podUID="646904be-ae32-46bb-8ccc-b3b1e359cf48" containerName="kube-rbac-proxy" containerID="cri-o://7fbe27669a9fc577f088bede4effb007340de7ee0a088d4bf1d2923fd5aa7aff" gracePeriod=30
May 25 11:36:10.265371 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:36:10.265335    2560 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-412bf-predictor-bd476c8cc-gg7d4"]
May 25 11:36:10.266348 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:36:10.266324    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47d7ff78-3506-4b71-931a-1b9788def94b" containerName="kserve-container"
May 25 11:36:10.266493 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:36:10.266483    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="47d7ff78-3506-4b71-931a-1b9788def94b" containerName="kserve-container"
May 25 11:36:10.266585 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:36:10.266575    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47d7ff78-3506-4b71-931a-1b9788def94b" containerName="kube-rbac-proxy"
May 25 11:36:10.266658 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:36:10.266650    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="47d7ff78-3506-4b71-931a-1b9788def94b" containerName="kube-rbac-proxy"
May 25 11:36:10.266899 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:36:10.266886    2560 memory_manager.go:356] "RemoveStaleState removing state" podUID="47d7ff78-3506-4b71-931a-1b9788def94b" containerName="kserve-container"
May 25 11:36:10.267002 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:36:10.266993    2560 memory_manager.go:356] "RemoveStaleState removing state" podUID="47d7ff78-3506-4b71-931a-1b9788def94b" containerName="kube-rbac-proxy"
May 25 11:36:10.272633 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:36:10.272613    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-412bf-predictor-bd476c8cc-gg7d4"
May 25 11:36:10.275485 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:36:10.275463    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-412bf-predictor-serving-cert\""
May 25 11:36:10.275574 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:36:10.275491    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-412bf-kube-rbac-proxy-sar-config\""
May 25 11:36:10.286938 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:36:10.286917    2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-412bf-predictor-bd476c8cc-gg7d4"]
May 25 11:36:10.388493 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:36:10.388460    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/be304323-b880-4e46-b36b-a4b1c9711750-proxy-tls\") pod \"error-404-isvc-412bf-predictor-bd476c8cc-gg7d4\" (UID: \"be304323-b880-4e46-b36b-a4b1c9711750\") " pod="kserve-ci-e2e-test/error-404-isvc-412bf-predictor-bd476c8cc-gg7d4"
May 25 11:36:10.388658 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:36:10.388498    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"error-404-isvc-412bf-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/be304323-b880-4e46-b36b-a4b1c9711750-error-404-isvc-412bf-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-412bf-predictor-bd476c8cc-gg7d4\" (UID: \"be304323-b880-4e46-b36b-a4b1c9711750\") " pod="kserve-ci-e2e-test/error-404-isvc-412bf-predictor-bd476c8cc-gg7d4"
May 25 11:36:10.388658 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:36:10.388528    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8mwq\" (UniqueName: \"kubernetes.io/projected/be304323-b880-4e46-b36b-a4b1c9711750-kube-api-access-l8mwq\") pod \"error-404-isvc-412bf-predictor-bd476c8cc-gg7d4\" (UID: \"be304323-b880-4e46-b36b-a4b1c9711750\") " pod="kserve-ci-e2e-test/error-404-isvc-412bf-predictor-bd476c8cc-gg7d4"
May 25 11:36:10.489151 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:36:10.489070    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-l8mwq\" (UniqueName: \"kubernetes.io/projected/be304323-b880-4e46-b36b-a4b1c9711750-kube-api-access-l8mwq\") pod \"error-404-isvc-412bf-predictor-bd476c8cc-gg7d4\" (UID: \"be304323-b880-4e46-b36b-a4b1c9711750\") " pod="kserve-ci-e2e-test/error-404-isvc-412bf-predictor-bd476c8cc-gg7d4"
May 25 11:36:10.489335 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:36:10.489198    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/be304323-b880-4e46-b36b-a4b1c9711750-proxy-tls\") pod \"error-404-isvc-412bf-predictor-bd476c8cc-gg7d4\" (UID: \"be304323-b880-4e46-b36b-a4b1c9711750\") " pod="kserve-ci-e2e-test/error-404-isvc-412bf-predictor-bd476c8cc-gg7d4"
May 25 11:36:10.489335 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:36:10.489257    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"error-404-isvc-412bf-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/be304323-b880-4e46-b36b-a4b1c9711750-error-404-isvc-412bf-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-412bf-predictor-bd476c8cc-gg7d4\" (UID: \"be304323-b880-4e46-b36b-a4b1c9711750\") " pod="kserve-ci-e2e-test/error-404-isvc-412bf-predictor-bd476c8cc-gg7d4"
May 25 11:36:10.489335 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:36:10.489294    2560 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-412bf-predictor-serving-cert: secret "error-404-isvc-412bf-predictor-serving-cert" not found
May 25 11:36:10.489730 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:36:10.489372    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be304323-b880-4e46-b36b-a4b1c9711750-proxy-tls podName:be304323-b880-4e46-b36b-a4b1c9711750 nodeName:}" failed. No retries permitted until 2026-05-25 11:36:10.989351546 +0000 UTC m=+1112.670329505 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/be304323-b880-4e46-b36b-a4b1c9711750-proxy-tls") pod "error-404-isvc-412bf-predictor-bd476c8cc-gg7d4" (UID: "be304323-b880-4e46-b36b-a4b1c9711750") : secret "error-404-isvc-412bf-predictor-serving-cert" not found
May 25 11:36:10.489974 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:36:10.489952    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"error-404-isvc-412bf-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/be304323-b880-4e46-b36b-a4b1c9711750-error-404-isvc-412bf-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-412bf-predictor-bd476c8cc-gg7d4\" (UID: \"be304323-b880-4e46-b36b-a4b1c9711750\") " pod="kserve-ci-e2e-test/error-404-isvc-412bf-predictor-bd476c8cc-gg7d4"
May 25 11:36:10.516326 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:36:10.516302    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8mwq\" (UniqueName: \"kubernetes.io/projected/be304323-b880-4e46-b36b-a4b1c9711750-kube-api-access-l8mwq\") pod \"error-404-isvc-412bf-predictor-bd476c8cc-gg7d4\" (UID: \"be304323-b880-4e46-b36b-a4b1c9711750\") " pod="kserve-ci-e2e-test/error-404-isvc-412bf-predictor-bd476c8cc-gg7d4"
May 25 11:36:10.730250 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:36:10.730149    2560 generic.go:358] "Generic (PLEG): container finished" podID="646904be-ae32-46bb-8ccc-b3b1e359cf48" containerID="7fbe27669a9fc577f088bede4effb007340de7ee0a088d4bf1d2923fd5aa7aff" exitCode=2
May 25 11:36:10.730250 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:36:10.730238    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-f16f5-predictor-d9979f5f9-2b89m" event={"ID":"646904be-ae32-46bb-8ccc-b3b1e359cf48","Type":"ContainerDied","Data":"7fbe27669a9fc577f088bede4effb007340de7ee0a088d4bf1d2923fd5aa7aff"}
May 25 11:36:10.994498 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:36:10.994463    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/be304323-b880-4e46-b36b-a4b1c9711750-proxy-tls\") pod \"error-404-isvc-412bf-predictor-bd476c8cc-gg7d4\" (UID: \"be304323-b880-4e46-b36b-a4b1c9711750\") " pod="kserve-ci-e2e-test/error-404-isvc-412bf-predictor-bd476c8cc-gg7d4"
May 25 11:36:10.996854 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:36:10.996831    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/be304323-b880-4e46-b36b-a4b1c9711750-proxy-tls\") pod \"error-404-isvc-412bf-predictor-bd476c8cc-gg7d4\" (UID: \"be304323-b880-4e46-b36b-a4b1c9711750\") " pod="kserve-ci-e2e-test/error-404-isvc-412bf-predictor-bd476c8cc-gg7d4"
May 25 11:36:11.183112 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:36:11.183074    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-412bf-predictor-bd476c8cc-gg7d4"
May 25 11:36:11.327250 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:36:11.327227    2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-412bf-predictor-bd476c8cc-gg7d4"]
May 25 11:36:11.329032 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:36:11.329004    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe304323_b880_4e46_b36b_a4b1c9711750.slice/crio-4f55bb9260298be65b56c726627b0e281f55e6a9b4d16d31392fbe151be8c89e WatchSource:0}: Error finding container 4f55bb9260298be65b56c726627b0e281f55e6a9b4d16d31392fbe151be8c89e: Status 404 returned error can't find the container with id 4f55bb9260298be65b56c726627b0e281f55e6a9b4d16d31392fbe151be8c89e
May 25 11:36:11.734644 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:36:11.734612    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-412bf-predictor-bd476c8cc-gg7d4" event={"ID":"be304323-b880-4e46-b36b-a4b1c9711750","Type":"ContainerStarted","Data":"b1efd4ed73dc4e174d4693d27720c35b156800a4f5a3d3c3cc83d5e30fac70b8"}
May 25 11:36:11.734644 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:36:11.734649    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-412bf-predictor-bd476c8cc-gg7d4" event={"ID":"be304323-b880-4e46-b36b-a4b1c9711750","Type":"ContainerStarted","Data":"3d9a1c1ce7e41ea50ac59d6880b24827cd0756c52deb8b7849ce2be3eb75f687"}
May 25 11:36:11.735063 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:36:11.734658    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-412bf-predictor-bd476c8cc-gg7d4" event={"ID":"be304323-b880-4e46-b36b-a4b1c9711750","Type":"ContainerStarted","Data":"4f55bb9260298be65b56c726627b0e281f55e6a9b4d16d31392fbe151be8c89e"}
May 25 11:36:11.735063 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:36:11.734702    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-412bf-predictor-bd476c8cc-gg7d4"
May 25 11:36:11.760890 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:36:11.760842    2560 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-412bf-predictor-bd476c8cc-gg7d4" podStartSLOduration=1.760829229 podStartE2EDuration="1.760829229s" podCreationTimestamp="2026-05-25 11:36:10 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 11:36:11.760096227 +0000 UTC m=+1113.441074211" watchObservedRunningTime="2026-05-25 11:36:11.760829229 +0000 UTC m=+1113.441807201"
May 25 11:36:11.890483 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:36:11.890443    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-f16f5-predictor-d9979f5f9-2b89m" podUID="646904be-ae32-46bb-8ccc-b3b1e359cf48" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.35:8643/healthz\": dial tcp 10.132.0.35:8643: connect: connection refused"
May 25 11:36:11.895080 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:36:11.895053    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-f16f5-predictor-d9979f5f9-2b89m" podUID="646904be-ae32-46bb-8ccc-b3b1e359cf48" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
May 25 11:36:12.738515 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:36:12.738483    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-412bf-predictor-bd476c8cc-gg7d4"
May 25 11:36:12.739651 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:36:12.739627    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-412bf-predictor-bd476c8cc-gg7d4" podUID="be304323-b880-4e46-b36b-a4b1c9711750" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
May 25 11:36:13.440972 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:36:13.440951    2560 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-f16f5-predictor-d9979f5f9-2b89m"
May 25 11:36:13.516701 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:36:13.516621    2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"error-404-isvc-f16f5-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/646904be-ae32-46bb-8ccc-b3b1e359cf48-error-404-isvc-f16f5-kube-rbac-proxy-sar-config\") pod \"646904be-ae32-46bb-8ccc-b3b1e359cf48\" (UID: \"646904be-ae32-46bb-8ccc-b3b1e359cf48\") "
May 25 11:36:13.516849 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:36:13.516713    2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vsk8s\" (UniqueName: \"kubernetes.io/projected/646904be-ae32-46bb-8ccc-b3b1e359cf48-kube-api-access-vsk8s\") pod \"646904be-ae32-46bb-8ccc-b3b1e359cf48\" (UID: \"646904be-ae32-46bb-8ccc-b3b1e359cf48\") "
May 25 11:36:13.516849 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:36:13.516753    2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/646904be-ae32-46bb-8ccc-b3b1e359cf48-proxy-tls\") pod \"646904be-ae32-46bb-8ccc-b3b1e359cf48\" (UID: \"646904be-ae32-46bb-8ccc-b3b1e359cf48\") "
May 25 11:36:13.517064 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:36:13.517035    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/646904be-ae32-46bb-8ccc-b3b1e359cf48-error-404-isvc-f16f5-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "error-404-isvc-f16f5-kube-rbac-proxy-sar-config") pod "646904be-ae32-46bb-8ccc-b3b1e359cf48" (UID: "646904be-ae32-46bb-8ccc-b3b1e359cf48"). InnerVolumeSpecName "error-404-isvc-f16f5-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 11:36:13.518787 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:36:13.518767    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/646904be-ae32-46bb-8ccc-b3b1e359cf48-kube-api-access-vsk8s" (OuterVolumeSpecName: "kube-api-access-vsk8s") pod "646904be-ae32-46bb-8ccc-b3b1e359cf48" (UID: "646904be-ae32-46bb-8ccc-b3b1e359cf48"). InnerVolumeSpecName "kube-api-access-vsk8s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 11:36:13.518862 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:36:13.518852    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/646904be-ae32-46bb-8ccc-b3b1e359cf48-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "646904be-ae32-46bb-8ccc-b3b1e359cf48" (UID: "646904be-ae32-46bb-8ccc-b3b1e359cf48"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 11:36:13.617659 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:36:13.617618    2560 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/646904be-ae32-46bb-8ccc-b3b1e359cf48-proxy-tls\") on node \"ip-10-0-128-146.ec2.internal\" DevicePath \"\""
May 25 11:36:13.617659 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:36:13.617653    2560 reconciler_common.go:299] "Volume detached for volume \"error-404-isvc-f16f5-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/646904be-ae32-46bb-8ccc-b3b1e359cf48-error-404-isvc-f16f5-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-146.ec2.internal\" DevicePath \"\""
May 25 11:36:13.617659 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:36:13.617667    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vsk8s\" (UniqueName: \"kubernetes.io/projected/646904be-ae32-46bb-8ccc-b3b1e359cf48-kube-api-access-vsk8s\") on node \"ip-10-0-128-146.ec2.internal\" DevicePath \"\""
May 25 11:36:13.742319 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:36:13.742276    2560 generic.go:358] "Generic (PLEG): container finished" podID="646904be-ae32-46bb-8ccc-b3b1e359cf48" containerID="f4b5631a275acf81755d466317ce62d0ed4dc1fd8a8eb95d4177da14a4e529e0" exitCode=0
May 25 11:36:13.742781 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:36:13.742357    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-f16f5-predictor-d9979f5f9-2b89m" event={"ID":"646904be-ae32-46bb-8ccc-b3b1e359cf48","Type":"ContainerDied","Data":"f4b5631a275acf81755d466317ce62d0ed4dc1fd8a8eb95d4177da14a4e529e0"}
May 25 11:36:13.742781 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:36:13.742406    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-f16f5-predictor-d9979f5f9-2b89m" event={"ID":"646904be-ae32-46bb-8ccc-b3b1e359cf48","Type":"ContainerDied","Data":"31e8d224c626d00ed34ef0f73a226b354513336c468e1318b5998df930226b4c"}
May 25 11:36:13.742781 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:36:13.742421    2560 scope.go:117] "RemoveContainer" containerID="7fbe27669a9fc577f088bede4effb007340de7ee0a088d4bf1d2923fd5aa7aff"
May 25 11:36:13.742781 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:36:13.742366    2560 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-f16f5-predictor-d9979f5f9-2b89m"
May 25 11:36:13.743109 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:36:13.743080    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-412bf-predictor-bd476c8cc-gg7d4" podUID="be304323-b880-4e46-b36b-a4b1c9711750" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
May 25 11:36:13.751374 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:36:13.751351    2560 scope.go:117] "RemoveContainer" containerID="f4b5631a275acf81755d466317ce62d0ed4dc1fd8a8eb95d4177da14a4e529e0"
May 25 11:36:13.758410 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:36:13.758395    2560 scope.go:117] "RemoveContainer" containerID="7fbe27669a9fc577f088bede4effb007340de7ee0a088d4bf1d2923fd5aa7aff"
May 25 11:36:13.758663 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:36:13.758639    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7fbe27669a9fc577f088bede4effb007340de7ee0a088d4bf1d2923fd5aa7aff\": container with ID starting with 7fbe27669a9fc577f088bede4effb007340de7ee0a088d4bf1d2923fd5aa7aff not found: ID does not exist" containerID="7fbe27669a9fc577f088bede4effb007340de7ee0a088d4bf1d2923fd5aa7aff"
May 25 11:36:13.758745 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:36:13.758669    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7fbe27669a9fc577f088bede4effb007340de7ee0a088d4bf1d2923fd5aa7aff"} err="failed to get container status \"7fbe27669a9fc577f088bede4effb007340de7ee0a088d4bf1d2923fd5aa7aff\": rpc error: code = NotFound desc = could not find container \"7fbe27669a9fc577f088bede4effb007340de7ee0a088d4bf1d2923fd5aa7aff\": container with ID starting with 7fbe27669a9fc577f088bede4effb007340de7ee0a088d4bf1d2923fd5aa7aff not found: ID does not exist"
May 25 11:36:13.758745 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:36:13.758687    2560 scope.go:117] "RemoveContainer" containerID="f4b5631a275acf81755d466317ce62d0ed4dc1fd8a8eb95d4177da14a4e529e0"
May 25 11:36:13.758919 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:36:13.758905    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4b5631a275acf81755d466317ce62d0ed4dc1fd8a8eb95d4177da14a4e529e0\": container with ID starting with f4b5631a275acf81755d466317ce62d0ed4dc1fd8a8eb95d4177da14a4e529e0 not found: ID does not exist" containerID="f4b5631a275acf81755d466317ce62d0ed4dc1fd8a8eb95d4177da14a4e529e0"
May 25 11:36:13.758957 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:36:13.758923    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4b5631a275acf81755d466317ce62d0ed4dc1fd8a8eb95d4177da14a4e529e0"} err="failed to get container status \"f4b5631a275acf81755d466317ce62d0ed4dc1fd8a8eb95d4177da14a4e529e0\": rpc error: code = NotFound desc = could not find container \"f4b5631a275acf81755d466317ce62d0ed4dc1fd8a8eb95d4177da14a4e529e0\": container with ID starting with f4b5631a275acf81755d466317ce62d0ed4dc1fd8a8eb95d4177da14a4e529e0 not found: ID does not exist"
May 25 11:36:13.777538 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:36:13.777463    2560 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-f16f5-predictor-d9979f5f9-2b89m"]
May 25 11:36:13.779062 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:36:13.779040    2560 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-f16f5-predictor-d9979f5f9-2b89m"]
May 25 11:36:14.934907 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:36:14.934875    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="646904be-ae32-46bb-8ccc-b3b1e359cf48" path="/var/lib/kubelet/pods/646904be-ae32-46bb-8ccc-b3b1e359cf48/volumes"
May 25 11:36:18.747377 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:36:18.747347    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-412bf-predictor-bd476c8cc-gg7d4"
May 25 11:36:18.747894 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:36:18.747862    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-412bf-predictor-bd476c8cc-gg7d4" podUID="be304323-b880-4e46-b36b-a4b1c9711750" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
May 25 11:36:19.643455 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:36:19.643408    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-49543-predictor-585f55b477-zzzff" podUID="a8bf0f51-c69e-4b1d-bcd9-9f5a4bd9a28a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
May 25 11:36:28.748577 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:36:28.748534    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-412bf-predictor-bd476c8cc-gg7d4" podUID="be304323-b880-4e46-b36b-a4b1c9711750" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
May 25 11:36:29.644835 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:36:29.644805    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-49543-predictor-585f55b477-zzzff"
May 25 11:36:38.748226 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:36:38.748168    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-412bf-predictor-bd476c8cc-gg7d4" podUID="be304323-b880-4e46-b36b-a4b1c9711750" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
May 25 11:36:48.748669 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:36:48.748571    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-412bf-predictor-bd476c8cc-gg7d4" podUID="be304323-b880-4e46-b36b-a4b1c9711750" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
May 25 11:36:50.900836 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:36:50.900801    2560 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-49543-predictor-585f55b477-zzzff"]
May 25 11:36:50.901297 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:36:50.901243    2560 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-49543-predictor-585f55b477-zzzff" podUID="a8bf0f51-c69e-4b1d-bcd9-9f5a4bd9a28a" containerName="kserve-container" containerID="cri-o://36b4111be561d4fdabce87b623e6565d7f959b0b7bff7913bd41dfe042e4f7ac" gracePeriod=30
May 25 11:36:50.901476 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:36:50.901419    2560 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-49543-predictor-585f55b477-zzzff" podUID="a8bf0f51-c69e-4b1d-bcd9-9f5a4bd9a28a" containerName="kube-rbac-proxy" containerID="cri-o://e2b00a61abe0d5fb156a7b668c5f5fd6dbd66594800a78387facc5c1fbb361c6" gracePeriod=30
May 25 11:36:50.959679 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:36:50.959652    2560 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-20622-predictor-57456b66c5-zxs4z"]
May 25 11:36:50.960014 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:36:50.959978    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="646904be-ae32-46bb-8ccc-b3b1e359cf48" containerName="kube-rbac-proxy"
May 25 11:36:50.960014 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:36:50.959999    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="646904be-ae32-46bb-8ccc-b3b1e359cf48" containerName="kube-rbac-proxy"
May 25 11:36:50.960095 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:36:50.960018    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="646904be-ae32-46bb-8ccc-b3b1e359cf48" containerName="kserve-container"
May 25 11:36:50.960095 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:36:50.960023    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="646904be-ae32-46bb-8ccc-b3b1e359cf48" containerName="kserve-container"
May 25 11:36:50.960155 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:36:50.960102    2560 memory_manager.go:356] "RemoveStaleState removing state" podUID="646904be-ae32-46bb-8ccc-b3b1e359cf48" containerName="kserve-container"
May 25 11:36:50.960155 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:36:50.960111    2560 memory_manager.go:356] "RemoveStaleState removing state" podUID="646904be-ae32-46bb-8ccc-b3b1e359cf48" containerName="kube-rbac-proxy"
May 25 11:36:50.964259 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:36:50.964239    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-20622-predictor-57456b66c5-zxs4z"
May 25 11:36:50.966875 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:36:50.966857    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-20622-predictor-serving-cert\""
May 25 11:36:50.966966 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:36:50.966859    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-20622-kube-rbac-proxy-sar-config\""
May 25 11:36:50.974772 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:36:50.974746    2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-20622-predictor-57456b66c5-zxs4z"]
May 25 11:36:51.036287 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:36:51.036255    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6d4b495f-f2db-40db-a7bb-e9b722a5101c-proxy-tls\") pod \"error-404-isvc-20622-predictor-57456b66c5-zxs4z\" (UID: \"6d4b495f-f2db-40db-a7bb-e9b722a5101c\") " pod="kserve-ci-e2e-test/error-404-isvc-20622-predictor-57456b66c5-zxs4z"
May 25 11:36:51.036441 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:36:51.036381    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"error-404-isvc-20622-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6d4b495f-f2db-40db-a7bb-e9b722a5101c-error-404-isvc-20622-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-20622-predictor-57456b66c5-zxs4z\" (UID: \"6d4b495f-f2db-40db-a7bb-e9b722a5101c\") " pod="kserve-ci-e2e-test/error-404-isvc-20622-predictor-57456b66c5-zxs4z"
May 25 11:36:51.036568 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:36:51.036542    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9d6j\" (UniqueName: \"kubernetes.io/projected/6d4b495f-f2db-40db-a7bb-e9b722a5101c-kube-api-access-n9d6j\") pod \"error-404-isvc-20622-predictor-57456b66c5-zxs4z\" (UID: \"6d4b495f-f2db-40db-a7bb-e9b722a5101c\") " pod="kserve-ci-e2e-test/error-404-isvc-20622-predictor-57456b66c5-zxs4z"
May 25 11:36:51.137372 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:36:51.137339    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-n9d6j\" (UniqueName: \"kubernetes.io/projected/6d4b495f-f2db-40db-a7bb-e9b722a5101c-kube-api-access-n9d6j\") pod \"error-404-isvc-20622-predictor-57456b66c5-zxs4z\" (UID: \"6d4b495f-f2db-40db-a7bb-e9b722a5101c\") " pod="kserve-ci-e2e-test/error-404-isvc-20622-predictor-57456b66c5-zxs4z"
May 25 11:36:51.137545 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:36:51.137504    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6d4b495f-f2db-40db-a7bb-e9b722a5101c-proxy-tls\") pod \"error-404-isvc-20622-predictor-57456b66c5-zxs4z\" (UID: \"6d4b495f-f2db-40db-a7bb-e9b722a5101c\") " pod="kserve-ci-e2e-test/error-404-isvc-20622-predictor-57456b66c5-zxs4z"
May 25 11:36:51.137644 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:36:51.137618    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"error-404-isvc-20622-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6d4b495f-f2db-40db-a7bb-e9b722a5101c-error-404-isvc-20622-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-20622-predictor-57456b66c5-zxs4z\" (UID: \"6d4b495f-f2db-40db-a7bb-e9b722a5101c\") " pod="kserve-ci-e2e-test/error-404-isvc-20622-predictor-57456b66c5-zxs4z"
May 25 11:36:51.138451 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:36:51.138428    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"error-404-isvc-20622-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6d4b495f-f2db-40db-a7bb-e9b722a5101c-error-404-isvc-20622-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-20622-predictor-57456b66c5-zxs4z\" (UID: \"6d4b495f-f2db-40db-a7bb-e9b722a5101c\") " pod="kserve-ci-e2e-test/error-404-isvc-20622-predictor-57456b66c5-zxs4z"
May 25 11:36:51.139927 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:36:51.139905    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6d4b495f-f2db-40db-a7bb-e9b722a5101c-proxy-tls\") pod \"error-404-isvc-20622-predictor-57456b66c5-zxs4z\" (UID: \"6d4b495f-f2db-40db-a7bb-e9b722a5101c\") " pod="kserve-ci-e2e-test/error-404-isvc-20622-predictor-57456b66c5-zxs4z"
May 25 11:36:51.146085 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:36:51.146008    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9d6j\" (UniqueName: \"kubernetes.io/projected/6d4b495f-f2db-40db-a7bb-e9b722a5101c-kube-api-access-n9d6j\") pod \"error-404-isvc-20622-predictor-57456b66c5-zxs4z\" (UID: \"6d4b495f-f2db-40db-a7bb-e9b722a5101c\") " pod="kserve-ci-e2e-test/error-404-isvc-20622-predictor-57456b66c5-zxs4z"
May 25 11:36:51.274578 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:36:51.274505    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-20622-predictor-57456b66c5-zxs4z"
May 25 11:36:51.411180 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:36:51.411154    2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-20622-predictor-57456b66c5-zxs4z"]
May 25 11:36:51.413194 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:36:51.413167    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d4b495f_f2db_40db_a7bb_e9b722a5101c.slice/crio-593e20ecb8f854811c53f6d5882184dda1799ec6a7aa6fc6c5c74356006a32ec WatchSource:0}: Error finding container 593e20ecb8f854811c53f6d5882184dda1799ec6a7aa6fc6c5c74356006a32ec: Status 404 returned error can't find the container with id 593e20ecb8f854811c53f6d5882184dda1799ec6a7aa6fc6c5c74356006a32ec
May 25 11:36:51.878693 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:36:51.878660    2560 generic.go:358] "Generic (PLEG): container finished" podID="a8bf0f51-c69e-4b1d-bcd9-9f5a4bd9a28a" containerID="e2b00a61abe0d5fb156a7b668c5f5fd6dbd66594800a78387facc5c1fbb361c6" exitCode=2
May 25 11:36:51.878857 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:36:51.878730    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-49543-predictor-585f55b477-zzzff" event={"ID":"a8bf0f51-c69e-4b1d-bcd9-9f5a4bd9a28a","Type":"ContainerDied","Data":"e2b00a61abe0d5fb156a7b668c5f5fd6dbd66594800a78387facc5c1fbb361c6"}
May 25 11:36:51.880252 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:36:51.880230    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-20622-predictor-57456b66c5-zxs4z" event={"ID":"6d4b495f-f2db-40db-a7bb-e9b722a5101c","Type":"ContainerStarted","Data":"fca6b179d5c50b3592480103eb861d9caa57ffdea2bc3650752aa2f11d0fcc61"}
May 25 11:36:51.880393 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:36:51.880260    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-20622-predictor-57456b66c5-zxs4z" event={"ID":"6d4b495f-f2db-40db-a7bb-e9b722a5101c","Type":"ContainerStarted","Data":"67dbf3bf33daf769a189851cce0d4f148941781c0a18a067804a9745c3b397f6"}
May 25 11:36:51.880393 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:36:51.880275    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-20622-predictor-57456b66c5-zxs4z" event={"ID":"6d4b495f-f2db-40db-a7bb-e9b722a5101c","Type":"ContainerStarted","Data":"593e20ecb8f854811c53f6d5882184dda1799ec6a7aa6fc6c5c74356006a32ec"}
May 25 11:36:51.880393 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:36:51.880370    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-20622-predictor-57456b66c5-zxs4z"
May 25 11:36:51.902893 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:36:51.902846    2560 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-20622-predictor-57456b66c5-zxs4z" podStartSLOduration=1.902834637 podStartE2EDuration="1.902834637s" podCreationTimestamp="2026-05-25 11:36:50 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 11:36:51.901490163 +0000 UTC m=+1153.582468135" watchObservedRunningTime="2026-05-25 11:36:51.902834637 +0000 UTC m=+1153.583812612"
May 25 11:36:52.883672 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:36:52.883641    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-20622-predictor-57456b66c5-zxs4z"
May 25 11:36:52.884979 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:36:52.884943    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-20622-predictor-57456b66c5-zxs4z" podUID="6d4b495f-f2db-40db-a7bb-e9b722a5101c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
May 25 11:36:53.889520 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:36:53.889486    2560 generic.go:358] "Generic (PLEG): container finished" podID="a8bf0f51-c69e-4b1d-bcd9-9f5a4bd9a28a" containerID="36b4111be561d4fdabce87b623e6565d7f959b0b7bff7913bd41dfe042e4f7ac" exitCode=0
May 25 11:36:53.889938 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:36:53.889564    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-49543-predictor-585f55b477-zzzff" event={"ID":"a8bf0f51-c69e-4b1d-bcd9-9f5a4bd9a28a","Type":"ContainerDied","Data":"36b4111be561d4fdabce87b623e6565d7f959b0b7bff7913bd41dfe042e4f7ac"}
May 25 11:36:53.890007 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:36:53.889931    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-20622-predictor-57456b66c5-zxs4z" podUID="6d4b495f-f2db-40db-a7bb-e9b722a5101c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
May 25 11:36:53.952912 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:36:53.952893    2560 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-49543-predictor-585f55b477-zzzff"
May 25 11:36:54.062690 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:36:54.062613    2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5vmdj\" (UniqueName: \"kubernetes.io/projected/a8bf0f51-c69e-4b1d-bcd9-9f5a4bd9a28a-kube-api-access-5vmdj\") pod \"a8bf0f51-c69e-4b1d-bcd9-9f5a4bd9a28a\" (UID: \"a8bf0f51-c69e-4b1d-bcd9-9f5a4bd9a28a\") "
May 25 11:36:54.062690 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:36:54.062672    2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"error-404-isvc-49543-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a8bf0f51-c69e-4b1d-bcd9-9f5a4bd9a28a-error-404-isvc-49543-kube-rbac-proxy-sar-config\") pod \"a8bf0f51-c69e-4b1d-bcd9-9f5a4bd9a28a\" (UID: \"a8bf0f51-c69e-4b1d-bcd9-9f5a4bd9a28a\") "
May 25 11:36:54.062892 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:36:54.062704    2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a8bf0f51-c69e-4b1d-bcd9-9f5a4bd9a28a-proxy-tls\") pod \"a8bf0f51-c69e-4b1d-bcd9-9f5a4bd9a28a\" (UID: \"a8bf0f51-c69e-4b1d-bcd9-9f5a4bd9a28a\") "
May 25 11:36:54.063070 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:36:54.063043    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8bf0f51-c69e-4b1d-bcd9-9f5a4bd9a28a-error-404-isvc-49543-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "error-404-isvc-49543-kube-rbac-proxy-sar-config") pod "a8bf0f51-c69e-4b1d-bcd9-9f5a4bd9a28a" (UID: "a8bf0f51-c69e-4b1d-bcd9-9f5a4bd9a28a"). InnerVolumeSpecName "error-404-isvc-49543-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 11:36:54.064830 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:36:54.064802    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8bf0f51-c69e-4b1d-bcd9-9f5a4bd9a28a-kube-api-access-5vmdj" (OuterVolumeSpecName: "kube-api-access-5vmdj") pod "a8bf0f51-c69e-4b1d-bcd9-9f5a4bd9a28a" (UID: "a8bf0f51-c69e-4b1d-bcd9-9f5a4bd9a28a"). InnerVolumeSpecName "kube-api-access-5vmdj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 11:36:54.064912 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:36:54.064828    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8bf0f51-c69e-4b1d-bcd9-9f5a4bd9a28a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a8bf0f51-c69e-4b1d-bcd9-9f5a4bd9a28a" (UID: "a8bf0f51-c69e-4b1d-bcd9-9f5a4bd9a28a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 11:36:54.163308 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:36:54.163273    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5vmdj\" (UniqueName: \"kubernetes.io/projected/a8bf0f51-c69e-4b1d-bcd9-9f5a4bd9a28a-kube-api-access-5vmdj\") on node \"ip-10-0-128-146.ec2.internal\" DevicePath \"\""
May 25 11:36:54.163308 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:36:54.163303    2560 reconciler_common.go:299] "Volume detached for volume \"error-404-isvc-49543-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a8bf0f51-c69e-4b1d-bcd9-9f5a4bd9a28a-error-404-isvc-49543-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-146.ec2.internal\" DevicePath \"\""
May 25 11:36:54.163308 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:36:54.163315    2560 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a8bf0f51-c69e-4b1d-bcd9-9f5a4bd9a28a-proxy-tls\") on node \"ip-10-0-128-146.ec2.internal\" DevicePath \"\""
May 25 11:36:54.894555 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:36:54.894521    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-49543-predictor-585f55b477-zzzff" event={"ID":"a8bf0f51-c69e-4b1d-bcd9-9f5a4bd9a28a","Type":"ContainerDied","Data":"c7b7a1476cc996cc666d068fee917838e354ac7583ff18521d881bcff7f1a4f4"}
May 25 11:36:54.894555 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:36:54.894560    2560 scope.go:117] "RemoveContainer" containerID="e2b00a61abe0d5fb156a7b668c5f5fd6dbd66594800a78387facc5c1fbb361c6"
May 25 11:36:54.895117 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:36:54.894570    2560 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-49543-predictor-585f55b477-zzzff"
May 25 11:36:54.903329 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:36:54.903312    2560 scope.go:117] "RemoveContainer" containerID="36b4111be561d4fdabce87b623e6565d7f959b0b7bff7913bd41dfe042e4f7ac"
May 25 11:36:54.918626 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:36:54.918606    2560 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-49543-predictor-585f55b477-zzzff"]
May 25 11:36:54.928244 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:36:54.928223    2560 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-49543-predictor-585f55b477-zzzff"]
May 25 11:36:54.934059 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:36:54.934037    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a8bf0f51-c69e-4b1d-bcd9-9f5a4bd9a28a" path="/var/lib/kubelet/pods/a8bf0f51-c69e-4b1d-bcd9-9f5a4bd9a28a/volumes"
May 25 11:36:58.748874 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:36:58.748841    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-412bf-predictor-bd476c8cc-gg7d4"
May 25 11:36:58.894546 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:36:58.894517    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-20622-predictor-57456b66c5-zxs4z"
May 25 11:36:58.894955 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:36:58.894925    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-20622-predictor-57456b66c5-zxs4z" podUID="6d4b495f-f2db-40db-a7bb-e9b722a5101c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
May 25 11:37:08.895055 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:37:08.895007    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-20622-predictor-57456b66c5-zxs4z" podUID="6d4b495f-f2db-40db-a7bb-e9b722a5101c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
May 25 11:37:18.895070 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:37:18.895028    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-20622-predictor-57456b66c5-zxs4z" podUID="6d4b495f-f2db-40db-a7bb-e9b722a5101c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
May 25 11:37:20.411561 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:37:20.411528    2560 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-412bf-predictor-bd476c8cc-gg7d4"]
May 25 11:37:20.411924 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:37:20.411787    2560 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-412bf-predictor-bd476c8cc-gg7d4" podUID="be304323-b880-4e46-b36b-a4b1c9711750" containerName="kserve-container" containerID="cri-o://3d9a1c1ce7e41ea50ac59d6880b24827cd0756c52deb8b7849ce2be3eb75f687" gracePeriod=30
May 25 11:37:20.411924 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:37:20.411857    2560 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-412bf-predictor-bd476c8cc-gg7d4" podUID="be304323-b880-4e46-b36b-a4b1c9711750" containerName="kube-rbac-proxy" containerID="cri-o://b1efd4ed73dc4e174d4693d27720c35b156800a4f5a3d3c3cc83d5e30fac70b8" gracePeriod=30
May 25 11:37:20.457339 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:37:20.457309    2560 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-b59e4-predictor-69f5bfb789-d9zqj"]
May 25 11:37:20.457682 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:37:20.457670    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8bf0f51-c69e-4b1d-bcd9-9f5a4bd9a28a" containerName="kube-rbac-proxy"
May 25 11:37:20.457723 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:37:20.457683    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8bf0f51-c69e-4b1d-bcd9-9f5a4bd9a28a" containerName="kube-rbac-proxy"
May 25 11:37:20.457723 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:37:20.457695    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8bf0f51-c69e-4b1d-bcd9-9f5a4bd9a28a" containerName="kserve-container"
May 25 11:37:20.457723 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:37:20.457700    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8bf0f51-c69e-4b1d-bcd9-9f5a4bd9a28a" containerName="kserve-container"
May 25 11:37:20.457817 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:37:20.457761    2560 memory_manager.go:356] "RemoveStaleState removing state" podUID="a8bf0f51-c69e-4b1d-bcd9-9f5a4bd9a28a" containerName="kube-rbac-proxy"
May 25 11:37:20.457817 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:37:20.457771    2560 memory_manager.go:356] "RemoveStaleState removing state" podUID="a8bf0f51-c69e-4b1d-bcd9-9f5a4bd9a28a" containerName="kserve-container"
May 25 11:37:20.462476 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:37:20.462459    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-b59e4-predictor-69f5bfb789-d9zqj"
May 25 11:37:20.465057 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:37:20.465038    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-b59e4-predictor-serving-cert\""
May 25 11:37:20.465140 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:37:20.465081    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-b59e4-kube-rbac-proxy-sar-config\""
May 25 11:37:20.470495 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:37:20.470476    2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-b59e4-predictor-69f5bfb789-d9zqj"]
May 25 11:37:20.584576 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:37:20.584543    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wsng\" (UniqueName: \"kubernetes.io/projected/4919c573-90ea-47be-8e87-88291649d15b-kube-api-access-6wsng\") pod \"error-404-isvc-b59e4-predictor-69f5bfb789-d9zqj\" (UID: \"4919c573-90ea-47be-8e87-88291649d15b\") " pod="kserve-ci-e2e-test/error-404-isvc-b59e4-predictor-69f5bfb789-d9zqj"
May 25 11:37:20.584737 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:37:20.584591    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4919c573-90ea-47be-8e87-88291649d15b-proxy-tls\") pod \"error-404-isvc-b59e4-predictor-69f5bfb789-d9zqj\" (UID: \"4919c573-90ea-47be-8e87-88291649d15b\") " pod="kserve-ci-e2e-test/error-404-isvc-b59e4-predictor-69f5bfb789-d9zqj"
May 25 11:37:20.584737 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:37:20.584671    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"error-404-isvc-b59e4-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4919c573-90ea-47be-8e87-88291649d15b-error-404-isvc-b59e4-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-b59e4-predictor-69f5bfb789-d9zqj\" (UID: \"4919c573-90ea-47be-8e87-88291649d15b\") " pod="kserve-ci-e2e-test/error-404-isvc-b59e4-predictor-69f5bfb789-d9zqj"
May 25 11:37:20.685511 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:37:20.685430    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-6wsng\" (UniqueName: \"kubernetes.io/projected/4919c573-90ea-47be-8e87-88291649d15b-kube-api-access-6wsng\") pod \"error-404-isvc-b59e4-predictor-69f5bfb789-d9zqj\" (UID: \"4919c573-90ea-47be-8e87-88291649d15b\") " pod="kserve-ci-e2e-test/error-404-isvc-b59e4-predictor-69f5bfb789-d9zqj"
May 25 11:37:20.685511 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:37:20.685483    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4919c573-90ea-47be-8e87-88291649d15b-proxy-tls\") pod \"error-404-isvc-b59e4-predictor-69f5bfb789-d9zqj\" (UID: \"4919c573-90ea-47be-8e87-88291649d15b\") " pod="kserve-ci-e2e-test/error-404-isvc-b59e4-predictor-69f5bfb789-d9zqj"
May 25 11:37:20.685732 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:37:20.685558    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"error-404-isvc-b59e4-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4919c573-90ea-47be-8e87-88291649d15b-error-404-isvc-b59e4-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-b59e4-predictor-69f5bfb789-d9zqj\" (UID: \"4919c573-90ea-47be-8e87-88291649d15b\") " pod="kserve-ci-e2e-test/error-404-isvc-b59e4-predictor-69f5bfb789-d9zqj"
May 25 11:37:20.685732 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:37:20.685624    2560 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-b59e4-predictor-serving-cert: secret "error-404-isvc-b59e4-predictor-serving-cert" not found
May 25 11:37:20.685732 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:37:20.685697    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4919c573-90ea-47be-8e87-88291649d15b-proxy-tls podName:4919c573-90ea-47be-8e87-88291649d15b nodeName:}" failed. No retries permitted until 2026-05-25 11:37:21.185677728 +0000 UTC m=+1182.866655698 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/4919c573-90ea-47be-8e87-88291649d15b-proxy-tls") pod "error-404-isvc-b59e4-predictor-69f5bfb789-d9zqj" (UID: "4919c573-90ea-47be-8e87-88291649d15b") : secret "error-404-isvc-b59e4-predictor-serving-cert" not found
May 25 11:37:20.686239 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:37:20.686191    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"error-404-isvc-b59e4-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4919c573-90ea-47be-8e87-88291649d15b-error-404-isvc-b59e4-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-b59e4-predictor-69f5bfb789-d9zqj\" (UID: \"4919c573-90ea-47be-8e87-88291649d15b\") " pod="kserve-ci-e2e-test/error-404-isvc-b59e4-predictor-69f5bfb789-d9zqj"
May 25 11:37:20.694558 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:37:20.694538    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wsng\" (UniqueName: \"kubernetes.io/projected/4919c573-90ea-47be-8e87-88291649d15b-kube-api-access-6wsng\") pod \"error-404-isvc-b59e4-predictor-69f5bfb789-d9zqj\" (UID: \"4919c573-90ea-47be-8e87-88291649d15b\") " pod="kserve-ci-e2e-test/error-404-isvc-b59e4-predictor-69f5bfb789-d9zqj"
May 25 11:37:20.980812 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:37:20.980727    2560 generic.go:358] "Generic (PLEG): container finished" podID="be304323-b880-4e46-b36b-a4b1c9711750" containerID="b1efd4ed73dc4e174d4693d27720c35b156800a4f5a3d3c3cc83d5e30fac70b8" exitCode=2
May 25 11:37:20.980945 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:37:20.980799    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-412bf-predictor-bd476c8cc-gg7d4" event={"ID":"be304323-b880-4e46-b36b-a4b1c9711750","Type":"ContainerDied","Data":"b1efd4ed73dc4e174d4693d27720c35b156800a4f5a3d3c3cc83d5e30fac70b8"}
May 25 11:37:21.188881 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:37:21.188851    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4919c573-90ea-47be-8e87-88291649d15b-proxy-tls\") pod \"error-404-isvc-b59e4-predictor-69f5bfb789-d9zqj\" (UID: \"4919c573-90ea-47be-8e87-88291649d15b\") " pod="kserve-ci-e2e-test/error-404-isvc-b59e4-predictor-69f5bfb789-d9zqj"
May 25 11:37:21.191218 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:37:21.191187    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4919c573-90ea-47be-8e87-88291649d15b-proxy-tls\") pod \"error-404-isvc-b59e4-predictor-69f5bfb789-d9zqj\" (UID: \"4919c573-90ea-47be-8e87-88291649d15b\") " pod="kserve-ci-e2e-test/error-404-isvc-b59e4-predictor-69f5bfb789-d9zqj"
May 25 11:37:21.374096 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:37:21.374059    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-b59e4-predictor-69f5bfb789-d9zqj"
May 25 11:37:21.506110 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:37:21.506078    2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-b59e4-predictor-69f5bfb789-d9zqj"]
May 25 11:37:21.509170 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:37:21.509131    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4919c573_90ea_47be_8e87_88291649d15b.slice/crio-91c9058325524560745a69f1cdf707d7aea55c1e0ef0a67bacf9aae0e55c9f00 WatchSource:0}: Error finding container 91c9058325524560745a69f1cdf707d7aea55c1e0ef0a67bacf9aae0e55c9f00: Status 404 returned error can't find the container with id 91c9058325524560745a69f1cdf707d7aea55c1e0ef0a67bacf9aae0e55c9f00
May 25 11:37:21.985475 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:37:21.985383    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-b59e4-predictor-69f5bfb789-d9zqj" event={"ID":"4919c573-90ea-47be-8e87-88291649d15b","Type":"ContainerStarted","Data":"756d18c7dbb4830235ab482f28cfc3b7e48e366813568ec9862b51acbebdb613"}
May 25 11:37:21.985475 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:37:21.985419    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-b59e4-predictor-69f5bfb789-d9zqj" event={"ID":"4919c573-90ea-47be-8e87-88291649d15b","Type":"ContainerStarted","Data":"9da952fff62cf4b8b122cf0d9aec7cc9e5490c21bc65503036b1dadb0d73f075"}
May 25 11:37:21.985475 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:37:21.985435    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-b59e4-predictor-69f5bfb789-d9zqj"
May 25 11:37:21.985475 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:37:21.985450    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-b59e4-predictor-69f5bfb789-d9zqj"
May 25 11:37:21.985475 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:37:21.985460    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-b59e4-predictor-69f5bfb789-d9zqj" event={"ID":"4919c573-90ea-47be-8e87-88291649d15b","Type":"ContainerStarted","Data":"91c9058325524560745a69f1cdf707d7aea55c1e0ef0a67bacf9aae0e55c9f00"}
May 25 11:37:21.986512 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:37:21.986487    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-b59e4-predictor-69f5bfb789-d9zqj" podUID="4919c573-90ea-47be-8e87-88291649d15b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
May 25 11:37:22.006019 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:37:22.005975    2560 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-b59e4-predictor-69f5bfb789-d9zqj" podStartSLOduration=2.005961622 podStartE2EDuration="2.005961622s" podCreationTimestamp="2026-05-25 11:37:20 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 11:37:22.004505578 +0000 UTC m=+1183.685483562" watchObservedRunningTime="2026-05-25 11:37:22.005961622 +0000 UTC m=+1183.686939595"
May 25 11:37:22.989079 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:37:22.989043    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-b59e4-predictor-69f5bfb789-d9zqj" podUID="4919c573-90ea-47be-8e87-88291649d15b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
May 25 11:37:23.743850 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:37:23.743806    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-412bf-predictor-bd476c8cc-gg7d4" podUID="be304323-b880-4e46-b36b-a4b1c9711750" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.37:8643/healthz\": dial tcp 10.132.0.37:8643: connect: connection refused"
May 25 11:37:23.996240 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:37:23.995921    2560 generic.go:358] "Generic (PLEG): container finished" podID="be304323-b880-4e46-b36b-a4b1c9711750" containerID="3d9a1c1ce7e41ea50ac59d6880b24827cd0756c52deb8b7849ce2be3eb75f687" exitCode=0
May 25 11:37:23.996240 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:37:23.996045    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-412bf-predictor-bd476c8cc-gg7d4" event={"ID":"be304323-b880-4e46-b36b-a4b1c9711750","Type":"ContainerDied","Data":"3d9a1c1ce7e41ea50ac59d6880b24827cd0756c52deb8b7849ce2be3eb75f687"}
May 25 11:37:24.060751 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:37:24.060729    2560 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-412bf-predictor-bd476c8cc-gg7d4"
May 25 11:37:24.216469 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:37:24.216437    2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/be304323-b880-4e46-b36b-a4b1c9711750-proxy-tls\") pod \"be304323-b880-4e46-b36b-a4b1c9711750\" (UID: \"be304323-b880-4e46-b36b-a4b1c9711750\") "
May 25 11:37:24.216692 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:37:24.216499    2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l8mwq\" (UniqueName: \"kubernetes.io/projected/be304323-b880-4e46-b36b-a4b1c9711750-kube-api-access-l8mwq\") pod \"be304323-b880-4e46-b36b-a4b1c9711750\" (UID: \"be304323-b880-4e46-b36b-a4b1c9711750\") "
May 25 11:37:24.216692 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:37:24.216533    2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"error-404-isvc-412bf-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/be304323-b880-4e46-b36b-a4b1c9711750-error-404-isvc-412bf-kube-rbac-proxy-sar-config\") pod \"be304323-b880-4e46-b36b-a4b1c9711750\" (UID: \"be304323-b880-4e46-b36b-a4b1c9711750\") "
May 25 11:37:24.216867 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:37:24.216845    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be304323-b880-4e46-b36b-a4b1c9711750-error-404-isvc-412bf-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "error-404-isvc-412bf-kube-rbac-proxy-sar-config") pod "be304323-b880-4e46-b36b-a4b1c9711750" (UID: "be304323-b880-4e46-b36b-a4b1c9711750"). InnerVolumeSpecName "error-404-isvc-412bf-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 11:37:24.218515 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:37:24.218476    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be304323-b880-4e46-b36b-a4b1c9711750-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "be304323-b880-4e46-b36b-a4b1c9711750" (UID: "be304323-b880-4e46-b36b-a4b1c9711750"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 11:37:24.218618 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:37:24.218552    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be304323-b880-4e46-b36b-a4b1c9711750-kube-api-access-l8mwq" (OuterVolumeSpecName: "kube-api-access-l8mwq") pod "be304323-b880-4e46-b36b-a4b1c9711750" (UID: "be304323-b880-4e46-b36b-a4b1c9711750"). InnerVolumeSpecName "kube-api-access-l8mwq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 11:37:24.317684 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:37:24.317598    2560 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/be304323-b880-4e46-b36b-a4b1c9711750-proxy-tls\") on node \"ip-10-0-128-146.ec2.internal\" DevicePath \"\""
May 25 11:37:24.317684 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:37:24.317635    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l8mwq\" (UniqueName: \"kubernetes.io/projected/be304323-b880-4e46-b36b-a4b1c9711750-kube-api-access-l8mwq\") on node \"ip-10-0-128-146.ec2.internal\" DevicePath \"\""
May 25 11:37:24.317684 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:37:24.317647    2560 reconciler_common.go:299] "Volume detached for volume \"error-404-isvc-412bf-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/be304323-b880-4e46-b36b-a4b1c9711750-error-404-isvc-412bf-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-146.ec2.internal\" DevicePath \"\""
May 25 11:37:25.000966 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:37:25.000935    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-412bf-predictor-bd476c8cc-gg7d4" event={"ID":"be304323-b880-4e46-b36b-a4b1c9711750","Type":"ContainerDied","Data":"4f55bb9260298be65b56c726627b0e281f55e6a9b4d16d31392fbe151be8c89e"}
May 25 11:37:25.001370 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:37:25.000978    2560 scope.go:117] "RemoveContainer" containerID="b1efd4ed73dc4e174d4693d27720c35b156800a4f5a3d3c3cc83d5e30fac70b8"
May 25 11:37:25.001370 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:37:25.001000    2560 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-412bf-predictor-bd476c8cc-gg7d4"
May 25 11:37:25.008862 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:37:25.008844    2560 scope.go:117] "RemoveContainer" containerID="3d9a1c1ce7e41ea50ac59d6880b24827cd0756c52deb8b7849ce2be3eb75f687"
May 25 11:37:25.023942 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:37:25.023917    2560 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-412bf-predictor-bd476c8cc-gg7d4"]
May 25 11:37:25.034701 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:37:25.034679    2560 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-412bf-predictor-bd476c8cc-gg7d4"]
May 25 11:37:26.935629 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:37:26.935596    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="be304323-b880-4e46-b36b-a4b1c9711750" path="/var/lib/kubelet/pods/be304323-b880-4e46-b36b-a4b1c9711750/volumes"
May 25 11:37:27.993464 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:37:27.993438    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-b59e4-predictor-69f5bfb789-d9zqj"
May 25 11:37:27.993990 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:37:27.993965    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-b59e4-predictor-69f5bfb789-d9zqj" podUID="4919c573-90ea-47be-8e87-88291649d15b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
May 25 11:37:28.894936 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:37:28.894901    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-20622-predictor-57456b66c5-zxs4z" podUID="6d4b495f-f2db-40db-a7bb-e9b722a5101c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
May 25 11:37:37.994529 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:37:37.994487    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-b59e4-predictor-69f5bfb789-d9zqj" podUID="4919c573-90ea-47be-8e87-88291649d15b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
May 25 11:37:38.895650 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:37:38.895621    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-20622-predictor-57456b66c5-zxs4z"
May 25 11:37:38.902336 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:37:38.902310    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-zd54m_2be32661-9288-4902-9d67-1a5e8f74670d/console-operator/1.log"
May 25 11:37:38.905255 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:37:38.905233    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-zd54m_2be32661-9288-4902-9d67-1a5e8f74670d/console-operator/1.log"
May 25 11:37:38.908681 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:37:38.908662    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m55v2_b93f3e9c-f4f9-49dc-bfba-6d0f80c0971d/ovn-acl-logging/0.log"
May 25 11:37:38.911923 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:37:38.911904    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m55v2_b93f3e9c-f4f9-49dc-bfba-6d0f80c0971d/ovn-acl-logging/0.log"
May 25 11:37:47.994736 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:37:47.994700    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-b59e4-predictor-69f5bfb789-d9zqj" podUID="4919c573-90ea-47be-8e87-88291649d15b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
May 25 11:37:57.994649 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:37:57.994610    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-b59e4-predictor-69f5bfb789-d9zqj" podUID="4919c573-90ea-47be-8e87-88291649d15b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
May 25 11:38:07.995340 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:38:07.995311    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-b59e4-predictor-69f5bfb789-d9zqj"
May 25 11:42:38.926300 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:42:38.926269    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-zd54m_2be32661-9288-4902-9d67-1a5e8f74670d/console-operator/1.log"
May 25 11:42:38.931280 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:42:38.931258    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-zd54m_2be32661-9288-4902-9d67-1a5e8f74670d/console-operator/1.log"
May 25 11:42:38.932755 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:42:38.932735    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m55v2_b93f3e9c-f4f9-49dc-bfba-6d0f80c0971d/ovn-acl-logging/0.log"
May 25 11:42:38.937787 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:42:38.937765    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m55v2_b93f3e9c-f4f9-49dc-bfba-6d0f80c0971d/ovn-acl-logging/0.log"
May 25 11:46:05.822275 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:46:05.822243    2560 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-20622-predictor-57456b66c5-zxs4z"]
May 25 11:46:05.824710 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:46:05.822509    2560 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-20622-predictor-57456b66c5-zxs4z" podUID="6d4b495f-f2db-40db-a7bb-e9b722a5101c" containerName="kserve-container" containerID="cri-o://67dbf3bf33daf769a189851cce0d4f148941781c0a18a067804a9745c3b397f6" gracePeriod=30
May 25 11:46:05.824710 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:46:05.822592    2560 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-20622-predictor-57456b66c5-zxs4z" podUID="6d4b495f-f2db-40db-a7bb-e9b722a5101c" containerName="kube-rbac-proxy" containerID="cri-o://fca6b179d5c50b3592480103eb861d9caa57ffdea2bc3650752aa2f11d0fcc61" gracePeriod=30
May 25 11:46:05.917828 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:46:05.917794    2560 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-c3e68-predictor-b87f65c65-xxt7d"]
May 25 11:46:05.918151 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:46:05.918139    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be304323-b880-4e46-b36b-a4b1c9711750" containerName="kube-rbac-proxy"
May 25 11:46:05.918199 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:46:05.918152    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="be304323-b880-4e46-b36b-a4b1c9711750" containerName="kube-rbac-proxy"
May 25 11:46:05.918199 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:46:05.918167    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be304323-b880-4e46-b36b-a4b1c9711750" containerName="kserve-container"
May 25 11:46:05.918199 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:46:05.918173    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="be304323-b880-4e46-b36b-a4b1c9711750" containerName="kserve-container"
May 25 11:46:05.918367 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:46:05.918244    2560 memory_manager.go:356] "RemoveStaleState removing state" podUID="be304323-b880-4e46-b36b-a4b1c9711750" containerName="kserve-container"
May 25 11:46:05.918367 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:46:05.918253    2560 memory_manager.go:356] "RemoveStaleState removing state" podUID="be304323-b880-4e46-b36b-a4b1c9711750" containerName="kube-rbac-proxy"
May 25 11:46:05.921250 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:46:05.921230    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-c3e68-predictor-b87f65c65-xxt7d"
May 25 11:46:05.923714 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:46:05.923696    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-c3e68-kube-rbac-proxy-sar-config\""
May 25 11:46:05.923853 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:46:05.923835    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-c3e68-predictor-serving-cert\""
May 25 11:46:05.930606 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:46:05.930584    2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-c3e68-predictor-b87f65c65-xxt7d"]
May 25 11:46:05.940580 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:46:05.940557    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"error-404-isvc-c3e68-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f1814527-8ba1-4118-ad92-4454338b4116-error-404-isvc-c3e68-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-c3e68-predictor-b87f65c65-xxt7d\" (UID: \"f1814527-8ba1-4118-ad92-4454338b4116\") " pod="kserve-ci-e2e-test/error-404-isvc-c3e68-predictor-b87f65c65-xxt7d"
May 25 11:46:05.940746 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:46:05.940724    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f1814527-8ba1-4118-ad92-4454338b4116-proxy-tls\") pod \"error-404-isvc-c3e68-predictor-b87f65c65-xxt7d\" (UID: \"f1814527-8ba1-4118-ad92-4454338b4116\") " pod="kserve-ci-e2e-test/error-404-isvc-c3e68-predictor-b87f65c65-xxt7d"
May 25 11:46:05.940800 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:46:05.940761    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48nwf\" (UniqueName: \"kubernetes.io/projected/f1814527-8ba1-4118-ad92-4454338b4116-kube-api-access-48nwf\") pod \"error-404-isvc-c3e68-predictor-b87f65c65-xxt7d\" (UID: \"f1814527-8ba1-4118-ad92-4454338b4116\") " pod="kserve-ci-e2e-test/error-404-isvc-c3e68-predictor-b87f65c65-xxt7d"
May 25 11:46:06.041275 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:46:06.041232    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f1814527-8ba1-4118-ad92-4454338b4116-proxy-tls\") pod \"error-404-isvc-c3e68-predictor-b87f65c65-xxt7d\" (UID: \"f1814527-8ba1-4118-ad92-4454338b4116\") " pod="kserve-ci-e2e-test/error-404-isvc-c3e68-predictor-b87f65c65-xxt7d"
May 25 11:46:06.041477 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:46:06.041289    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-48nwf\" (UniqueName: \"kubernetes.io/projected/f1814527-8ba1-4118-ad92-4454338b4116-kube-api-access-48nwf\") pod \"error-404-isvc-c3e68-predictor-b87f65c65-xxt7d\" (UID: \"f1814527-8ba1-4118-ad92-4454338b4116\") " pod="kserve-ci-e2e-test/error-404-isvc-c3e68-predictor-b87f65c65-xxt7d"
May 25 11:46:06.041477 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:46:06.041352    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"error-404-isvc-c3e68-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f1814527-8ba1-4118-ad92-4454338b4116-error-404-isvc-c3e68-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-c3e68-predictor-b87f65c65-xxt7d\" (UID: \"f1814527-8ba1-4118-ad92-4454338b4116\") " pod="kserve-ci-e2e-test/error-404-isvc-c3e68-predictor-b87f65c65-xxt7d"
May 25 11:46:06.041477 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:46:06.041358    2560 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-c3e68-predictor-serving-cert: secret "error-404-isvc-c3e68-predictor-serving-cert" not found
May 25 11:46:06.041477 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:46:06.041475    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1814527-8ba1-4118-ad92-4454338b4116-proxy-tls podName:f1814527-8ba1-4118-ad92-4454338b4116 nodeName:}" failed. No retries permitted until 2026-05-25 11:46:06.541453962 +0000 UTC m=+1708.222431931 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/f1814527-8ba1-4118-ad92-4454338b4116-proxy-tls") pod "error-404-isvc-c3e68-predictor-b87f65c65-xxt7d" (UID: "f1814527-8ba1-4118-ad92-4454338b4116") : secret "error-404-isvc-c3e68-predictor-serving-cert" not found
May 25 11:46:06.041985 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:46:06.041964    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"error-404-isvc-c3e68-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f1814527-8ba1-4118-ad92-4454338b4116-error-404-isvc-c3e68-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-c3e68-predictor-b87f65c65-xxt7d\" (UID: \"f1814527-8ba1-4118-ad92-4454338b4116\") " pod="kserve-ci-e2e-test/error-404-isvc-c3e68-predictor-b87f65c65-xxt7d"
May 25 11:46:06.053253 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:46:06.053203    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-48nwf\" (UniqueName: \"kubernetes.io/projected/f1814527-8ba1-4118-ad92-4454338b4116-kube-api-access-48nwf\") pod \"error-404-isvc-c3e68-predictor-b87f65c65-xxt7d\" (UID: \"f1814527-8ba1-4118-ad92-4454338b4116\") " pod="kserve-ci-e2e-test/error-404-isvc-c3e68-predictor-b87f65c65-xxt7d"
May 25 11:46:06.546462 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:46:06.546417    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f1814527-8ba1-4118-ad92-4454338b4116-proxy-tls\") pod \"error-404-isvc-c3e68-predictor-b87f65c65-xxt7d\" (UID: \"f1814527-8ba1-4118-ad92-4454338b4116\") " pod="kserve-ci-e2e-test/error-404-isvc-c3e68-predictor-b87f65c65-xxt7d"
May 25 11:46:06.548842 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:46:06.548810    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f1814527-8ba1-4118-ad92-4454338b4116-proxy-tls\") pod \"error-404-isvc-c3e68-predictor-b87f65c65-xxt7d\" (UID: \"f1814527-8ba1-4118-ad92-4454338b4116\") " pod="kserve-ci-e2e-test/error-404-isvc-c3e68-predictor-b87f65c65-xxt7d"
May 25 11:46:06.741419 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:46:06.741387    2560 generic.go:358] "Generic (PLEG): container finished" podID="6d4b495f-f2db-40db-a7bb-e9b722a5101c" containerID="fca6b179d5c50b3592480103eb861d9caa57ffdea2bc3650752aa2f11d0fcc61" exitCode=2
May 25 11:46:06.741589 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:46:06.741456    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-20622-predictor-57456b66c5-zxs4z" event={"ID":"6d4b495f-f2db-40db-a7bb-e9b722a5101c","Type":"ContainerDied","Data":"fca6b179d5c50b3592480103eb861d9caa57ffdea2bc3650752aa2f11d0fcc61"}
May 25 11:46:06.833560 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:46:06.833477    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-c3e68-predictor-b87f65c65-xxt7d"
May 25 11:46:06.968194 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:46:06.968169    2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-c3e68-predictor-b87f65c65-xxt7d"]
May 25 11:46:06.970346 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:46:06.970314    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1814527_8ba1_4118_ad92_4454338b4116.slice/crio-078fd49e6dbb02b6695a765b6af1ce37ff4c78dccaa0117543405f8770a4f030 WatchSource:0}: Error finding container 078fd49e6dbb02b6695a765b6af1ce37ff4c78dccaa0117543405f8770a4f030: Status 404 returned error can't find the container with id 078fd49e6dbb02b6695a765b6af1ce37ff4c78dccaa0117543405f8770a4f030
May 25 11:46:06.972236 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:46:06.972200    2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 25 11:46:07.746045 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:46:07.746010    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-c3e68-predictor-b87f65c65-xxt7d" event={"ID":"f1814527-8ba1-4118-ad92-4454338b4116","Type":"ContainerStarted","Data":"c988f0cf1724b58b618758d36b59e753a2259acd53d5dcf70f689e2afd3d3cd9"}
May 25 11:46:07.746045 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:46:07.746052    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-c3e68-predictor-b87f65c65-xxt7d" event={"ID":"f1814527-8ba1-4118-ad92-4454338b4116","Type":"ContainerStarted","Data":"ec05206985f85669abe7f22638944a301917aadb292d7f410563e43424e0feb5"}
May 25 11:46:07.746285 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:46:07.746063    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-c3e68-predictor-b87f65c65-xxt7d" event={"ID":"f1814527-8ba1-4118-ad92-4454338b4116","Type":"ContainerStarted","Data":"078fd49e6dbb02b6695a765b6af1ce37ff4c78dccaa0117543405f8770a4f030"}
May 25 11:46:07.746285 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:46:07.746224    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-c3e68-predictor-b87f65c65-xxt7d"
May 25 11:46:07.766849 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:46:07.766806    2560 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-c3e68-predictor-b87f65c65-xxt7d" podStartSLOduration=2.766793518 podStartE2EDuration="2.766793518s" podCreationTimestamp="2026-05-25 11:46:05 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 11:46:07.766071661 +0000 UTC m=+1709.447049635" watchObservedRunningTime="2026-05-25 11:46:07.766793518 +0000 UTC m=+1709.447771491"
May 25 11:46:08.749909 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:46:08.749883    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-c3e68-predictor-b87f65c65-xxt7d"
May 25 11:46:08.751441 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:46:08.751413    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c3e68-predictor-b87f65c65-xxt7d" podUID="f1814527-8ba1-4118-ad92-4454338b4116" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
May 25 11:46:08.890774 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:46:08.890733    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-20622-predictor-57456b66c5-zxs4z" podUID="6d4b495f-f2db-40db-a7bb-e9b722a5101c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.38:8643/healthz\": dial tcp 10.132.0.38:8643: connect: connection refused"
May 25 11:46:08.895072 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:46:08.895049    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-20622-predictor-57456b66c5-zxs4z" podUID="6d4b495f-f2db-40db-a7bb-e9b722a5101c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
May 25 11:46:09.370521 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:46:09.370496    2560 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-20622-predictor-57456b66c5-zxs4z"
May 25 11:46:09.465057 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:46:09.464971    2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"error-404-isvc-20622-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6d4b495f-f2db-40db-a7bb-e9b722a5101c-error-404-isvc-20622-kube-rbac-proxy-sar-config\") pod \"6d4b495f-f2db-40db-a7bb-e9b722a5101c\" (UID: \"6d4b495f-f2db-40db-a7bb-e9b722a5101c\") "
May 25 11:46:09.465057 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:46:09.465055    2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n9d6j\" (UniqueName: \"kubernetes.io/projected/6d4b495f-f2db-40db-a7bb-e9b722a5101c-kube-api-access-n9d6j\") pod \"6d4b495f-f2db-40db-a7bb-e9b722a5101c\" (UID: \"6d4b495f-f2db-40db-a7bb-e9b722a5101c\") "
May 25 11:46:09.465293 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:46:09.465078    2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6d4b495f-f2db-40db-a7bb-e9b722a5101c-proxy-tls\") pod \"6d4b495f-f2db-40db-a7bb-e9b722a5101c\" (UID: \"6d4b495f-f2db-40db-a7bb-e9b722a5101c\") "
May 25 11:46:09.465361 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:46:09.465340    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d4b495f-f2db-40db-a7bb-e9b722a5101c-error-404-isvc-20622-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "error-404-isvc-20622-kube-rbac-proxy-sar-config") pod "6d4b495f-f2db-40db-a7bb-e9b722a5101c" (UID: "6d4b495f-f2db-40db-a7bb-e9b722a5101c"). InnerVolumeSpecName "error-404-isvc-20622-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 11:46:09.467184 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:46:09.467149    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d4b495f-f2db-40db-a7bb-e9b722a5101c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6d4b495f-f2db-40db-a7bb-e9b722a5101c" (UID: "6d4b495f-f2db-40db-a7bb-e9b722a5101c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 11:46:09.467326 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:46:09.467287    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d4b495f-f2db-40db-a7bb-e9b722a5101c-kube-api-access-n9d6j" (OuterVolumeSpecName: "kube-api-access-n9d6j") pod "6d4b495f-f2db-40db-a7bb-e9b722a5101c" (UID: "6d4b495f-f2db-40db-a7bb-e9b722a5101c"). InnerVolumeSpecName "kube-api-access-n9d6j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 11:46:09.565662 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:46:09.565621    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n9d6j\" (UniqueName: \"kubernetes.io/projected/6d4b495f-f2db-40db-a7bb-e9b722a5101c-kube-api-access-n9d6j\") on node \"ip-10-0-128-146.ec2.internal\" DevicePath \"\""
May 25 11:46:09.565662 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:46:09.565649    2560 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6d4b495f-f2db-40db-a7bb-e9b722a5101c-proxy-tls\") on node \"ip-10-0-128-146.ec2.internal\" DevicePath \"\""
May 25 11:46:09.565662 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:46:09.565660    2560 reconciler_common.go:299] "Volume detached for volume \"error-404-isvc-20622-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6d4b495f-f2db-40db-a7bb-e9b722a5101c-error-404-isvc-20622-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-146.ec2.internal\" DevicePath \"\""
May 25 11:46:09.755115 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:46:09.755035    2560 generic.go:358] "Generic (PLEG): container finished" podID="6d4b495f-f2db-40db-a7bb-e9b722a5101c" containerID="67dbf3bf33daf769a189851cce0d4f148941781c0a18a067804a9745c3b397f6" exitCode=0
May 25 11:46:09.755115 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:46:09.755105    2560 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-20622-predictor-57456b66c5-zxs4z"
May 25 11:46:09.755566 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:46:09.755120    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-20622-predictor-57456b66c5-zxs4z" event={"ID":"6d4b495f-f2db-40db-a7bb-e9b722a5101c","Type":"ContainerDied","Data":"67dbf3bf33daf769a189851cce0d4f148941781c0a18a067804a9745c3b397f6"}
May 25 11:46:09.755566 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:46:09.755155    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-20622-predictor-57456b66c5-zxs4z" event={"ID":"6d4b495f-f2db-40db-a7bb-e9b722a5101c","Type":"ContainerDied","Data":"593e20ecb8f854811c53f6d5882184dda1799ec6a7aa6fc6c5c74356006a32ec"}
May 25 11:46:09.755566 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:46:09.755175    2560 scope.go:117] "RemoveContainer" containerID="fca6b179d5c50b3592480103eb861d9caa57ffdea2bc3650752aa2f11d0fcc61"
May 25 11:46:09.755683 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:46:09.755586    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c3e68-predictor-b87f65c65-xxt7d" podUID="f1814527-8ba1-4118-ad92-4454338b4116" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
May 25 11:46:09.763679 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:46:09.763664    2560 scope.go:117] "RemoveContainer" containerID="67dbf3bf33daf769a189851cce0d4f148941781c0a18a067804a9745c3b397f6"
May 25 11:46:09.771706 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:46:09.771687    2560 scope.go:117] "RemoveContainer" containerID="fca6b179d5c50b3592480103eb861d9caa57ffdea2bc3650752aa2f11d0fcc61"
May 25 11:46:09.771936 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:46:09.771920    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fca6b179d5c50b3592480103eb861d9caa57ffdea2bc3650752aa2f11d0fcc61\": container with ID starting with fca6b179d5c50b3592480103eb861d9caa57ffdea2bc3650752aa2f11d0fcc61 not found: ID does not exist" containerID="fca6b179d5c50b3592480103eb861d9caa57ffdea2bc3650752aa2f11d0fcc61"
May 25 11:46:09.771992 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:46:09.771942    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fca6b179d5c50b3592480103eb861d9caa57ffdea2bc3650752aa2f11d0fcc61"} err="failed to get container status \"fca6b179d5c50b3592480103eb861d9caa57ffdea2bc3650752aa2f11d0fcc61\": rpc error: code = NotFound desc = could not find container \"fca6b179d5c50b3592480103eb861d9caa57ffdea2bc3650752aa2f11d0fcc61\": container with ID starting with fca6b179d5c50b3592480103eb861d9caa57ffdea2bc3650752aa2f11d0fcc61 not found: ID does not exist"
May 25 11:46:09.771992 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:46:09.771957    2560 scope.go:117] "RemoveContainer" containerID="67dbf3bf33daf769a189851cce0d4f148941781c0a18a067804a9745c3b397f6"
May 25 11:46:09.772174 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:46:09.772156    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67dbf3bf33daf769a189851cce0d4f148941781c0a18a067804a9745c3b397f6\": container with ID starting with 67dbf3bf33daf769a189851cce0d4f148941781c0a18a067804a9745c3b397f6 not found: ID does not exist" containerID="67dbf3bf33daf769a189851cce0d4f148941781c0a18a067804a9745c3b397f6"
May 25 11:46:09.772223 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:46:09.772181    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67dbf3bf33daf769a189851cce0d4f148941781c0a18a067804a9745c3b397f6"} err="failed to get container status \"67dbf3bf33daf769a189851cce0d4f148941781c0a18a067804a9745c3b397f6\": rpc error: code = NotFound desc = could not find container \"67dbf3bf33daf769a189851cce0d4f148941781c0a18a067804a9745c3b397f6\": container with ID starting with 67dbf3bf33daf769a189851cce0d4f148941781c0a18a067804a9745c3b397f6 not found: ID does not exist"
May 25 11:46:09.777628 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:46:09.777607    2560 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-20622-predictor-57456b66c5-zxs4z"]
May 25 11:46:09.781002 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:46:09.780981    2560 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-20622-predictor-57456b66c5-zxs4z"]
May 25 11:46:10.935704 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:46:10.935667    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6d4b495f-f2db-40db-a7bb-e9b722a5101c" path="/var/lib/kubelet/pods/6d4b495f-f2db-40db-a7bb-e9b722a5101c/volumes"
May 25 11:46:14.760185 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:46:14.760152    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-c3e68-predictor-b87f65c65-xxt7d"
May 25 11:46:14.760742 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:46:14.760712    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c3e68-predictor-b87f65c65-xxt7d" podUID="f1814527-8ba1-4118-ad92-4454338b4116" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
May 25 11:46:24.761116 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:46:24.761071    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c3e68-predictor-b87f65c65-xxt7d" podUID="f1814527-8ba1-4118-ad92-4454338b4116" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
May 25 11:46:34.761300 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:46:34.761259    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c3e68-predictor-b87f65c65-xxt7d" podUID="f1814527-8ba1-4118-ad92-4454338b4116" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
May 25 11:46:35.215981 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:46:35.215945    2560 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-b59e4-predictor-69f5bfb789-d9zqj"]
May 25 11:46:35.216320 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:46:35.216264    2560 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-b59e4-predictor-69f5bfb789-d9zqj" podUID="4919c573-90ea-47be-8e87-88291649d15b" containerName="kserve-container" containerID="cri-o://9da952fff62cf4b8b122cf0d9aec7cc9e5490c21bc65503036b1dadb0d73f075" gracePeriod=30
May 25 11:46:35.216447 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:46:35.216285    2560 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-b59e4-predictor-69f5bfb789-d9zqj" podUID="4919c573-90ea-47be-8e87-88291649d15b" containerName="kube-rbac-proxy" containerID="cri-o://756d18c7dbb4830235ab482f28cfc3b7e48e366813568ec9862b51acbebdb613" gracePeriod=30
May 25 11:46:35.258394 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:46:35.258359    2560 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-c1cab-predictor-6c58cc8979-mf6jk"]
May 25 11:46:35.258760 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:46:35.258746    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d4b495f-f2db-40db-a7bb-e9b722a5101c" containerName="kserve-container"
May 25 11:46:35.258801 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:46:35.258762    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d4b495f-f2db-40db-a7bb-e9b722a5101c" containerName="kserve-container"
May 25 11:46:35.258801 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:46:35.258772    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d4b495f-f2db-40db-a7bb-e9b722a5101c" containerName="kube-rbac-proxy"
May 25 11:46:35.258801 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:46:35.258779    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d4b495f-f2db-40db-a7bb-e9b722a5101c" containerName="kube-rbac-proxy"
May 25 11:46:35.258892 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:46:35.258837    2560 memory_manager.go:356] "RemoveStaleState removing state" podUID="6d4b495f-f2db-40db-a7bb-e9b722a5101c" containerName="kube-rbac-proxy"
May 25 11:46:35.258892 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:46:35.258850    2560 memory_manager.go:356] "RemoveStaleState removing state" podUID="6d4b495f-f2db-40db-a7bb-e9b722a5101c" containerName="kserve-container"
May 25 11:46:35.261903 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:46:35.261887    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-c1cab-predictor-6c58cc8979-mf6jk"
May 25 11:46:35.264567 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:46:35.264545    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-c1cab-predictor-serving-cert\""
May 25 11:46:35.264759 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:46:35.264747    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-c1cab-kube-rbac-proxy-sar-config\""
May 25 11:46:35.272287 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:46:35.272266    2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-c1cab-predictor-6c58cc8979-mf6jk"]
May 25 11:46:35.364774 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:46:35.364742    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fzcb\" (UniqueName: \"kubernetes.io/projected/3ee3c379-e619-4591-bd6a-62de2dc77a02-kube-api-access-5fzcb\") pod \"error-404-isvc-c1cab-predictor-6c58cc8979-mf6jk\" (UID: \"3ee3c379-e619-4591-bd6a-62de2dc77a02\") " pod="kserve-ci-e2e-test/error-404-isvc-c1cab-predictor-6c58cc8979-mf6jk"
May 25 11:46:35.364959 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:46:35.364794    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"error-404-isvc-c1cab-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3ee3c379-e619-4591-bd6a-62de2dc77a02-error-404-isvc-c1cab-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-c1cab-predictor-6c58cc8979-mf6jk\" (UID: \"3ee3c379-e619-4591-bd6a-62de2dc77a02\") " pod="kserve-ci-e2e-test/error-404-isvc-c1cab-predictor-6c58cc8979-mf6jk"
May 25 11:46:35.364959 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:46:35.364867    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3ee3c379-e619-4591-bd6a-62de2dc77a02-proxy-tls\") pod \"error-404-isvc-c1cab-predictor-6c58cc8979-mf6jk\" (UID: \"3ee3c379-e619-4591-bd6a-62de2dc77a02\") " pod="kserve-ci-e2e-test/error-404-isvc-c1cab-predictor-6c58cc8979-mf6jk"
May 25 11:46:35.465554 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:46:35.465519    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"error-404-isvc-c1cab-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3ee3c379-e619-4591-bd6a-62de2dc77a02-error-404-isvc-c1cab-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-c1cab-predictor-6c58cc8979-mf6jk\" (UID: \"3ee3c379-e619-4591-bd6a-62de2dc77a02\") " pod="kserve-ci-e2e-test/error-404-isvc-c1cab-predictor-6c58cc8979-mf6jk"
May 25 11:46:35.465554 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:46:35.465570    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3ee3c379-e619-4591-bd6a-62de2dc77a02-proxy-tls\") pod \"error-404-isvc-c1cab-predictor-6c58cc8979-mf6jk\" (UID: \"3ee3c379-e619-4591-bd6a-62de2dc77a02\") " pod="kserve-ci-e2e-test/error-404-isvc-c1cab-predictor-6c58cc8979-mf6jk"
May 25 11:46:35.465778 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:46:35.465623    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-5fzcb\" (UniqueName: \"kubernetes.io/projected/3ee3c379-e619-4591-bd6a-62de2dc77a02-kube-api-access-5fzcb\") pod \"error-404-isvc-c1cab-predictor-6c58cc8979-mf6jk\" (UID: \"3ee3c379-e619-4591-bd6a-62de2dc77a02\") " pod="kserve-ci-e2e-test/error-404-isvc-c1cab-predictor-6c58cc8979-mf6jk"
May 25 11:46:35.465778 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:46:35.465742    2560 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-c1cab-predictor-serving-cert: secret "error-404-isvc-c1cab-predictor-serving-cert" not found
May 25 11:46:35.465850 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:46:35.465801    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ee3c379-e619-4591-bd6a-62de2dc77a02-proxy-tls podName:3ee3c379-e619-4591-bd6a-62de2dc77a02 nodeName:}" failed. No retries permitted until 2026-05-25 11:46:35.965785422 +0000 UTC m=+1737.646763372 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/3ee3c379-e619-4591-bd6a-62de2dc77a02-proxy-tls") pod "error-404-isvc-c1cab-predictor-6c58cc8979-mf6jk" (UID: "3ee3c379-e619-4591-bd6a-62de2dc77a02") : secret "error-404-isvc-c1cab-predictor-serving-cert" not found
May 25 11:46:35.466242 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:46:35.466178    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"error-404-isvc-c1cab-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3ee3c379-e619-4591-bd6a-62de2dc77a02-error-404-isvc-c1cab-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-c1cab-predictor-6c58cc8979-mf6jk\" (UID: \"3ee3c379-e619-4591-bd6a-62de2dc77a02\") " pod="kserve-ci-e2e-test/error-404-isvc-c1cab-predictor-6c58cc8979-mf6jk"
May 25 11:46:35.477632 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:46:35.477598    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fzcb\" (UniqueName: \"kubernetes.io/projected/3ee3c379-e619-4591-bd6a-62de2dc77a02-kube-api-access-5fzcb\") pod \"error-404-isvc-c1cab-predictor-6c58cc8979-mf6jk\" (UID: \"3ee3c379-e619-4591-bd6a-62de2dc77a02\") " pod="kserve-ci-e2e-test/error-404-isvc-c1cab-predictor-6c58cc8979-mf6jk"
May 25 11:46:35.847086 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:46:35.847047    2560 generic.go:358] "Generic (PLEG): container finished" podID="4919c573-90ea-47be-8e87-88291649d15b" containerID="756d18c7dbb4830235ab482f28cfc3b7e48e366813568ec9862b51acbebdb613" exitCode=2
May 25 11:46:35.847481 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:46:35.847124    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-b59e4-predictor-69f5bfb789-d9zqj" event={"ID":"4919c573-90ea-47be-8e87-88291649d15b","Type":"ContainerDied","Data":"756d18c7dbb4830235ab482f28cfc3b7e48e366813568ec9862b51acbebdb613"}
May 25 11:46:35.969761 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:46:35.969720    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3ee3c379-e619-4591-bd6a-62de2dc77a02-proxy-tls\") pod \"error-404-isvc-c1cab-predictor-6c58cc8979-mf6jk\" (UID: \"3ee3c379-e619-4591-bd6a-62de2dc77a02\") " pod="kserve-ci-e2e-test/error-404-isvc-c1cab-predictor-6c58cc8979-mf6jk"
May 25 11:46:35.972186 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:46:35.972156    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3ee3c379-e619-4591-bd6a-62de2dc77a02-proxy-tls\") pod \"error-404-isvc-c1cab-predictor-6c58cc8979-mf6jk\" (UID: \"3ee3c379-e619-4591-bd6a-62de2dc77a02\") " pod="kserve-ci-e2e-test/error-404-isvc-c1cab-predictor-6c58cc8979-mf6jk"
May 25 11:46:36.172633 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:46:36.172541    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-c1cab-predictor-6c58cc8979-mf6jk"
May 25 11:46:36.306487 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:46:36.306453    2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-c1cab-predictor-6c58cc8979-mf6jk"]
May 25 11:46:36.309839 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:46:36.309812    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ee3c379_e619_4591_bd6a_62de2dc77a02.slice/crio-cf4b304327618f62631059defa7eaae959168257addb2fed811d294138675243 WatchSource:0}: Error finding container cf4b304327618f62631059defa7eaae959168257addb2fed811d294138675243: Status 404 returned error can't find the container with id cf4b304327618f62631059defa7eaae959168257addb2fed811d294138675243
May 25 11:46:36.853228 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:46:36.853183    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-c1cab-predictor-6c58cc8979-mf6jk" event={"ID":"3ee3c379-e619-4591-bd6a-62de2dc77a02","Type":"ContainerStarted","Data":"ba49e1d05af71721e652ba08b91cd8ee8e9fbcb6915e6fb399c13dfabc269276"}
May 25 11:46:36.853665 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:46:36.853244    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-c1cab-predictor-6c58cc8979-mf6jk" event={"ID":"3ee3c379-e619-4591-bd6a-62de2dc77a02","Type":"ContainerStarted","Data":"f15d1008119bd1a9197bbe200be2fa9e9f7b15f88a7247b66cdcb3ef75c48ad9"}
May 25 11:46:36.853665 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:46:36.853256    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-c1cab-predictor-6c58cc8979-mf6jk" event={"ID":"3ee3c379-e619-4591-bd6a-62de2dc77a02","Type":"ContainerStarted","Data":"cf4b304327618f62631059defa7eaae959168257addb2fed811d294138675243"}
May 25 11:46:36.853665 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:46:36.853315    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-c1cab-predictor-6c58cc8979-mf6jk"
May 25 11:46:36.875366 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:46:36.875309    2560 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-c1cab-predictor-6c58cc8979-mf6jk" podStartSLOduration=1.875293916 podStartE2EDuration="1.875293916s" podCreationTimestamp="2026-05-25 11:46:35 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 11:46:36.873340481 +0000 UTC m=+1738.554318453" watchObservedRunningTime="2026-05-25 11:46:36.875293916 +0000 UTC m=+1738.556271889"
May 25 11:46:37.856501 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:46:37.856471    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-c1cab-predictor-6c58cc8979-mf6jk"
May 25 11:46:37.857509 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:46:37.857485    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c1cab-predictor-6c58cc8979-mf6jk" podUID="3ee3c379-e619-4591-bd6a-62de2dc77a02" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
May 25 11:46:37.989702 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:46:37.989656    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-b59e4-predictor-69f5bfb789-d9zqj" podUID="4919c573-90ea-47be-8e87-88291649d15b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.39:8643/healthz\": dial tcp 10.132.0.39:8643: connect: connection refused"
May 25 11:46:37.993960 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:46:37.993933    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-b59e4-predictor-69f5bfb789-d9zqj" podUID="4919c573-90ea-47be-8e87-88291649d15b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
May 25 11:46:38.471217 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:46:38.471180    2560 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-b59e4-predictor-69f5bfb789-d9zqj"
May 25 11:46:38.593754 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:46:38.593728    2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4919c573-90ea-47be-8e87-88291649d15b-proxy-tls\") pod \"4919c573-90ea-47be-8e87-88291649d15b\" (UID: \"4919c573-90ea-47be-8e87-88291649d15b\") "
May 25 11:46:38.593937 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:46:38.593769    2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6wsng\" (UniqueName: \"kubernetes.io/projected/4919c573-90ea-47be-8e87-88291649d15b-kube-api-access-6wsng\") pod \"4919c573-90ea-47be-8e87-88291649d15b\" (UID: \"4919c573-90ea-47be-8e87-88291649d15b\") "
May 25 11:46:38.593937 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:46:38.593792    2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"error-404-isvc-b59e4-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4919c573-90ea-47be-8e87-88291649d15b-error-404-isvc-b59e4-kube-rbac-proxy-sar-config\") pod \"4919c573-90ea-47be-8e87-88291649d15b\" (UID: \"4919c573-90ea-47be-8e87-88291649d15b\") "
May 25 11:46:38.594162 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:46:38.594138    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4919c573-90ea-47be-8e87-88291649d15b-error-404-isvc-b59e4-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "error-404-isvc-b59e4-kube-rbac-proxy-sar-config") pod "4919c573-90ea-47be-8e87-88291649d15b" (UID: "4919c573-90ea-47be-8e87-88291649d15b"). InnerVolumeSpecName "error-404-isvc-b59e4-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 11:46:38.595906 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:46:38.595875    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4919c573-90ea-47be-8e87-88291649d15b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4919c573-90ea-47be-8e87-88291649d15b" (UID: "4919c573-90ea-47be-8e87-88291649d15b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 11:46:38.596010 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:46:38.595926    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4919c573-90ea-47be-8e87-88291649d15b-kube-api-access-6wsng" (OuterVolumeSpecName: "kube-api-access-6wsng") pod "4919c573-90ea-47be-8e87-88291649d15b" (UID: "4919c573-90ea-47be-8e87-88291649d15b"). InnerVolumeSpecName "kube-api-access-6wsng". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 11:46:38.694703 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:46:38.694663    2560 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4919c573-90ea-47be-8e87-88291649d15b-proxy-tls\") on node \"ip-10-0-128-146.ec2.internal\" DevicePath \"\""
May 25 11:46:38.694703 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:46:38.694697    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6wsng\" (UniqueName: \"kubernetes.io/projected/4919c573-90ea-47be-8e87-88291649d15b-kube-api-access-6wsng\") on node \"ip-10-0-128-146.ec2.internal\" DevicePath \"\""
May 25 11:46:38.694703 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:46:38.694708    2560 reconciler_common.go:299] "Volume detached for volume \"error-404-isvc-b59e4-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4919c573-90ea-47be-8e87-88291649d15b-error-404-isvc-b59e4-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-146.ec2.internal\" DevicePath \"\""
May 25 11:46:38.860869 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:46:38.860793    2560 generic.go:358] "Generic (PLEG): container finished" podID="4919c573-90ea-47be-8e87-88291649d15b" containerID="9da952fff62cf4b8b122cf0d9aec7cc9e5490c21bc65503036b1dadb0d73f075" exitCode=0
May 25 11:46:38.860869 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:46:38.860848    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-b59e4-predictor-69f5bfb789-d9zqj" event={"ID":"4919c573-90ea-47be-8e87-88291649d15b","Type":"ContainerDied","Data":"9da952fff62cf4b8b122cf0d9aec7cc9e5490c21bc65503036b1dadb0d73f075"}
May 25 11:46:38.861356 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:46:38.860888    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-b59e4-predictor-69f5bfb789-d9zqj" event={"ID":"4919c573-90ea-47be-8e87-88291649d15b","Type":"ContainerDied","Data":"91c9058325524560745a69f1cdf707d7aea55c1e0ef0a67bacf9aae0e55c9f00"}
May 25 11:46:38.861356 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:46:38.860890    2560 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-b59e4-predictor-69f5bfb789-d9zqj"
May 25 11:46:38.861356 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:46:38.860907    2560 scope.go:117] "RemoveContainer" containerID="756d18c7dbb4830235ab482f28cfc3b7e48e366813568ec9862b51acbebdb613"
May 25 11:46:38.861505 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:46:38.861443    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c1cab-predictor-6c58cc8979-mf6jk" podUID="3ee3c379-e619-4591-bd6a-62de2dc77a02" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
May 25 11:46:38.869440 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:46:38.869420    2560 scope.go:117] "RemoveContainer" containerID="9da952fff62cf4b8b122cf0d9aec7cc9e5490c21bc65503036b1dadb0d73f075"
May 25 11:46:38.876186 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:46:38.876170    2560 scope.go:117] "RemoveContainer" containerID="756d18c7dbb4830235ab482f28cfc3b7e48e366813568ec9862b51acbebdb613"
May 25 11:46:38.876448 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:46:38.876429    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"756d18c7dbb4830235ab482f28cfc3b7e48e366813568ec9862b51acbebdb613\": container with ID starting with 756d18c7dbb4830235ab482f28cfc3b7e48e366813568ec9862b51acbebdb613 not found: ID does not exist" containerID="756d18c7dbb4830235ab482f28cfc3b7e48e366813568ec9862b51acbebdb613"
May 25 11:46:38.876490 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:46:38.876455    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"756d18c7dbb4830235ab482f28cfc3b7e48e366813568ec9862b51acbebdb613"} err="failed to get container status \"756d18c7dbb4830235ab482f28cfc3b7e48e366813568ec9862b51acbebdb613\": rpc error: code = NotFound desc = could not find container \"756d18c7dbb4830235ab482f28cfc3b7e48e366813568ec9862b51acbebdb613\": container with ID starting with 756d18c7dbb4830235ab482f28cfc3b7e48e366813568ec9862b51acbebdb613 not found: ID does not exist"
May 25 11:46:38.876490 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:46:38.876470    2560 scope.go:117] "RemoveContainer" containerID="9da952fff62cf4b8b122cf0d9aec7cc9e5490c21bc65503036b1dadb0d73f075"
May 25 11:46:38.876659 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:46:38.876643    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9da952fff62cf4b8b122cf0d9aec7cc9e5490c21bc65503036b1dadb0d73f075\": container with ID starting with 9da952fff62cf4b8b122cf0d9aec7cc9e5490c21bc65503036b1dadb0d73f075 not found: ID does not exist" containerID="9da952fff62cf4b8b122cf0d9aec7cc9e5490c21bc65503036b1dadb0d73f075"
May 25 11:46:38.876697 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:46:38.876665    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9da952fff62cf4b8b122cf0d9aec7cc9e5490c21bc65503036b1dadb0d73f075"} err="failed to get container status \"9da952fff62cf4b8b122cf0d9aec7cc9e5490c21bc65503036b1dadb0d73f075\": rpc error: code = NotFound desc = could not find container \"9da952fff62cf4b8b122cf0d9aec7cc9e5490c21bc65503036b1dadb0d73f075\": container with ID starting with 9da952fff62cf4b8b122cf0d9aec7cc9e5490c21bc65503036b1dadb0d73f075 not found: ID does not exist"
May 25 11:46:38.884871 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:46:38.884848    2560 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-b59e4-predictor-69f5bfb789-d9zqj"]
May 25 11:46:38.889530 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:46:38.889510    2560 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-b59e4-predictor-69f5bfb789-d9zqj"]
May 25 11:46:38.935864 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:46:38.935835    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4919c573-90ea-47be-8e87-88291649d15b" path="/var/lib/kubelet/pods/4919c573-90ea-47be-8e87-88291649d15b/volumes"
May 25 11:46:43.866140 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:46:43.866103    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-c1cab-predictor-6c58cc8979-mf6jk"
May 25 11:46:43.866581 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:46:43.866556    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c1cab-predictor-6c58cc8979-mf6jk" podUID="3ee3c379-e619-4591-bd6a-62de2dc77a02" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
May 25 11:46:44.760888 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:46:44.760848    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c3e68-predictor-b87f65c65-xxt7d" podUID="f1814527-8ba1-4118-ad92-4454338b4116" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
May 25 11:46:53.867090 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:46:53.867046    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c1cab-predictor-6c58cc8979-mf6jk" podUID="3ee3c379-e619-4591-bd6a-62de2dc77a02" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
May 25 11:46:54.761360 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:46:54.761329    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-c3e68-predictor-b87f65c65-xxt7d"
May 25 11:47:03.866806 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:47:03.866763    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c1cab-predictor-6c58cc8979-mf6jk" podUID="3ee3c379-e619-4591-bd6a-62de2dc77a02" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
May 25 11:47:13.867296 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:47:13.867251    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c1cab-predictor-6c58cc8979-mf6jk" podUID="3ee3c379-e619-4591-bd6a-62de2dc77a02" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
May 25 11:47:16.394753 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:47:16.394718    2560 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-c3e68-predictor-b87f65c65-xxt7d"]
May 25 11:47:16.395266 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:47:16.395138    2560 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-c3e68-predictor-b87f65c65-xxt7d" podUID="f1814527-8ba1-4118-ad92-4454338b4116" containerName="kserve-container" containerID="cri-o://ec05206985f85669abe7f22638944a301917aadb292d7f410563e43424e0feb5" gracePeriod=30
May 25 11:47:16.395266 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:47:16.395173    2560 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-c3e68-predictor-b87f65c65-xxt7d" podUID="f1814527-8ba1-4118-ad92-4454338b4116" containerName="kube-rbac-proxy" containerID="cri-o://c988f0cf1724b58b618758d36b59e753a2259acd53d5dcf70f689e2afd3d3cd9" gracePeriod=30
May 25 11:47:16.453503 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:47:16.453458    2560 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-c7f41-predictor-78fcf75fdf-r2ff6"]
May 25 11:47:16.453878 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:47:16.453864    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4919c573-90ea-47be-8e87-88291649d15b" containerName="kserve-container"
May 25 11:47:16.453921 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:47:16.453880    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="4919c573-90ea-47be-8e87-88291649d15b" containerName="kserve-container"
May 25 11:47:16.453921 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:47:16.453894    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4919c573-90ea-47be-8e87-88291649d15b" containerName="kube-rbac-proxy"
May 25 11:47:16.453921 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:47:16.453900    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="4919c573-90ea-47be-8e87-88291649d15b" containerName="kube-rbac-proxy"
May 25 11:47:16.454016 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:47:16.453954    2560 memory_manager.go:356] "RemoveStaleState removing state" podUID="4919c573-90ea-47be-8e87-88291649d15b" containerName="kube-rbac-proxy"
May 25 11:47:16.454016 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:47:16.453968    2560 memory_manager.go:356] "RemoveStaleState removing state" podUID="4919c573-90ea-47be-8e87-88291649d15b" containerName="kserve-container"
May 25 11:47:16.457247 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:47:16.457231    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-c7f41-predictor-78fcf75fdf-r2ff6"
May 25 11:47:16.460083 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:47:16.460056    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-c7f41-kube-rbac-proxy-sar-config\""
May 25 11:47:16.460235 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:47:16.460121    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-c7f41-predictor-serving-cert\""
May 25 11:47:16.468926 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:47:16.468900    2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-c7f41-predictor-78fcf75fdf-r2ff6"]
May 25 11:47:16.513177 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:47:16.513149    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfvpq\" (UniqueName: \"kubernetes.io/projected/e1ac9df7-55b7-4b46-a326-fd3d3d803fda-kube-api-access-vfvpq\") pod \"error-404-isvc-c7f41-predictor-78fcf75fdf-r2ff6\" (UID: \"e1ac9df7-55b7-4b46-a326-fd3d3d803fda\") " pod="kserve-ci-e2e-test/error-404-isvc-c7f41-predictor-78fcf75fdf-r2ff6"
May 25 11:47:16.513332 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:47:16.513201    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e1ac9df7-55b7-4b46-a326-fd3d3d803fda-proxy-tls\") pod \"error-404-isvc-c7f41-predictor-78fcf75fdf-r2ff6\" (UID: \"e1ac9df7-55b7-4b46-a326-fd3d3d803fda\") " pod="kserve-ci-e2e-test/error-404-isvc-c7f41-predictor-78fcf75fdf-r2ff6"
May 25 11:47:16.513332 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:47:16.513248    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"error-404-isvc-c7f41-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e1ac9df7-55b7-4b46-a326-fd3d3d803fda-error-404-isvc-c7f41-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-c7f41-predictor-78fcf75fdf-r2ff6\" (UID: \"e1ac9df7-55b7-4b46-a326-fd3d3d803fda\") " pod="kserve-ci-e2e-test/error-404-isvc-c7f41-predictor-78fcf75fdf-r2ff6"
May 25 11:47:16.613843 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:47:16.613804    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-vfvpq\" (UniqueName: \"kubernetes.io/projected/e1ac9df7-55b7-4b46-a326-fd3d3d803fda-kube-api-access-vfvpq\") pod \"error-404-isvc-c7f41-predictor-78fcf75fdf-r2ff6\" (UID: \"e1ac9df7-55b7-4b46-a326-fd3d3d803fda\") " pod="kserve-ci-e2e-test/error-404-isvc-c7f41-predictor-78fcf75fdf-r2ff6"
May 25 11:47:16.614051 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:47:16.613866    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e1ac9df7-55b7-4b46-a326-fd3d3d803fda-proxy-tls\") pod \"error-404-isvc-c7f41-predictor-78fcf75fdf-r2ff6\" (UID: \"e1ac9df7-55b7-4b46-a326-fd3d3d803fda\") " pod="kserve-ci-e2e-test/error-404-isvc-c7f41-predictor-78fcf75fdf-r2ff6"
May 25 11:47:16.614051 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:47:16.613893    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"error-404-isvc-c7f41-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e1ac9df7-55b7-4b46-a326-fd3d3d803fda-error-404-isvc-c7f41-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-c7f41-predictor-78fcf75fdf-r2ff6\" (UID: \"e1ac9df7-55b7-4b46-a326-fd3d3d803fda\") " pod="kserve-ci-e2e-test/error-404-isvc-c7f41-predictor-78fcf75fdf-r2ff6"
May 25 11:47:16.614051 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:47:16.614019    2560 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-c7f41-predictor-serving-cert: secret "error-404-isvc-c7f41-predictor-serving-cert" not found
May 25 11:47:16.614242 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:47:16.614087    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e1ac9df7-55b7-4b46-a326-fd3d3d803fda-proxy-tls podName:e1ac9df7-55b7-4b46-a326-fd3d3d803fda nodeName:}" failed. No retries permitted until 2026-05-25 11:47:17.1140714 +0000 UTC m=+1778.795049356 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/e1ac9df7-55b7-4b46-a326-fd3d3d803fda-proxy-tls") pod "error-404-isvc-c7f41-predictor-78fcf75fdf-r2ff6" (UID: "e1ac9df7-55b7-4b46-a326-fd3d3d803fda") : secret "error-404-isvc-c7f41-predictor-serving-cert" not found
May 25 11:47:16.614614 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:47:16.614591    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"error-404-isvc-c7f41-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e1ac9df7-55b7-4b46-a326-fd3d3d803fda-error-404-isvc-c7f41-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-c7f41-predictor-78fcf75fdf-r2ff6\" (UID: \"e1ac9df7-55b7-4b46-a326-fd3d3d803fda\") " pod="kserve-ci-e2e-test/error-404-isvc-c7f41-predictor-78fcf75fdf-r2ff6"
May 25 11:47:16.627386 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:47:16.627355    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfvpq\" (UniqueName: \"kubernetes.io/projected/e1ac9df7-55b7-4b46-a326-fd3d3d803fda-kube-api-access-vfvpq\") pod \"error-404-isvc-c7f41-predictor-78fcf75fdf-r2ff6\" (UID: \"e1ac9df7-55b7-4b46-a326-fd3d3d803fda\") " pod="kserve-ci-e2e-test/error-404-isvc-c7f41-predictor-78fcf75fdf-r2ff6"
May 25 11:47:16.990907 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:47:16.990868    2560 generic.go:358] "Generic (PLEG): container finished" podID="f1814527-8ba1-4118-ad92-4454338b4116" containerID="c988f0cf1724b58b618758d36b59e753a2259acd53d5dcf70f689e2afd3d3cd9" exitCode=2
May 25 11:47:16.991078 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:47:16.990934    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-c3e68-predictor-b87f65c65-xxt7d" event={"ID":"f1814527-8ba1-4118-ad92-4454338b4116","Type":"ContainerDied","Data":"c988f0cf1724b58b618758d36b59e753a2259acd53d5dcf70f689e2afd3d3cd9"}
May 25 11:47:17.119400 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:47:17.119368    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e1ac9df7-55b7-4b46-a326-fd3d3d803fda-proxy-tls\") pod \"error-404-isvc-c7f41-predictor-78fcf75fdf-r2ff6\" (UID: \"e1ac9df7-55b7-4b46-a326-fd3d3d803fda\") " pod="kserve-ci-e2e-test/error-404-isvc-c7f41-predictor-78fcf75fdf-r2ff6"
May 25 11:47:17.121802 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:47:17.121779    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e1ac9df7-55b7-4b46-a326-fd3d3d803fda-proxy-tls\") pod \"error-404-isvc-c7f41-predictor-78fcf75fdf-r2ff6\" (UID: \"e1ac9df7-55b7-4b46-a326-fd3d3d803fda\") " pod="kserve-ci-e2e-test/error-404-isvc-c7f41-predictor-78fcf75fdf-r2ff6"
May 25 11:47:17.367502 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:47:17.367470    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-c7f41-predictor-78fcf75fdf-r2ff6"
May 25 11:47:17.501236 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:47:17.501135    2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-c7f41-predictor-78fcf75fdf-r2ff6"]
May 25 11:47:17.503700 ip-10-0-128-146 kubenswrapper[2560]: W0525 11:47:17.503672    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1ac9df7_55b7_4b46_a326_fd3d3d803fda.slice/crio-4a620c43190aff58bdc2eeb61caac9549f1e28acfe16b53a948a269cbfdd138a WatchSource:0}: Error finding container 4a620c43190aff58bdc2eeb61caac9549f1e28acfe16b53a948a269cbfdd138a: Status 404 returned error can't find the container with id 4a620c43190aff58bdc2eeb61caac9549f1e28acfe16b53a948a269cbfdd138a
May 25 11:47:17.996259 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:47:17.996121    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-c7f41-predictor-78fcf75fdf-r2ff6" event={"ID":"e1ac9df7-55b7-4b46-a326-fd3d3d803fda","Type":"ContainerStarted","Data":"e142f45b81b62ff8daf72a3a3e0581b54b9ad02d377b7a77fe24a5224b81d15a"}
May 25 11:47:17.996259 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:47:17.996159    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-c7f41-predictor-78fcf75fdf-r2ff6" event={"ID":"e1ac9df7-55b7-4b46-a326-fd3d3d803fda","Type":"ContainerStarted","Data":"0242b9466f8731020c267b1b825ecaa0b79b90e05cfc38a4f05348273128930a"}
May 25 11:47:17.996259 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:47:17.996170    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-c7f41-predictor-78fcf75fdf-r2ff6" event={"ID":"e1ac9df7-55b7-4b46-a326-fd3d3d803fda","Type":"ContainerStarted","Data":"4a620c43190aff58bdc2eeb61caac9549f1e28acfe16b53a948a269cbfdd138a"}
May 25 11:47:17.996259 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:47:17.996252    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-c7f41-predictor-78fcf75fdf-r2ff6"
May 25 11:47:18.022678 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:47:18.022628    2560 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-c7f41-predictor-78fcf75fdf-r2ff6" podStartSLOduration=2.022613912 podStartE2EDuration="2.022613912s" podCreationTimestamp="2026-05-25 11:47:16 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 11:47:18.020126328 +0000 UTC m=+1779.701104300" watchObservedRunningTime="2026-05-25 11:47:18.022613912 +0000 UTC m=+1779.703591886"
May 25 11:47:18.999180 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:47:18.999139    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-c7f41-predictor-78fcf75fdf-r2ff6"
May 25 11:47:19.000349 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:47:19.000319    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c7f41-predictor-78fcf75fdf-r2ff6" podUID="e1ac9df7-55b7-4b46-a326-fd3d3d803fda" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
May 25 11:47:19.734154 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:47:19.734131    2560 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-c3e68-predictor-b87f65c65-xxt7d"
May 25 11:47:19.741906 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:47:19.741887    2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f1814527-8ba1-4118-ad92-4454338b4116-proxy-tls\") pod \"f1814527-8ba1-4118-ad92-4454338b4116\" (UID: \"f1814527-8ba1-4118-ad92-4454338b4116\") "
May 25 11:47:19.742023 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:47:19.741915    2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-48nwf\" (UniqueName: \"kubernetes.io/projected/f1814527-8ba1-4118-ad92-4454338b4116-kube-api-access-48nwf\") pod \"f1814527-8ba1-4118-ad92-4454338b4116\" (UID: \"f1814527-8ba1-4118-ad92-4454338b4116\") "
May 25 11:47:19.742023 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:47:19.741981    2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"error-404-isvc-c3e68-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f1814527-8ba1-4118-ad92-4454338b4116-error-404-isvc-c3e68-kube-rbac-proxy-sar-config\") pod \"f1814527-8ba1-4118-ad92-4454338b4116\" (UID: \"f1814527-8ba1-4118-ad92-4454338b4116\") "
May 25 11:47:19.742419 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:47:19.742390    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1814527-8ba1-4118-ad92-4454338b4116-error-404-isvc-c3e68-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "error-404-isvc-c3e68-kube-rbac-proxy-sar-config") pod "f1814527-8ba1-4118-ad92-4454338b4116" (UID: "f1814527-8ba1-4118-ad92-4454338b4116"). InnerVolumeSpecName "error-404-isvc-c3e68-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 11:47:19.744128 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:47:19.744106    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1814527-8ba1-4118-ad92-4454338b4116-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f1814527-8ba1-4118-ad92-4454338b4116" (UID: "f1814527-8ba1-4118-ad92-4454338b4116"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 11:47:19.744251 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:47:19.744172    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1814527-8ba1-4118-ad92-4454338b4116-kube-api-access-48nwf" (OuterVolumeSpecName: "kube-api-access-48nwf") pod "f1814527-8ba1-4118-ad92-4454338b4116" (UID: "f1814527-8ba1-4118-ad92-4454338b4116"). InnerVolumeSpecName "kube-api-access-48nwf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 11:47:19.843644 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:47:19.843608    2560 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f1814527-8ba1-4118-ad92-4454338b4116-proxy-tls\") on node \"ip-10-0-128-146.ec2.internal\" DevicePath \"\""
May 25 11:47:19.843795 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:47:19.843651    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-48nwf\" (UniqueName: \"kubernetes.io/projected/f1814527-8ba1-4118-ad92-4454338b4116-kube-api-access-48nwf\") on node \"ip-10-0-128-146.ec2.internal\" DevicePath \"\""
May 25 11:47:19.843795 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:47:19.843667    2560 reconciler_common.go:299] "Volume detached for volume \"error-404-isvc-c3e68-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f1814527-8ba1-4118-ad92-4454338b4116-error-404-isvc-c3e68-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-146.ec2.internal\" DevicePath \"\""
May 25 11:47:20.003458 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:47:20.003422    2560 generic.go:358] "Generic (PLEG): container finished" podID="f1814527-8ba1-4118-ad92-4454338b4116" containerID="ec05206985f85669abe7f22638944a301917aadb292d7f410563e43424e0feb5" exitCode=0
May 25 11:47:20.003917 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:47:20.003508    2560 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-c3e68-predictor-b87f65c65-xxt7d"
May 25 11:47:20.003917 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:47:20.003509    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-c3e68-predictor-b87f65c65-xxt7d" event={"ID":"f1814527-8ba1-4118-ad92-4454338b4116","Type":"ContainerDied","Data":"ec05206985f85669abe7f22638944a301917aadb292d7f410563e43424e0feb5"}
May 25 11:47:20.003917 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:47:20.003608    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-c3e68-predictor-b87f65c65-xxt7d" event={"ID":"f1814527-8ba1-4118-ad92-4454338b4116","Type":"ContainerDied","Data":"078fd49e6dbb02b6695a765b6af1ce37ff4c78dccaa0117543405f8770a4f030"}
May 25 11:47:20.003917 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:47:20.003624    2560 scope.go:117] "RemoveContainer" containerID="c988f0cf1724b58b618758d36b59e753a2259acd53d5dcf70f689e2afd3d3cd9"
May 25 11:47:20.004188 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:47:20.004163    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c7f41-predictor-78fcf75fdf-r2ff6" podUID="e1ac9df7-55b7-4b46-a326-fd3d3d803fda" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
May 25 11:47:20.012108 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:47:20.012089    2560 scope.go:117] "RemoveContainer" containerID="ec05206985f85669abe7f22638944a301917aadb292d7f410563e43424e0feb5"
May 25 11:47:20.019374 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:47:20.019356    2560 scope.go:117] "RemoveContainer" containerID="c988f0cf1724b58b618758d36b59e753a2259acd53d5dcf70f689e2afd3d3cd9"
May 25 11:47:20.019633 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:47:20.019613    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c988f0cf1724b58b618758d36b59e753a2259acd53d5dcf70f689e2afd3d3cd9\": container with ID starting with c988f0cf1724b58b618758d36b59e753a2259acd53d5dcf70f689e2afd3d3cd9 not found: ID does not exist" containerID="c988f0cf1724b58b618758d36b59e753a2259acd53d5dcf70f689e2afd3d3cd9"
May 25 11:47:20.019732 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:47:20.019639    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c988f0cf1724b58b618758d36b59e753a2259acd53d5dcf70f689e2afd3d3cd9"} err="failed to get container status \"c988f0cf1724b58b618758d36b59e753a2259acd53d5dcf70f689e2afd3d3cd9\": rpc error: code = NotFound desc = could not find container \"c988f0cf1724b58b618758d36b59e753a2259acd53d5dcf70f689e2afd3d3cd9\": container with ID starting with c988f0cf1724b58b618758d36b59e753a2259acd53d5dcf70f689e2afd3d3cd9 not found: ID does not exist"
May 25 11:47:20.019732 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:47:20.019657    2560 scope.go:117] "RemoveContainer" containerID="ec05206985f85669abe7f22638944a301917aadb292d7f410563e43424e0feb5"
May 25 11:47:20.019876 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:47:20.019861    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec05206985f85669abe7f22638944a301917aadb292d7f410563e43424e0feb5\": container with ID starting with ec05206985f85669abe7f22638944a301917aadb292d7f410563e43424e0feb5 not found: ID does not exist" containerID="ec05206985f85669abe7f22638944a301917aadb292d7f410563e43424e0feb5"
May 25 11:47:20.019916 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:47:20.019882    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec05206985f85669abe7f22638944a301917aadb292d7f410563e43424e0feb5"} err="failed to get container status \"ec05206985f85669abe7f22638944a301917aadb292d7f410563e43424e0feb5\": rpc error: code = NotFound desc = could not find container \"ec05206985f85669abe7f22638944a301917aadb292d7f410563e43424e0feb5\": container with ID starting with ec05206985f85669abe7f22638944a301917aadb292d7f410563e43424e0feb5 not found: ID does not exist"
May 25 11:47:20.025599 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:47:20.025569    2560 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-c3e68-predictor-b87f65c65-xxt7d"]
May 25 11:47:20.029331 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:47:20.029308    2560 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-c3e68-predictor-b87f65c65-xxt7d"]
May 25 11:47:20.935151 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:47:20.935119    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f1814527-8ba1-4118-ad92-4454338b4116" path="/var/lib/kubelet/pods/f1814527-8ba1-4118-ad92-4454338b4116/volumes"
May 25 11:47:23.867376 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:47:23.867348    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-c1cab-predictor-6c58cc8979-mf6jk"
May 25 11:47:25.008840 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:47:25.008812    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-c7f41-predictor-78fcf75fdf-r2ff6"
May 25 11:47:25.009352 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:47:25.009325    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c7f41-predictor-78fcf75fdf-r2ff6" podUID="e1ac9df7-55b7-4b46-a326-fd3d3d803fda" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
May 25 11:47:35.009360 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:47:35.009316    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c7f41-predictor-78fcf75fdf-r2ff6" podUID="e1ac9df7-55b7-4b46-a326-fd3d3d803fda" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
May 25 11:47:38.952045 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:47:38.952013    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-zd54m_2be32661-9288-4902-9d67-1a5e8f74670d/console-operator/1.log"
May 25 11:47:38.957971 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:47:38.957949    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-zd54m_2be32661-9288-4902-9d67-1a5e8f74670d/console-operator/1.log"
May 25 11:47:38.958479 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:47:38.958464    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m55v2_b93f3e9c-f4f9-49dc-bfba-6d0f80c0971d/ovn-acl-logging/0.log"
May 25 11:47:38.963773 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:47:38.963758    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m55v2_b93f3e9c-f4f9-49dc-bfba-6d0f80c0971d/ovn-acl-logging/0.log"
May 25 11:47:45.010286 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:47:45.010246    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c7f41-predictor-78fcf75fdf-r2ff6" podUID="e1ac9df7-55b7-4b46-a326-fd3d3d803fda" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
May 25 11:47:55.010317 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:47:55.010271    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c7f41-predictor-78fcf75fdf-r2ff6" podUID="e1ac9df7-55b7-4b46-a326-fd3d3d803fda" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
May 25 11:48:05.010404 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:48:05.010372    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-c7f41-predictor-78fcf75fdf-r2ff6"
May 25 11:52:38.980877 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:52:38.980849    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-zd54m_2be32661-9288-4902-9d67-1a5e8f74670d/console-operator/1.log"
May 25 11:52:38.985874 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:52:38.985850    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-zd54m_2be32661-9288-4902-9d67-1a5e8f74670d/console-operator/1.log"
May 25 11:52:38.987627 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:52:38.987606    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m55v2_b93f3e9c-f4f9-49dc-bfba-6d0f80c0971d/ovn-acl-logging/0.log"
May 25 11:52:38.992323 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:52:38.992305    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m55v2_b93f3e9c-f4f9-49dc-bfba-6d0f80c0971d/ovn-acl-logging/0.log"
May 25 11:56:31.043718 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:56:31.043630    2560 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-c7f41-predictor-78fcf75fdf-r2ff6"]
May 25 11:56:31.046290 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:56:31.044006    2560 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-c7f41-predictor-78fcf75fdf-r2ff6" podUID="e1ac9df7-55b7-4b46-a326-fd3d3d803fda" containerName="kserve-container" containerID="cri-o://0242b9466f8731020c267b1b825ecaa0b79b90e05cfc38a4f05348273128930a" gracePeriod=30
May 25 11:56:31.046290 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:56:31.044025    2560 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-c7f41-predictor-78fcf75fdf-r2ff6" podUID="e1ac9df7-55b7-4b46-a326-fd3d3d803fda" containerName="kube-rbac-proxy" containerID="cri-o://e142f45b81b62ff8daf72a3a3e0581b54b9ad02d377b7a77fe24a5224b81d15a" gracePeriod=30
May 25 11:56:31.836646 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:56:31.836612    2560 generic.go:358] "Generic (PLEG): container finished" podID="e1ac9df7-55b7-4b46-a326-fd3d3d803fda" containerID="e142f45b81b62ff8daf72a3a3e0581b54b9ad02d377b7a77fe24a5224b81d15a" exitCode=2
May 25 11:56:31.836823 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:56:31.836684    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-c7f41-predictor-78fcf75fdf-r2ff6" event={"ID":"e1ac9df7-55b7-4b46-a326-fd3d3d803fda","Type":"ContainerDied","Data":"e142f45b81b62ff8daf72a3a3e0581b54b9ad02d377b7a77fe24a5224b81d15a"}
May 25 11:56:34.194807 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:56:34.194780    2560 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-c7f41-predictor-78fcf75fdf-r2ff6"
May 25 11:56:34.261991 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:56:34.261905    2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"error-404-isvc-c7f41-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e1ac9df7-55b7-4b46-a326-fd3d3d803fda-error-404-isvc-c7f41-kube-rbac-proxy-sar-config\") pod \"e1ac9df7-55b7-4b46-a326-fd3d3d803fda\" (UID: \"e1ac9df7-55b7-4b46-a326-fd3d3d803fda\") "
May 25 11:56:34.261991 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:56:34.261981    2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e1ac9df7-55b7-4b46-a326-fd3d3d803fda-proxy-tls\") pod \"e1ac9df7-55b7-4b46-a326-fd3d3d803fda\" (UID: \"e1ac9df7-55b7-4b46-a326-fd3d3d803fda\") "
May 25 11:56:34.262182 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:56:34.262013    2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vfvpq\" (UniqueName: \"kubernetes.io/projected/e1ac9df7-55b7-4b46-a326-fd3d3d803fda-kube-api-access-vfvpq\") pod \"e1ac9df7-55b7-4b46-a326-fd3d3d803fda\" (UID: \"e1ac9df7-55b7-4b46-a326-fd3d3d803fda\") "
May 25 11:56:34.262298 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:56:34.262274    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1ac9df7-55b7-4b46-a326-fd3d3d803fda-error-404-isvc-c7f41-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "error-404-isvc-c7f41-kube-rbac-proxy-sar-config") pod "e1ac9df7-55b7-4b46-a326-fd3d3d803fda" (UID: "e1ac9df7-55b7-4b46-a326-fd3d3d803fda"). InnerVolumeSpecName "error-404-isvc-c7f41-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 11:56:34.263995 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:56:34.263970    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1ac9df7-55b7-4b46-a326-fd3d3d803fda-kube-api-access-vfvpq" (OuterVolumeSpecName: "kube-api-access-vfvpq") pod "e1ac9df7-55b7-4b46-a326-fd3d3d803fda" (UID: "e1ac9df7-55b7-4b46-a326-fd3d3d803fda"). InnerVolumeSpecName "kube-api-access-vfvpq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 11:56:34.264124 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:56:34.264065    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1ac9df7-55b7-4b46-a326-fd3d3d803fda-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e1ac9df7-55b7-4b46-a326-fd3d3d803fda" (UID: "e1ac9df7-55b7-4b46-a326-fd3d3d803fda"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 11:56:34.362582 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:56:34.362549    2560 reconciler_common.go:299] "Volume detached for volume \"error-404-isvc-c7f41-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e1ac9df7-55b7-4b46-a326-fd3d3d803fda-error-404-isvc-c7f41-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-146.ec2.internal\" DevicePath \"\""
May 25 11:56:34.362582 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:56:34.362579    2560 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e1ac9df7-55b7-4b46-a326-fd3d3d803fda-proxy-tls\") on node \"ip-10-0-128-146.ec2.internal\" DevicePath \"\""
May 25 11:56:34.362582 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:56:34.362590    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vfvpq\" (UniqueName: \"kubernetes.io/projected/e1ac9df7-55b7-4b46-a326-fd3d3d803fda-kube-api-access-vfvpq\") on node \"ip-10-0-128-146.ec2.internal\" DevicePath \"\""
May 25 11:56:34.848269 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:56:34.848231    2560 generic.go:358] "Generic (PLEG): container finished" podID="e1ac9df7-55b7-4b46-a326-fd3d3d803fda" containerID="0242b9466f8731020c267b1b825ecaa0b79b90e05cfc38a4f05348273128930a" exitCode=0
May 25 11:56:34.848440 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:56:34.848300    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-c7f41-predictor-78fcf75fdf-r2ff6" event={"ID":"e1ac9df7-55b7-4b46-a326-fd3d3d803fda","Type":"ContainerDied","Data":"0242b9466f8731020c267b1b825ecaa0b79b90e05cfc38a4f05348273128930a"}
May 25 11:56:34.848440 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:56:34.848329    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-c7f41-predictor-78fcf75fdf-r2ff6" event={"ID":"e1ac9df7-55b7-4b46-a326-fd3d3d803fda","Type":"ContainerDied","Data":"4a620c43190aff58bdc2eeb61caac9549f1e28acfe16b53a948a269cbfdd138a"}
May 25 11:56:34.848440 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:56:34.848345    2560 scope.go:117] "RemoveContainer" containerID="e142f45b81b62ff8daf72a3a3e0581b54b9ad02d377b7a77fe24a5224b81d15a"
May 25 11:56:34.848440 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:56:34.848306    2560 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-c7f41-predictor-78fcf75fdf-r2ff6"
May 25 11:56:34.858884 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:56:34.858861    2560 scope.go:117] "RemoveContainer" containerID="0242b9466f8731020c267b1b825ecaa0b79b90e05cfc38a4f05348273128930a"
May 25 11:56:34.866918 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:56:34.866890    2560 scope.go:117] "RemoveContainer" containerID="e142f45b81b62ff8daf72a3a3e0581b54b9ad02d377b7a77fe24a5224b81d15a"
May 25 11:56:34.867626 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:56:34.867597    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e142f45b81b62ff8daf72a3a3e0581b54b9ad02d377b7a77fe24a5224b81d15a\": container with ID starting with e142f45b81b62ff8daf72a3a3e0581b54b9ad02d377b7a77fe24a5224b81d15a not found: ID does not exist" containerID="e142f45b81b62ff8daf72a3a3e0581b54b9ad02d377b7a77fe24a5224b81d15a"
May 25 11:56:34.867732 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:56:34.867639    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e142f45b81b62ff8daf72a3a3e0581b54b9ad02d377b7a77fe24a5224b81d15a"} err="failed to get container status \"e142f45b81b62ff8daf72a3a3e0581b54b9ad02d377b7a77fe24a5224b81d15a\": rpc error: code = NotFound desc = could not find container \"e142f45b81b62ff8daf72a3a3e0581b54b9ad02d377b7a77fe24a5224b81d15a\": container with ID starting with e142f45b81b62ff8daf72a3a3e0581b54b9ad02d377b7a77fe24a5224b81d15a not found: ID does not exist"
May 25 11:56:34.867732 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:56:34.867666    2560 scope.go:117] "RemoveContainer" containerID="0242b9466f8731020c267b1b825ecaa0b79b90e05cfc38a4f05348273128930a"
May 25 11:56:34.867950 ip-10-0-128-146 kubenswrapper[2560]: E0525 11:56:34.867934    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0242b9466f8731020c267b1b825ecaa0b79b90e05cfc38a4f05348273128930a\": container with ID starting with 0242b9466f8731020c267b1b825ecaa0b79b90e05cfc38a4f05348273128930a not found: ID does not exist" containerID="0242b9466f8731020c267b1b825ecaa0b79b90e05cfc38a4f05348273128930a"
May 25 11:56:34.868004 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:56:34.867966    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0242b9466f8731020c267b1b825ecaa0b79b90e05cfc38a4f05348273128930a"} err="failed to get container status \"0242b9466f8731020c267b1b825ecaa0b79b90e05cfc38a4f05348273128930a\": rpc error: code = NotFound desc = could not find container \"0242b9466f8731020c267b1b825ecaa0b79b90e05cfc38a4f05348273128930a\": container with ID starting with 0242b9466f8731020c267b1b825ecaa0b79b90e05cfc38a4f05348273128930a not found: ID does not exist"
May 25 11:56:34.869277 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:56:34.869256    2560 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-c7f41-predictor-78fcf75fdf-r2ff6"]
May 25 11:56:34.872019 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:56:34.872000    2560 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-c7f41-predictor-78fcf75fdf-r2ff6"]
May 25 11:56:34.934712 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:56:34.934672    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e1ac9df7-55b7-4b46-a326-fd3d3d803fda" path="/var/lib/kubelet/pods/e1ac9df7-55b7-4b46-a326-fd3d3d803fda/volumes"
May 25 11:57:39.003883 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:57:39.003852    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-zd54m_2be32661-9288-4902-9d67-1a5e8f74670d/console-operator/1.log"
May 25 11:57:39.008920 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:57:39.008897    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-zd54m_2be32661-9288-4902-9d67-1a5e8f74670d/console-operator/1.log"
May 25 11:57:39.010280 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:57:39.010257    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m55v2_b93f3e9c-f4f9-49dc-bfba-6d0f80c0971d/ovn-acl-logging/0.log"
May 25 11:57:39.014519 ip-10-0-128-146 kubenswrapper[2560]: I0525 11:57:39.014504    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m55v2_b93f3e9c-f4f9-49dc-bfba-6d0f80c0971d/ovn-acl-logging/0.log"
May 25 12:02:39.026706 ip-10-0-128-146 kubenswrapper[2560]: I0525 12:02:39.026633    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-zd54m_2be32661-9288-4902-9d67-1a5e8f74670d/console-operator/1.log"
May 25 12:02:39.031823 ip-10-0-128-146 kubenswrapper[2560]: I0525 12:02:39.031801    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-zd54m_2be32661-9288-4902-9d67-1a5e8f74670d/console-operator/1.log"
May 25 12:02:39.033094 ip-10-0-128-146 kubenswrapper[2560]: I0525 12:02:39.033074    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m55v2_b93f3e9c-f4f9-49dc-bfba-6d0f80c0971d/ovn-acl-logging/0.log"
May 25 12:02:39.039315 ip-10-0-128-146 kubenswrapper[2560]: I0525 12:02:39.039297    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m55v2_b93f3e9c-f4f9-49dc-bfba-6d0f80c0971d/ovn-acl-logging/0.log"
May 25 12:03:54.534197 ip-10-0-128-146 kubenswrapper[2560]: I0525 12:03:54.534122    2560 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-c1cab-predictor-6c58cc8979-mf6jk"]
May 25 12:03:54.534662 ip-10-0-128-146 kubenswrapper[2560]: I0525 12:03:54.534434    2560 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-c1cab-predictor-6c58cc8979-mf6jk" podUID="3ee3c379-e619-4591-bd6a-62de2dc77a02" containerName="kserve-container" containerID="cri-o://f15d1008119bd1a9197bbe200be2fa9e9f7b15f88a7247b66cdcb3ef75c48ad9" gracePeriod=30
May 25 12:03:54.534662 ip-10-0-128-146 kubenswrapper[2560]: I0525 12:03:54.534451    2560 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-c1cab-predictor-6c58cc8979-mf6jk" podUID="3ee3c379-e619-4591-bd6a-62de2dc77a02" containerName="kube-rbac-proxy" containerID="cri-o://ba49e1d05af71721e652ba08b91cd8ee8e9fbcb6915e6fb399c13dfabc269276" gracePeriod=30
May 25 12:03:55.276021 ip-10-0-128-146 kubenswrapper[2560]: I0525 12:03:55.275982    2560 generic.go:358] "Generic (PLEG): container finished" podID="3ee3c379-e619-4591-bd6a-62de2dc77a02" containerID="ba49e1d05af71721e652ba08b91cd8ee8e9fbcb6915e6fb399c13dfabc269276" exitCode=2
May 25 12:03:55.276198 ip-10-0-128-146 kubenswrapper[2560]: I0525 12:03:55.276050    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-c1cab-predictor-6c58cc8979-mf6jk" event={"ID":"3ee3c379-e619-4591-bd6a-62de2dc77a02","Type":"ContainerDied","Data":"ba49e1d05af71721e652ba08b91cd8ee8e9fbcb6915e6fb399c13dfabc269276"}
May 25 12:03:57.583434 ip-10-0-128-146 kubenswrapper[2560]: I0525 12:03:57.583411    2560 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-c1cab-predictor-6c58cc8979-mf6jk"
May 25 12:03:57.638520 ip-10-0-128-146 kubenswrapper[2560]: I0525 12:03:57.638453    2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3ee3c379-e619-4591-bd6a-62de2dc77a02-proxy-tls\") pod \"3ee3c379-e619-4591-bd6a-62de2dc77a02\" (UID: \"3ee3c379-e619-4591-bd6a-62de2dc77a02\") "
May 25 12:03:57.638520 ip-10-0-128-146 kubenswrapper[2560]: I0525 12:03:57.638487    2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"error-404-isvc-c1cab-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3ee3c379-e619-4591-bd6a-62de2dc77a02-error-404-isvc-c1cab-kube-rbac-proxy-sar-config\") pod \"3ee3c379-e619-4591-bd6a-62de2dc77a02\" (UID: \"3ee3c379-e619-4591-bd6a-62de2dc77a02\") "
May 25 12:03:57.638668 ip-10-0-128-146 kubenswrapper[2560]: I0525 12:03:57.638556    2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5fzcb\" (UniqueName: \"kubernetes.io/projected/3ee3c379-e619-4591-bd6a-62de2dc77a02-kube-api-access-5fzcb\") pod \"3ee3c379-e619-4591-bd6a-62de2dc77a02\" (UID: \"3ee3c379-e619-4591-bd6a-62de2dc77a02\") "
May 25 12:03:57.638892 ip-10-0-128-146 kubenswrapper[2560]: I0525 12:03:57.638869    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ee3c379-e619-4591-bd6a-62de2dc77a02-error-404-isvc-c1cab-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "error-404-isvc-c1cab-kube-rbac-proxy-sar-config") pod "3ee3c379-e619-4591-bd6a-62de2dc77a02" (UID: "3ee3c379-e619-4591-bd6a-62de2dc77a02"). InnerVolumeSpecName "error-404-isvc-c1cab-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 12:03:57.640589 ip-10-0-128-146 kubenswrapper[2560]: I0525 12:03:57.640567    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ee3c379-e619-4591-bd6a-62de2dc77a02-kube-api-access-5fzcb" (OuterVolumeSpecName: "kube-api-access-5fzcb") pod "3ee3c379-e619-4591-bd6a-62de2dc77a02" (UID: "3ee3c379-e619-4591-bd6a-62de2dc77a02"). InnerVolumeSpecName "kube-api-access-5fzcb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 12:03:57.640688 ip-10-0-128-146 kubenswrapper[2560]: I0525 12:03:57.640596    2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ee3c379-e619-4591-bd6a-62de2dc77a02-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3ee3c379-e619-4591-bd6a-62de2dc77a02" (UID: "3ee3c379-e619-4591-bd6a-62de2dc77a02"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 12:03:57.738967 ip-10-0-128-146 kubenswrapper[2560]: I0525 12:03:57.738941    2560 reconciler_common.go:299] "Volume detached for volume \"error-404-isvc-c1cab-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3ee3c379-e619-4591-bd6a-62de2dc77a02-error-404-isvc-c1cab-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-146.ec2.internal\" DevicePath \"\""
May 25 12:03:57.738967 ip-10-0-128-146 kubenswrapper[2560]: I0525 12:03:57.738965    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5fzcb\" (UniqueName: \"kubernetes.io/projected/3ee3c379-e619-4591-bd6a-62de2dc77a02-kube-api-access-5fzcb\") on node \"ip-10-0-128-146.ec2.internal\" DevicePath \"\""
May 25 12:03:57.739126 ip-10-0-128-146 kubenswrapper[2560]: I0525 12:03:57.738974    2560 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3ee3c379-e619-4591-bd6a-62de2dc77a02-proxy-tls\") on node \"ip-10-0-128-146.ec2.internal\" DevicePath \"\""
May 25 12:03:58.286298 ip-10-0-128-146 kubenswrapper[2560]: I0525 12:03:58.286258    2560 generic.go:358] "Generic (PLEG): container finished" podID="3ee3c379-e619-4591-bd6a-62de2dc77a02" containerID="f15d1008119bd1a9197bbe200be2fa9e9f7b15f88a7247b66cdcb3ef75c48ad9" exitCode=0
May 25 12:03:58.286471 ip-10-0-128-146 kubenswrapper[2560]: I0525 12:03:58.286337    2560 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-c1cab-predictor-6c58cc8979-mf6jk"
May 25 12:03:58.286471 ip-10-0-128-146 kubenswrapper[2560]: I0525 12:03:58.286341    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-c1cab-predictor-6c58cc8979-mf6jk" event={"ID":"3ee3c379-e619-4591-bd6a-62de2dc77a02","Type":"ContainerDied","Data":"f15d1008119bd1a9197bbe200be2fa9e9f7b15f88a7247b66cdcb3ef75c48ad9"}
May 25 12:03:58.286471 ip-10-0-128-146 kubenswrapper[2560]: I0525 12:03:58.286382    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-c1cab-predictor-6c58cc8979-mf6jk" event={"ID":"3ee3c379-e619-4591-bd6a-62de2dc77a02","Type":"ContainerDied","Data":"cf4b304327618f62631059defa7eaae959168257addb2fed811d294138675243"}
May 25 12:03:58.286471 ip-10-0-128-146 kubenswrapper[2560]: I0525 12:03:58.286398    2560 scope.go:117] "RemoveContainer" containerID="ba49e1d05af71721e652ba08b91cd8ee8e9fbcb6915e6fb399c13dfabc269276"
May 25 12:03:58.300353 ip-10-0-128-146 kubenswrapper[2560]: I0525 12:03:58.300332    2560 scope.go:117] "RemoveContainer" containerID="f15d1008119bd1a9197bbe200be2fa9e9f7b15f88a7247b66cdcb3ef75c48ad9"
May 25 12:03:58.308239 ip-10-0-128-146 kubenswrapper[2560]: I0525 12:03:58.308013    2560 scope.go:117] "RemoveContainer" containerID="ba49e1d05af71721e652ba08b91cd8ee8e9fbcb6915e6fb399c13dfabc269276"
May 25 12:03:58.308908 ip-10-0-128-146 kubenswrapper[2560]: E0525 12:03:58.308882    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba49e1d05af71721e652ba08b91cd8ee8e9fbcb6915e6fb399c13dfabc269276\": container with ID starting with ba49e1d05af71721e652ba08b91cd8ee8e9fbcb6915e6fb399c13dfabc269276 not found: ID does not exist" containerID="ba49e1d05af71721e652ba08b91cd8ee8e9fbcb6915e6fb399c13dfabc269276"
May 25 12:03:58.308982 ip-10-0-128-146 kubenswrapper[2560]: I0525 12:03:58.308916    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba49e1d05af71721e652ba08b91cd8ee8e9fbcb6915e6fb399c13dfabc269276"} err="failed to get container status \"ba49e1d05af71721e652ba08b91cd8ee8e9fbcb6915e6fb399c13dfabc269276\": rpc error: code = NotFound desc = could not find container \"ba49e1d05af71721e652ba08b91cd8ee8e9fbcb6915e6fb399c13dfabc269276\": container with ID starting with ba49e1d05af71721e652ba08b91cd8ee8e9fbcb6915e6fb399c13dfabc269276 not found: ID does not exist"
May 25 12:03:58.308982 ip-10-0-128-146 kubenswrapper[2560]: I0525 12:03:58.308935    2560 scope.go:117] "RemoveContainer" containerID="f15d1008119bd1a9197bbe200be2fa9e9f7b15f88a7247b66cdcb3ef75c48ad9"
May 25 12:03:58.309186 ip-10-0-128-146 kubenswrapper[2560]: E0525 12:03:58.309168    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f15d1008119bd1a9197bbe200be2fa9e9f7b15f88a7247b66cdcb3ef75c48ad9\": container with ID starting with f15d1008119bd1a9197bbe200be2fa9e9f7b15f88a7247b66cdcb3ef75c48ad9 not found: ID does not exist" containerID="f15d1008119bd1a9197bbe200be2fa9e9f7b15f88a7247b66cdcb3ef75c48ad9"
May 25 12:03:58.309270 ip-10-0-128-146 kubenswrapper[2560]: I0525 12:03:58.309191    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f15d1008119bd1a9197bbe200be2fa9e9f7b15f88a7247b66cdcb3ef75c48ad9"} err="failed to get container status \"f15d1008119bd1a9197bbe200be2fa9e9f7b15f88a7247b66cdcb3ef75c48ad9\": rpc error: code = NotFound desc = could not find container \"f15d1008119bd1a9197bbe200be2fa9e9f7b15f88a7247b66cdcb3ef75c48ad9\": container with ID starting with f15d1008119bd1a9197bbe200be2fa9e9f7b15f88a7247b66cdcb3ef75c48ad9 not found: ID does not exist"
May 25 12:03:58.310304 ip-10-0-128-146 kubenswrapper[2560]: I0525 12:03:58.310285    2560 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-c1cab-predictor-6c58cc8979-mf6jk"]
May 25 12:03:58.313295 ip-10-0-128-146 kubenswrapper[2560]: I0525 12:03:58.313277    2560 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-c1cab-predictor-6c58cc8979-mf6jk"]
May 25 12:03:58.934608 ip-10-0-128-146 kubenswrapper[2560]: I0525 12:03:58.934579    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3ee3c379-e619-4591-bd6a-62de2dc77a02" path="/var/lib/kubelet/pods/3ee3c379-e619-4591-bd6a-62de2dc77a02/volumes"
May 25 12:04:25.587447 ip-10-0-128-146 kubenswrapper[2560]: I0525 12:04:25.587418    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-nq67w_c0068087-c30b-4ff5-a794-b84738aaa168/global-pull-secret-syncer/0.log"
May 25 12:04:25.695197 ip-10-0-128-146 kubenswrapper[2560]: I0525 12:04:25.695163    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-fbljm_5d252e49-a8a8-4794-befd-50a55e95b17b/konnectivity-agent/0.log"
May 25 12:04:25.758127 ip-10-0-128-146 kubenswrapper[2560]: I0525 12:04:25.758098    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-128-146.ec2.internal_205c636549bb34e2af97bc6edc40842f/haproxy/0.log"
May 25 12:04:29.437332 ip-10-0-128-146 kubenswrapper[2560]: I0525 12:04:29.437301    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6c75959455-n8lh8_251cca37-3cc4-4187-895f-e273d9eac261/cluster-monitoring-operator/0.log"
May 25 12:04:29.741541 ip-10-0-128-146 kubenswrapper[2560]: I0525 12:04:29.741463    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rk5fb_6089fc6e-f837-4d7f-ae29-cc5588936435/node-exporter/0.log"
May 25 12:04:29.762114 ip-10-0-128-146 kubenswrapper[2560]: I0525 12:04:29.762090    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rk5fb_6089fc6e-f837-4d7f-ae29-cc5588936435/kube-rbac-proxy/0.log"
May 25 12:04:29.783009 ip-10-0-128-146 kubenswrapper[2560]: I0525 12:04:29.782984    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rk5fb_6089fc6e-f837-4d7f-ae29-cc5588936435/init-textfile/0.log"
May 25 12:04:29.809122 ip-10-0-128-146 kubenswrapper[2560]: I0525 12:04:29.809095    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-9c959559f-x8bkd_1b402763-7a26-428e-8165-0248e508c1f5/kube-rbac-proxy-main/0.log"
May 25 12:04:29.832649 ip-10-0-128-146 kubenswrapper[2560]: I0525 12:04:29.832627    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-9c959559f-x8bkd_1b402763-7a26-428e-8165-0248e508c1f5/kube-rbac-proxy-self/0.log"
May 25 12:04:29.855354 ip-10-0-128-146 kubenswrapper[2560]: I0525 12:04:29.855330    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-9c959559f-x8bkd_1b402763-7a26-428e-8165-0248e508c1f5/openshift-state-metrics/0.log"
May 25 12:04:30.114906 ip-10-0-128-146 kubenswrapper[2560]: I0525 12:04:30.114880    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-68b857dfd5-xdvx6_3c3fed2e-c3d2-4e64-94a5-07eed2348adc/telemeter-client/0.log"
May 25 12:04:30.135081 ip-10-0-128-146 kubenswrapper[2560]: I0525 12:04:30.135055    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-68b857dfd5-xdvx6_3c3fed2e-c3d2-4e64-94a5-07eed2348adc/reload/0.log"
May 25 12:04:30.158254 ip-10-0-128-146 kubenswrapper[2560]: I0525 12:04:30.158236    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-68b857dfd5-xdvx6_3c3fed2e-c3d2-4e64-94a5-07eed2348adc/kube-rbac-proxy/0.log"
May 25 12:04:31.849152 ip-10-0-128-146 kubenswrapper[2560]: I0525 12:04:31.849098    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-zd54m_2be32661-9288-4902-9d67-1a5e8f74670d/console-operator/1.log"
May 25 12:04:31.853295 ip-10-0-128-146 kubenswrapper[2560]: I0525 12:04:31.853278    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-zd54m_2be32661-9288-4902-9d67-1a5e8f74670d/console-operator/2.log"
May 25 12:04:32.235047 ip-10-0-128-146 kubenswrapper[2560]: I0525 12:04:32.234978    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-7c4bbdcc86-bnkkf_af689d73-9b4a-4020-b811-b5930019e908/console/0.log"
May 25 12:04:32.521638 ip-10-0-128-146 kubenswrapper[2560]: I0525 12:04:32.521566    2560 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-must-gather-fzvv7/perf-node-gather-daemonset-j5hpg"]
May 25 12:04:32.521905 ip-10-0-128-146 kubenswrapper[2560]: I0525 12:04:32.521890    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f1814527-8ba1-4118-ad92-4454338b4116" containerName="kube-rbac-proxy"
May 25 12:04:32.521953 ip-10-0-128-146 kubenswrapper[2560]: I0525 12:04:32.521907    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="f1814527-8ba1-4118-ad92-4454338b4116" containerName="kube-rbac-proxy"
May 25 12:04:32.521953 ip-10-0-128-146 kubenswrapper[2560]: I0525 12:04:32.521919    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ee3c379-e619-4591-bd6a-62de2dc77a02" containerName="kserve-container"
May 25 12:04:32.521953 ip-10-0-128-146 kubenswrapper[2560]: I0525 12:04:32.521924    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ee3c379-e619-4591-bd6a-62de2dc77a02" containerName="kserve-container"
May 25 12:04:32.521953 ip-10-0-128-146 kubenswrapper[2560]: I0525 12:04:32.521939    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e1ac9df7-55b7-4b46-a326-fd3d3d803fda" containerName="kserve-container"
May 25 12:04:32.521953 ip-10-0-128-146 kubenswrapper[2560]: I0525 12:04:32.521945    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="e1ac9df7-55b7-4b46-a326-fd3d3d803fda" containerName="kserve-container"
May 25 12:04:32.522113 ip-10-0-128-146 kubenswrapper[2560]: I0525 12:04:32.521955    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f1814527-8ba1-4118-ad92-4454338b4116" containerName="kserve-container"
May 25 12:04:32.522113 ip-10-0-128-146 kubenswrapper[2560]: I0525 12:04:32.521961    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="f1814527-8ba1-4118-ad92-4454338b4116" containerName="kserve-container"
May 25 12:04:32.522113 ip-10-0-128-146 kubenswrapper[2560]: I0525 12:04:32.521969    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ee3c379-e619-4591-bd6a-62de2dc77a02" containerName="kube-rbac-proxy"
May 25 12:04:32.522113 ip-10-0-128-146 kubenswrapper[2560]: I0525 12:04:32.521978    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ee3c379-e619-4591-bd6a-62de2dc77a02" containerName="kube-rbac-proxy"
May 25 12:04:32.522113 ip-10-0-128-146 kubenswrapper[2560]: I0525 12:04:32.521984    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e1ac9df7-55b7-4b46-a326-fd3d3d803fda" containerName="kube-rbac-proxy"
May 25 12:04:32.522113 ip-10-0-128-146 kubenswrapper[2560]: I0525 12:04:32.521989    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="e1ac9df7-55b7-4b46-a326-fd3d3d803fda" containerName="kube-rbac-proxy"
May 25 12:04:32.522113 ip-10-0-128-146 kubenswrapper[2560]: I0525 12:04:32.522045    2560 memory_manager.go:356] "RemoveStaleState removing state" podUID="3ee3c379-e619-4591-bd6a-62de2dc77a02" containerName="kserve-container"
May 25 12:04:32.522113 ip-10-0-128-146 kubenswrapper[2560]: I0525 12:04:32.522056    2560 memory_manager.go:356] "RemoveStaleState removing state" podUID="e1ac9df7-55b7-4b46-a326-fd3d3d803fda" containerName="kserve-container"
May 25 12:04:32.522113 ip-10-0-128-146 kubenswrapper[2560]: I0525 12:04:32.522064    2560 memory_manager.go:356] "RemoveStaleState removing state" podUID="f1814527-8ba1-4118-ad92-4454338b4116" containerName="kube-rbac-proxy"
May 25 12:04:32.522113 ip-10-0-128-146 kubenswrapper[2560]: I0525 12:04:32.522071    2560 memory_manager.go:356] "RemoveStaleState removing state" podUID="3ee3c379-e619-4591-bd6a-62de2dc77a02" containerName="kube-rbac-proxy"
May 25 12:04:32.522113 ip-10-0-128-146 kubenswrapper[2560]: I0525 12:04:32.522079    2560 memory_manager.go:356] "RemoveStaleState removing state" podUID="f1814527-8ba1-4118-ad92-4454338b4116" containerName="kserve-container"
May 25 12:04:32.522113 ip-10-0-128-146 kubenswrapper[2560]: I0525 12:04:32.522085    2560 memory_manager.go:356] "RemoveStaleState removing state" podUID="e1ac9df7-55b7-4b46-a326-fd3d3d803fda" containerName="kube-rbac-proxy"
May 25 12:04:32.525005 ip-10-0-128-146 kubenswrapper[2560]: I0525 12:04:32.524989    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-fzvv7/perf-node-gather-daemonset-j5hpg"
May 25 12:04:32.527691 ip-10-0-128-146 kubenswrapper[2560]: I0525 12:04:32.527659    2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-fzvv7\"/\"default-dockercfg-mcf27\""
May 25 12:04:32.527691 ip-10-0-128-146 kubenswrapper[2560]: I0525 12:04:32.527676    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-fzvv7\"/\"kube-root-ca.crt\""
May 25 12:04:32.527691 ip-10-0-128-146 kubenswrapper[2560]: I0525 12:04:32.527683    2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-fzvv7\"/\"openshift-service-ca.crt\""
May 25 12:04:32.531472 ip-10-0-128-146 kubenswrapper[2560]: I0525 12:04:32.531452    2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-fzvv7/perf-node-gather-daemonset-j5hpg"]
May 25 12:04:32.625802 ip-10-0-128-146 kubenswrapper[2560]: I0525 12:04:32.625772    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f8db9b0e-d512-4ea6-aa61-4ed50812aa27-sys\") pod \"perf-node-gather-daemonset-j5hpg\" (UID: \"f8db9b0e-d512-4ea6-aa61-4ed50812aa27\") " pod="openshift-must-gather-fzvv7/perf-node-gather-daemonset-j5hpg"
May 25 12:04:32.625940 ip-10-0-128-146 kubenswrapper[2560]: I0525 12:04:32.625808    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f8db9b0e-d512-4ea6-aa61-4ed50812aa27-podres\") pod \"perf-node-gather-daemonset-j5hpg\" (UID: \"f8db9b0e-d512-4ea6-aa61-4ed50812aa27\") " pod="openshift-must-gather-fzvv7/perf-node-gather-daemonset-j5hpg"
May 25 12:04:32.625940 ip-10-0-128-146 kubenswrapper[2560]: I0525 12:04:32.625847    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4sk7\" (UniqueName: \"kubernetes.io/projected/f8db9b0e-d512-4ea6-aa61-4ed50812aa27-kube-api-access-g4sk7\") pod \"perf-node-gather-daemonset-j5hpg\" (UID: \"f8db9b0e-d512-4ea6-aa61-4ed50812aa27\") " pod="openshift-must-gather-fzvv7/perf-node-gather-daemonset-j5hpg"
May 25 12:04:32.625940 ip-10-0-128-146 kubenswrapper[2560]: I0525 12:04:32.625888    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f8db9b0e-d512-4ea6-aa61-4ed50812aa27-lib-modules\") pod \"perf-node-gather-daemonset-j5hpg\" (UID: \"f8db9b0e-d512-4ea6-aa61-4ed50812aa27\") " pod="openshift-must-gather-fzvv7/perf-node-gather-daemonset-j5hpg"
May 25 12:04:32.626048 ip-10-0-128-146 kubenswrapper[2560]: I0525 12:04:32.625956    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f8db9b0e-d512-4ea6-aa61-4ed50812aa27-proc\") pod \"perf-node-gather-daemonset-j5hpg\" (UID: \"f8db9b0e-d512-4ea6-aa61-4ed50812aa27\") " pod="openshift-must-gather-fzvv7/perf-node-gather-daemonset-j5hpg"
May 25 12:04:32.701825 ip-10-0-128-146 kubenswrapper[2560]: I0525 12:04:32.701802    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-5cf8c89d64-l2xsm_f39a5f22-307a-41f0-9ac8-2a37349a345d/volume-data-source-validator/0.log"
May 25 12:04:32.726747 ip-10-0-128-146 kubenswrapper[2560]: I0525 12:04:32.726722    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f8db9b0e-d512-4ea6-aa61-4ed50812aa27-proc\") pod \"perf-node-gather-daemonset-j5hpg\" (UID: \"f8db9b0e-d512-4ea6-aa61-4ed50812aa27\") " pod="openshift-must-gather-fzvv7/perf-node-gather-daemonset-j5hpg"
May 25 12:04:32.726855 ip-10-0-128-146 kubenswrapper[2560]: I0525 12:04:32.726752    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f8db9b0e-d512-4ea6-aa61-4ed50812aa27-sys\") pod \"perf-node-gather-daemonset-j5hpg\" (UID: \"f8db9b0e-d512-4ea6-aa61-4ed50812aa27\") " pod="openshift-must-gather-fzvv7/perf-node-gather-daemonset-j5hpg"
May 25 12:04:32.726855 ip-10-0-128-146 kubenswrapper[2560]: I0525 12:04:32.726775    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f8db9b0e-d512-4ea6-aa61-4ed50812aa27-podres\") pod \"perf-node-gather-daemonset-j5hpg\" (UID: \"f8db9b0e-d512-4ea6-aa61-4ed50812aa27\") " pod="openshift-must-gather-fzvv7/perf-node-gather-daemonset-j5hpg"
May 25 12:04:32.726855 ip-10-0-128-146 kubenswrapper[2560]: I0525 12:04:32.726820    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-g4sk7\" (UniqueName: \"kubernetes.io/projected/f8db9b0e-d512-4ea6-aa61-4ed50812aa27-kube-api-access-g4sk7\") pod \"perf-node-gather-daemonset-j5hpg\" (UID: \"f8db9b0e-d512-4ea6-aa61-4ed50812aa27\") " pod="openshift-must-gather-fzvv7/perf-node-gather-daemonset-j5hpg"
May 25 12:04:32.726855 ip-10-0-128-146 kubenswrapper[2560]: I0525 12:04:32.726832    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f8db9b0e-d512-4ea6-aa61-4ed50812aa27-proc\") pod \"perf-node-gather-daemonset-j5hpg\" (UID: \"f8db9b0e-d512-4ea6-aa61-4ed50812aa27\") " pod="openshift-must-gather-fzvv7/perf-node-gather-daemonset-j5hpg"
May 25 12:04:32.726855 ip-10-0-128-146 kubenswrapper[2560]: I0525 12:04:32.726843    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f8db9b0e-d512-4ea6-aa61-4ed50812aa27-sys\") pod \"perf-node-gather-daemonset-j5hpg\" (UID: \"f8db9b0e-d512-4ea6-aa61-4ed50812aa27\") " pod="openshift-must-gather-fzvv7/perf-node-gather-daemonset-j5hpg"
May 25 12:04:32.726855 ip-10-0-128-146 kubenswrapper[2560]: I0525 12:04:32.726849    2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f8db9b0e-d512-4ea6-aa61-4ed50812aa27-lib-modules\") pod \"perf-node-gather-daemonset-j5hpg\" (UID: \"f8db9b0e-d512-4ea6-aa61-4ed50812aa27\") " pod="openshift-must-gather-fzvv7/perf-node-gather-daemonset-j5hpg"
May 25 12:04:32.727045 ip-10-0-128-146 kubenswrapper[2560]: I0525 12:04:32.726946    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f8db9b0e-d512-4ea6-aa61-4ed50812aa27-podres\") pod \"perf-node-gather-daemonset-j5hpg\" (UID: \"f8db9b0e-d512-4ea6-aa61-4ed50812aa27\") " pod="openshift-must-gather-fzvv7/perf-node-gather-daemonset-j5hpg"
May 25 12:04:32.727045 ip-10-0-128-146 kubenswrapper[2560]: I0525 12:04:32.726949    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f8db9b0e-d512-4ea6-aa61-4ed50812aa27-lib-modules\") pod \"perf-node-gather-daemonset-j5hpg\" (UID: \"f8db9b0e-d512-4ea6-aa61-4ed50812aa27\") " pod="openshift-must-gather-fzvv7/perf-node-gather-daemonset-j5hpg"
May 25 12:04:32.735702 ip-10-0-128-146 kubenswrapper[2560]: I0525 12:04:32.735675    2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4sk7\" (UniqueName: \"kubernetes.io/projected/f8db9b0e-d512-4ea6-aa61-4ed50812aa27-kube-api-access-g4sk7\") pod \"perf-node-gather-daemonset-j5hpg\" (UID: \"f8db9b0e-d512-4ea6-aa61-4ed50812aa27\") " pod="openshift-must-gather-fzvv7/perf-node-gather-daemonset-j5hpg"
May 25 12:04:32.836195 ip-10-0-128-146 kubenswrapper[2560]: I0525 12:04:32.836166    2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-fzvv7/perf-node-gather-daemonset-j5hpg"
May 25 12:04:32.957708 ip-10-0-128-146 kubenswrapper[2560]: I0525 12:04:32.957675    2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-fzvv7/perf-node-gather-daemonset-j5hpg"]
May 25 12:04:32.961233 ip-10-0-128-146 kubenswrapper[2560]: W0525 12:04:32.961192    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podf8db9b0e_d512_4ea6_aa61_4ed50812aa27.slice/crio-9f359ba18a70347f254c5f6e722f1132463618a19c6a90fa2676cbaeb6291ead WatchSource:0}: Error finding container 9f359ba18a70347f254c5f6e722f1132463618a19c6a90fa2676cbaeb6291ead: Status 404 returned error can't find the container with id 9f359ba18a70347f254c5f6e722f1132463618a19c6a90fa2676cbaeb6291ead
May 25 12:04:32.962772 ip-10-0-128-146 kubenswrapper[2560]: I0525 12:04:32.962756    2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 25 12:04:33.406391 ip-10-0-128-146 kubenswrapper[2560]: I0525 12:04:33.406359    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fzvv7/perf-node-gather-daemonset-j5hpg" event={"ID":"f8db9b0e-d512-4ea6-aa61-4ed50812aa27","Type":"ContainerStarted","Data":"e7ceaa7563e743a156703f067f711f07bde17a6dcd5c040f55506baff7ba93b9"}
May 25 12:04:33.406391 ip-10-0-128-146 kubenswrapper[2560]: I0525 12:04:33.406394    2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fzvv7/perf-node-gather-daemonset-j5hpg" event={"ID":"f8db9b0e-d512-4ea6-aa61-4ed50812aa27","Type":"ContainerStarted","Data":"9f359ba18a70347f254c5f6e722f1132463618a19c6a90fa2676cbaeb6291ead"}
May 25 12:04:33.406657 ip-10-0-128-146 kubenswrapper[2560]: I0525 12:04:33.406443    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-fzvv7/perf-node-gather-daemonset-j5hpg"
May 25 12:04:33.424231 ip-10-0-128-146 kubenswrapper[2560]: I0525 12:04:33.424141    2560 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-must-gather-fzvv7/perf-node-gather-daemonset-j5hpg" podStartSLOduration=1.424127718 podStartE2EDuration="1.424127718s" podCreationTimestamp="2026-05-25 12:04:32 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 12:04:33.422289577 +0000 UTC m=+2815.103267565" watchObservedRunningTime="2026-05-25 12:04:33.424127718 +0000 UTC m=+2815.105105725"
May 25 12:04:33.492835 ip-10-0-128-146 kubenswrapper[2560]: I0525 12:04:33.492809    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-vvfw5_69432372-dd12-48f9-9cf4-8b194109b426/dns/0.log"
May 25 12:04:33.513362 ip-10-0-128-146 kubenswrapper[2560]: I0525 12:04:33.513339    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-vvfw5_69432372-dd12-48f9-9cf4-8b194109b426/kube-rbac-proxy/0.log"
May 25 12:04:33.588114 ip-10-0-128-146 kubenswrapper[2560]: I0525 12:04:33.588079    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-kbgpq_dc7c628a-0fd2-456b-a47f-b7741b83d46e/dns-node-resolver/0.log"
May 25 12:04:34.045408 ip-10-0-128-146 kubenswrapper[2560]: I0525 12:04:34.045374    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-8vzrv_2d4d0de1-9c41-46ef-89e3-9a0615a665af/node-ca/0.log"
May 25 12:04:34.875844 ip-10-0-128-146 kubenswrapper[2560]: I0525 12:04:34.875814    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-65cf6cdd84-r95n8_6854cf61-d901-4c43-8718-34de98e26880/router/0.log"
May 25 12:04:35.218602 ip-10-0-128-146 kubenswrapper[2560]: I0525 12:04:35.218531    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-2mqc7_ed3452ef-2260-4664-92ab-549403628cc8/serve-healthcheck-canary/0.log"
May 25 12:04:35.625495 ip-10-0-128-146 kubenswrapper[2560]: I0525 12:04:35.625465    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-78bdfb888c-rqcll_3ca8ede5-d33b-4a16-bd65-73ec9f2a2735/insights-operator/0.log"
May 25 12:04:35.625990 ip-10-0-128-146 kubenswrapper[2560]: I0525 12:04:35.625973    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-78bdfb888c-rqcll_3ca8ede5-d33b-4a16-bd65-73ec9f2a2735/insights-operator/1.log"
May 25 12:04:35.725599 ip-10-0-128-146 kubenswrapper[2560]: I0525 12:04:35.725572    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-7dgx7_64750c18-e28d-484d-891f-44b12392fe2c/kube-rbac-proxy/0.log"
May 25 12:04:35.746968 ip-10-0-128-146 kubenswrapper[2560]: I0525 12:04:35.746944    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-7dgx7_64750c18-e28d-484d-891f-44b12392fe2c/exporter/0.log"
May 25 12:04:35.769030 ip-10-0-128-146 kubenswrapper[2560]: I0525 12:04:35.768962    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-7dgx7_64750c18-e28d-484d-891f-44b12392fe2c/extractor/0.log"
May 25 12:04:37.797071 ip-10-0-128-146 kubenswrapper[2560]: I0525 12:04:37.797040    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_kserve-controller-manager-559cc6dd6f-qxcs2_f2b8ad4e-8b6d-494d-88f6-f1e516d7f58b/manager/0.log"
May 25 12:04:37.819222 ip-10-0-128-146 kubenswrapper[2560]: I0525 12:04:37.819185    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_llmisvc-controller-manager-6b94ff949c-hrsrt_6539f1f2-5123-4750-ba1b-3e6159bdb1a6/manager/0.log"
May 25 12:04:37.844597 ip-10-0-128-146 kubenswrapper[2560]: I0525 12:04:37.844576    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-86f7b4b499-68fvm_555cfdd5-04ae-4933-8b5b-24597359444b/server/0.log"
May 25 12:04:39.418787 ip-10-0-128-146 kubenswrapper[2560]: I0525 12:04:39.418760    2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-fzvv7/perf-node-gather-daemonset-j5hpg"
May 25 12:04:42.773327 ip-10-0-128-146 kubenswrapper[2560]: I0525 12:04:42.773294    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-565644f49-fcqbz_96664b24-d104-4b57-83e8-63b8c9b3212b/kube-storage-version-migrator-operator/1.log"
May 25 12:04:42.774013 ip-10-0-128-146 kubenswrapper[2560]: I0525 12:04:42.773997    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-565644f49-fcqbz_96664b24-d104-4b57-83e8-63b8c9b3212b/kube-storage-version-migrator-operator/0.log"
May 25 12:04:43.714656 ip-10-0-128-146 kubenswrapper[2560]: I0525 12:04:43.714613    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-5qqv9_9bfeee33-bbdf-4407-9f32-5dfe1a29d075/kube-multus/0.log"
May 25 12:04:44.088275 ip-10-0-128-146 kubenswrapper[2560]: I0525 12:04:44.088247    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vdfgw_372c0836-0d39-4561-b0fa-7171eecad7ca/kube-multus-additional-cni-plugins/0.log"
May 25 12:04:44.116163 ip-10-0-128-146 kubenswrapper[2560]: I0525 12:04:44.116139    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vdfgw_372c0836-0d39-4561-b0fa-7171eecad7ca/egress-router-binary-copy/0.log"
May 25 12:04:44.139178 ip-10-0-128-146 kubenswrapper[2560]: I0525 12:04:44.139092    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vdfgw_372c0836-0d39-4561-b0fa-7171eecad7ca/cni-plugins/0.log"
May 25 12:04:44.164985 ip-10-0-128-146 kubenswrapper[2560]: I0525 12:04:44.164960    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vdfgw_372c0836-0d39-4561-b0fa-7171eecad7ca/bond-cni-plugin/0.log"
May 25 12:04:44.187787 ip-10-0-128-146 kubenswrapper[2560]: I0525 12:04:44.187760    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vdfgw_372c0836-0d39-4561-b0fa-7171eecad7ca/routeoverride-cni/0.log"
May 25 12:04:44.218263 ip-10-0-128-146 kubenswrapper[2560]: I0525 12:04:44.218244    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vdfgw_372c0836-0d39-4561-b0fa-7171eecad7ca/whereabouts-cni-bincopy/0.log"
May 25 12:04:44.250705 ip-10-0-128-146 kubenswrapper[2560]: I0525 12:04:44.250681    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vdfgw_372c0836-0d39-4561-b0fa-7171eecad7ca/whereabouts-cni/0.log"
May 25 12:04:44.427588 ip-10-0-128-146 kubenswrapper[2560]: I0525 12:04:44.427513    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-hxdhk_b0dc07c1-2dce-4b08-9fee-41a5937a7bae/network-metrics-daemon/0.log"
May 25 12:04:44.456384 ip-10-0-128-146 kubenswrapper[2560]: I0525 12:04:44.456360    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-hxdhk_b0dc07c1-2dce-4b08-9fee-41a5937a7bae/kube-rbac-proxy/0.log"
May 25 12:04:45.491863 ip-10-0-128-146 kubenswrapper[2560]: I0525 12:04:45.491823    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m55v2_b93f3e9c-f4f9-49dc-bfba-6d0f80c0971d/ovn-controller/0.log"
May 25 12:04:45.512001 ip-10-0-128-146 kubenswrapper[2560]: I0525 12:04:45.511975    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m55v2_b93f3e9c-f4f9-49dc-bfba-6d0f80c0971d/ovn-acl-logging/0.log"
May 25 12:04:45.523866 ip-10-0-128-146 kubenswrapper[2560]: I0525 12:04:45.523844    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m55v2_b93f3e9c-f4f9-49dc-bfba-6d0f80c0971d/ovn-acl-logging/1.log"
May 25 12:04:45.542241 ip-10-0-128-146 kubenswrapper[2560]: I0525 12:04:45.542200    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m55v2_b93f3e9c-f4f9-49dc-bfba-6d0f80c0971d/kube-rbac-proxy-node/0.log"
May 25 12:04:45.566666 ip-10-0-128-146 kubenswrapper[2560]: I0525 12:04:45.566625    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m55v2_b93f3e9c-f4f9-49dc-bfba-6d0f80c0971d/kube-rbac-proxy-ovn-metrics/0.log"
May 25 12:04:45.591080 ip-10-0-128-146 kubenswrapper[2560]: I0525 12:04:45.591064    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m55v2_b93f3e9c-f4f9-49dc-bfba-6d0f80c0971d/northd/0.log"
May 25 12:04:45.614584 ip-10-0-128-146 kubenswrapper[2560]: I0525 12:04:45.614560    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m55v2_b93f3e9c-f4f9-49dc-bfba-6d0f80c0971d/nbdb/0.log"
May 25 12:04:45.637185 ip-10-0-128-146 kubenswrapper[2560]: I0525 12:04:45.637166    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m55v2_b93f3e9c-f4f9-49dc-bfba-6d0f80c0971d/sbdb/0.log"
May 25 12:04:45.765093 ip-10-0-128-146 kubenswrapper[2560]: I0525 12:04:45.765014    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m55v2_b93f3e9c-f4f9-49dc-bfba-6d0f80c0971d/ovnkube-controller/0.log"
May 25 12:04:47.235706 ip-10-0-128-146 kubenswrapper[2560]: I0525 12:04:47.235669    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-zfljp_0d6828f5-3d2a-417d-ab45-6750dd8cdb92/network-check-target-container/0.log"
May 25 12:04:48.204824 ip-10-0-128-146 kubenswrapper[2560]: I0525 12:04:48.204744    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-kj6wk_0a24f6a0-7ac5-4f1e-84f0-301e16006665/iptables-alerter/0.log"
May 25 12:04:49.001937 ip-10-0-128-146 kubenswrapper[2560]: I0525 12:04:49.001905    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-tgqrm_448f7dca-880c-4e81-8adc-d29e391dfefd/tuned/0.log"