May 25 13:01:57.883306 ip-10-0-135-103 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
May 25 13:01:57.883316 ip-10-0-135-103 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
May 25 13:01:57.883323 ip-10-0-135-103 systemd[1]: kubelet.service: Failed with result 'resources'.
May 25 13:01:57.883587 ip-10-0-135-103 systemd[1]: Failed to start Kubernetes Kubelet.
May 25 13:02:08.130852 ip-10-0-135-103 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
May 25 13:02:08.130868 ip-10-0-135-103 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot c1c7694781434ac9b3c874610939189b --
May 25 13:04:59.691668 ip-10-0-135-103 systemd[1]: Starting Kubernetes Kubelet...
May 25 13:05:00.120040 ip-10-0-135-103 kubenswrapper[2580]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
May 25 13:05:00.120040 ip-10-0-135-103 kubenswrapper[2580]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
May 25 13:05:00.120040 ip-10-0-135-103 kubenswrapper[2580]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
May 25 13:05:00.120040 ip-10-0-135-103 kubenswrapper[2580]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
May 25 13:05:00.120040 ip-10-0-135-103 kubenswrapper[2580]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
May 25 13:05:00.121291 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.121202    2580 server.go:212] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
May 25 13:05:00.125006 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.124984    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 25 13:05:00.125006 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.125002    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 25 13:05:00.125006 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.125008    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 25 13:05:00.125006 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.125011    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 25 13:05:00.125264 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.125018    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 25 13:05:00.125264 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.125023    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 25 13:05:00.125264 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.125026    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 25 13:05:00.125264 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.125030    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 25 13:05:00.125264 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.125034    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 25 13:05:00.125264 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.125037    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 25 13:05:00.125264 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.125041    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 25 13:05:00.125264 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.125044    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 25 13:05:00.125264 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.125048    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 25 13:05:00.125264 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.125052    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 25 13:05:00.125264 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.125056    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 25 13:05:00.125264 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.125060    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 25 13:05:00.125264 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.125063    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 25 13:05:00.125264 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.125067    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 25 13:05:00.125264 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.125073    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 25 13:05:00.125264 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.125078    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 25 13:05:00.125264 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.125082    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 25 13:05:00.125264 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.125086    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
May 25 13:05:00.125264 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.125091    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 25 13:05:00.126070 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.125095    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 25 13:05:00.126070 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.125099    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
May 25 13:05:00.126070 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.125111    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 25 13:05:00.126070 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.125116    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 25 13:05:00.126070 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.125120    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 25 13:05:00.126070 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.125123    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 25 13:05:00.126070 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.125127    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 25 13:05:00.126070 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.125134    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 25 13:05:00.126070 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.125138    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 25 13:05:00.126070 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.125142    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 25 13:05:00.126070 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.125146    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 25 13:05:00.126070 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.125151    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 25 13:05:00.126070 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.125156    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 25 13:05:00.126070 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.125160    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 25 13:05:00.126070 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.125165    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 25 13:05:00.126070 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.125169    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 25 13:05:00.126070 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.125173    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 25 13:05:00.126070 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.125177    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 25 13:05:00.126070 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.125181    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 25 13:05:00.126070 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.125185    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 25 13:05:00.126821 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.125189    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 25 13:05:00.126821 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.125193    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 25 13:05:00.126821 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.125197    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 25 13:05:00.126821 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.125201    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 25 13:05:00.126821 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.125206    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 25 13:05:00.126821 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.125210    2580 feature_gate.go:328] unrecognized feature gate: Example
May 25 13:05:00.126821 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.125215    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 25 13:05:00.126821 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.125218    2580 feature_gate.go:328] unrecognized feature gate: Example2
May 25 13:05:00.126821 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.125223    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 25 13:05:00.126821 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.125227    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 25 13:05:00.126821 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.125231    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 25 13:05:00.126821 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.125235    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 25 13:05:00.126821 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.125239    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 25 13:05:00.126821 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.125243    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 25 13:05:00.126821 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.125248    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 25 13:05:00.126821 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.125252    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 25 13:05:00.126821 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.125256    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 25 13:05:00.126821 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.125260    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 25 13:05:00.126821 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.125263    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 25 13:05:00.126821 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.125267    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 25 13:05:00.127311 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.125271    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 25 13:05:00.127311 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.125274    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 25 13:05:00.127311 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.125278    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 25 13:05:00.127311 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.125282    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 25 13:05:00.127311 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.125286    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 25 13:05:00.127311 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.125289    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 25 13:05:00.127311 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.125295    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 25 13:05:00.127311 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.125299    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 25 13:05:00.127311 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.125303    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 25 13:05:00.127311 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.125309    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 25 13:05:00.127311 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.125314    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 25 13:05:00.127311 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.125318    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 25 13:05:00.127311 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.125325    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 25 13:05:00.127311 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.125329    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 25 13:05:00.127311 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.125333    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 25 13:05:00.127311 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.125338    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 25 13:05:00.127311 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.125346    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 25 13:05:00.127311 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.125351    2580 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 25 13:05:00.127311 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.125355    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 25 13:05:00.127311 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.125359    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 25 13:05:00.127828 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.125382    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 25 13:05:00.127828 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.125387    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 25 13:05:00.127828 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.125391    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 25 13:05:00.127828 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.126021    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 25 13:05:00.127828 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.126029    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 25 13:05:00.127828 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.126034    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 25 13:05:00.127828 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.126040    2580 feature_gate.go:328] unrecognized feature gate: Example2
May 25 13:05:00.127828 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.126044    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 25 13:05:00.127828 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.126049    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 25 13:05:00.127828 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.126053    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 25 13:05:00.127828 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.126057    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 25 13:05:00.127828 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.126062    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 25 13:05:00.127828 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.126066    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 25 13:05:00.127828 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.126069    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 25 13:05:00.127828 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.126073    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 25 13:05:00.127828 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.126077    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 25 13:05:00.127828 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.126080    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 25 13:05:00.127828 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.126084    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 25 13:05:00.127828 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.126089    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 25 13:05:00.127828 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.126093    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 25 13:05:00.128607 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.126098    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 25 13:05:00.128607 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.126103    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 25 13:05:00.128607 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.126107    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 25 13:05:00.128607 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.126112    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 25 13:05:00.128607 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.126116    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 25 13:05:00.128607 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.126120    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 25 13:05:00.128607 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.126126    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 25 13:05:00.128607 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.126132    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 25 13:05:00.128607 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.126136    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 25 13:05:00.128607 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.126140    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 25 13:05:00.128607 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.126144    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 25 13:05:00.128607 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.126148    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 25 13:05:00.128607 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.126152    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 25 13:05:00.128607 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.126156    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 25 13:05:00.128607 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.126160    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 25 13:05:00.128607 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.126164    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 25 13:05:00.128607 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.126168    2580 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 25 13:05:00.128607 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.126172    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 25 13:05:00.128607 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.126177    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 25 13:05:00.129499 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.126181    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 25 13:05:00.129499 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.126186    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 25 13:05:00.129499 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.126190    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 25 13:05:00.129499 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.126195    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 25 13:05:00.129499 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.126199    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 25 13:05:00.129499 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.126203    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 25 13:05:00.129499 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.126208    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 25 13:05:00.129499 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.126211    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 25 13:05:00.129499 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.126216    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 25 13:05:00.129499 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.126220    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 25 13:05:00.129499 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.126224    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 25 13:05:00.129499 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.126229    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 25 13:05:00.129499 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.126233    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 25 13:05:00.129499 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.126238    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 25 13:05:00.129499 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.126242    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 25 13:05:00.129499 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.126247    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 25 13:05:00.129499 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.126251    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 25 13:05:00.129499 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.126256    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
May 25 13:05:00.129499 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.126259    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 25 13:05:00.129499 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.126264    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 25 13:05:00.129993 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.126268    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 25 13:05:00.129993 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.126274    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 25 13:05:00.129993 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.126278    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 25 13:05:00.129993 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.126282    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 25 13:05:00.129993 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.126286    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 25 13:05:00.129993 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.126290    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 25 13:05:00.129993 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.126294    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 25 13:05:00.129993 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.126298    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 25 13:05:00.129993 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.126304    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 25 13:05:00.129993 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.126310    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 25 13:05:00.129993 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.126315    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 25 13:05:00.129993 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.126320    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 25 13:05:00.129993 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.126324    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 25 13:05:00.129993 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.126329    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 25 13:05:00.129993 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.126333    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 25 13:05:00.129993 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.126338    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 25 13:05:00.129993 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.126342    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 25 13:05:00.129993 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.126348    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 25 13:05:00.129993 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.126352    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 25 13:05:00.130632 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.126356    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 25 13:05:00.130632 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.126360    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 25 13:05:00.130632 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.126384    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 25 13:05:00.130632 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.126389    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 25 13:05:00.130632 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.126394    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 25 13:05:00.130632 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.126398    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 25 13:05:00.130632 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.126402    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 25 13:05:00.130632 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.126407    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 25 13:05:00.130632 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.126410    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 25 13:05:00.130632 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.126416    2580 feature_gate.go:328] unrecognized feature gate: Example
May 25 13:05:00.130632 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.126421    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
May 25 13:05:00.130632 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.127947    2580 flags.go:64] FLAG: --address="0.0.0.0"
May 25 13:05:00.130632 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.127965    2580 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
May 25 13:05:00.130632 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.127977    2580 flags.go:64] FLAG: --anonymous-auth="true"
May 25 13:05:00.130632 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.127984    2580 flags.go:64] FLAG: --application-metrics-count-limit="100"
May 25 13:05:00.130632 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.127991    2580 flags.go:64] FLAG: --authentication-token-webhook="false"
May 25 13:05:00.130632 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.127997    2580 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
May 25 13:05:00.130632 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.128004    2580 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
May 25 13:05:00.130632 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.128017    2580 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
May 25 13:05:00.130632 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.128022    2580 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
May 25 13:05:00.130632 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.128027    2580 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
May 25 13:05:00.131354 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.128033    2580 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
May 25 13:05:00.131354 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.128039    2580 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
May 25 13:05:00.131354 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.128044    2580 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
May 25 13:05:00.131354 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.128049    2580 flags.go:64] FLAG: --cgroup-root=""
May 25 13:05:00.131354 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.128054    2580 flags.go:64] FLAG: --cgroups-per-qos="true"
May 25 13:05:00.131354 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.128060    2580 flags.go:64] FLAG: --client-ca-file=""
May 25 13:05:00.131354 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.128065    2580 flags.go:64] FLAG: --cloud-config=""
May 25 13:05:00.131354 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.128069    2580 flags.go:64] FLAG: --cloud-provider="external"
May 25 13:05:00.131354 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.128074    2580 flags.go:64] FLAG: --cluster-dns="[]"
May 25 13:05:00.131354 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.128080    2580 flags.go:64] FLAG: --cluster-domain=""
May 25 13:05:00.131354 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.128085    2580 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
May 25 13:05:00.131354 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.128090    2580 flags.go:64] FLAG: --config-dir=""
May 25 13:05:00.131354 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.128095    2580 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
May 25 13:05:00.131354 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.128101    2580 flags.go:64] FLAG: --container-log-max-files="5"
May 25 13:05:00.131354 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.128107    2580 flags.go:64] FLAG: --container-log-max-size="10Mi"
May 25 13:05:00.131354 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.128112    2580 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
May 25 13:05:00.131354 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.128118    2580 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
May 25 13:05:00.131354 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.128123    2580 flags.go:64] FLAG: --containerd-namespace="k8s.io"
May 25 13:05:00.131354 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.128128    2580 flags.go:64] FLAG: --contention-profiling="false"
May 25 13:05:00.131354 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.128133    2580 flags.go:64] FLAG: --cpu-cfs-quota="true"
May 25 13:05:00.131354 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.128137    2580 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
May 25 13:05:00.131354 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.128144    2580 flags.go:64] FLAG: --cpu-manager-policy="none"
May 25 13:05:00.131354 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.128149    2580 flags.go:64] FLAG: --cpu-manager-policy-options=""
May 25 13:05:00.131354 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.128156    2580 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
May 25 13:05:00.131354 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.128161    2580 flags.go:64] FLAG: --enable-controller-attach-detach="true"
May 25 13:05:00.131995 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.128166    2580 flags.go:64] FLAG: --enable-debugging-handlers="true"
May 25 13:05:00.131995 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.128171    2580 flags.go:64] FLAG: --enable-load-reader="false"
May 25 13:05:00.131995 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.128176    2580 flags.go:64] FLAG: --enable-server="true"
May 25 13:05:00.131995 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.128181    2580 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
May 25 13:05:00.131995 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.128189    2580 flags.go:64] FLAG: --event-burst="100"
May 25 13:05:00.131995 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.128194    2580 flags.go:64] FLAG: --event-qps="50"
May 25 13:05:00.131995 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.128199    2580 flags.go:64] FLAG: --event-storage-age-limit="default=0"
May 25 13:05:00.131995 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.128204    2580 flags.go:64] FLAG: --event-storage-event-limit="default=0"
May 25 13:05:00.131995 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.128209    2580 flags.go:64] FLAG: --eviction-hard=""
May 25 13:05:00.131995 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.128215    2580 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
May 25 13:05:00.131995 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.128220    2580 flags.go:64] FLAG: --eviction-minimum-reclaim=""
May 25 13:05:00.131995 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.128225    2580 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
May 25 13:05:00.131995 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.128231    2580 flags.go:64] FLAG: --eviction-soft=""
May 25 13:05:00.131995 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.128236    2580 flags.go:64] FLAG: --eviction-soft-grace-period=""
May 25 13:05:00.131995 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.128241    2580 flags.go:64] FLAG: --exit-on-lock-contention="false"
May 25 13:05:00.131995 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.128246    2580 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
May 25 13:05:00.131995 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.128251    2580 flags.go:64] FLAG: --experimental-mounter-path=""
May 25 13:05:00.131995 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.128256    2580 flags.go:64] FLAG: --fail-cgroupv1="false"
May 25 13:05:00.131995 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.128260    2580 flags.go:64] FLAG: --fail-swap-on="true"
May 25 13:05:00.131995 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.128265    2580 flags.go:64] FLAG: --feature-gates=""
May 25 13:05:00.131995 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.128271    2580 flags.go:64] FLAG: --file-check-frequency="20s"
May 25 13:05:00.131995 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.128276    2580 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
May 25 13:05:00.131995 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.128282    2580 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
May 25 13:05:00.131995 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.128294    2580 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
May 25 13:05:00.131995 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.128299    2580 flags.go:64] FLAG: --healthz-port="10248"
May 25 13:05:00.131995 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.128304    2580 flags.go:64] FLAG: --help="false"
May 25 13:05:00.132731 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.128309    2580 flags.go:64] FLAG: --hostname-override="ip-10-0-135-103.ec2.internal"
May 25 13:05:00.132731 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.128314    2580 flags.go:64] FLAG: --housekeeping-interval="10s"
May 25 13:05:00.132731 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.128319    2580 flags.go:64] FLAG: --http-check-frequency="20s"
May 25 13:05:00.132731 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.128334    2580 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
May 25 13:05:00.132731 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.128340    2580 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
May 25 13:05:00.132731 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.128346    2580 flags.go:64] FLAG: --image-gc-high-threshold="85"
May 25 13:05:00.132731 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.128351    2580 flags.go:64] FLAG: --image-gc-low-threshold="80"
May 25 13:05:00.132731 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.128356    2580 flags.go:64] FLAG: --image-service-endpoint=""
May 25 13:05:00.132731 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.128361    2580 flags.go:64] FLAG: --kernel-memcg-notification="false"
May 25 13:05:00.132731 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.128385    2580 flags.go:64] FLAG: --kube-api-burst="100"
May 25 13:05:00.132731 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.128390    2580 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
May 25 13:05:00.132731 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.128395    2580 flags.go:64] FLAG: --kube-api-qps="50"
May 25 13:05:00.132731 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.128400    2580 flags.go:64] FLAG: --kube-reserved=""
May 25 13:05:00.132731 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.128405    2580 flags.go:64] FLAG: --kube-reserved-cgroup=""
May 25 13:05:00.132731 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.128409    2580 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
May 25 13:05:00.132731 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.128414    2580 flags.go:64] FLAG: --kubelet-cgroups=""
May 25 13:05:00.132731 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.128419    2580 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
May 25 13:05:00.132731 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.128423    2580 flags.go:64] FLAG: --lock-file=""
May 25 13:05:00.132731 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.128427    2580 flags.go:64] FLAG: --log-cadvisor-usage="false"
May 25 13:05:00.132731 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.128432    2580 flags.go:64] FLAG: --log-flush-frequency="5s"
May 25 13:05:00.132731 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.128437    2580 flags.go:64] FLAG: --log-json-info-buffer-size="0"
May 25 13:05:00.132731 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.128446    2580 flags.go:64] FLAG: --log-json-split-stream="false"
May 25 13:05:00.132731 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.128451    2580 flags.go:64] FLAG: --log-text-info-buffer-size="0"
May 25 13:05:00.133287 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.128456    2580 flags.go:64] FLAG: --log-text-split-stream="false"
May 25 13:05:00.133287 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.128460    2580 flags.go:64] FLAG: --logging-format="text"
May 25 13:05:00.133287 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.128465    2580 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
May 25 13:05:00.133287 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.128470    2580 flags.go:64] FLAG: --make-iptables-util-chains="true"
May 25 13:05:00.133287 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.128475    2580 flags.go:64] FLAG: --manifest-url=""
May 25 13:05:00.133287 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.128479    2580 flags.go:64] FLAG: --manifest-url-header=""
May 25 13:05:00.133287 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.128489    2580 flags.go:64] FLAG: --max-housekeeping-interval="15s"
May 25 13:05:00.133287 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.128494    2580 flags.go:64] FLAG: --max-open-files="1000000"
May 25 13:05:00.133287 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.128500    2580 flags.go:64] FLAG: --max-pods="110"
May 25 13:05:00.133287 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.128505    2580 flags.go:64] FLAG: --maximum-dead-containers="-1"
May 25 13:05:00.133287 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.128510    2580 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
May 25 13:05:00.133287 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.128515    2580 flags.go:64] FLAG: --memory-manager-policy="None"
May 25 13:05:00.133287 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.128519    2580 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
May 25 13:05:00.133287 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.128524    2580 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
May 25 13:05:00.133287 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.128530    2580 flags.go:64] FLAG: --node-ip="0.0.0.0"
May 25 13:05:00.133287 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.128535    2580 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
May 25 13:05:00.133287 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.128547    2580 flags.go:64] FLAG: --node-status-max-images="50"
May 25 13:05:00.133287 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.128552    2580 flags.go:64] FLAG: --node-status-update-frequency="10s"
May 25 13:05:00.133287 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.128557    2580 flags.go:64] FLAG: --oom-score-adj="-999"
May 25 13:05:00.133287 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.128561    2580 flags.go:64] FLAG: --pod-cidr=""
May 25 13:05:00.133287 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.128566    2580 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:afe88671bf163c42b467d159dc6f0b784dbe28b1ed44e171edc9c06fd99cc6fc"
May 25 13:05:00.133287 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.128576    2580 flags.go:64] FLAG: --pod-manifest-path=""
May 25 13:05:00.133287 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.128580    2580 flags.go:64] FLAG: --pod-max-pids="-1"
May 25 13:05:00.133287 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.128585    2580 flags.go:64] FLAG: --pods-per-core="0"
May 25 13:05:00.133910 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.128590    2580 flags.go:64] FLAG: --port="10250"
May 25 13:05:00.133910 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.128595    2580 flags.go:64] FLAG: --protect-kernel-defaults="false"
May 25 13:05:00.133910 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.128599    2580 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0d39f722893b641e0"
May 25 13:05:00.133910 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.128604    2580 flags.go:64] FLAG: --qos-reserved=""
May 25 13:05:00.133910 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.128608    2580 flags.go:64] FLAG: --read-only-port="10255"
May 25 13:05:00.133910 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.128613    2580 flags.go:64] FLAG: --register-node="true"
May 25 13:05:00.133910 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.128617    2580 flags.go:64] FLAG: --register-schedulable="true"
May 25 13:05:00.133910 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.128621    2580 flags.go:64] FLAG: --register-with-taints=""
May 25 13:05:00.133910 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.128627    2580 flags.go:64] FLAG: --registry-burst="10"
May 25 13:05:00.133910 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.128631    2580 flags.go:64] FLAG: --registry-qps="5"
May 25 13:05:00.133910 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.128636    2580 flags.go:64] FLAG: --reserved-cpus=""
May 25 13:05:00.133910 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.128640    2580 flags.go:64] FLAG: --reserved-memory=""
May 25 13:05:00.133910 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.128647    2580 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
May 25 13:05:00.133910 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.128651    2580 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
May 25 13:05:00.133910 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.128661    2580 flags.go:64] FLAG: --rotate-certificates="false"
May 25 13:05:00.133910 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.128666    2580 flags.go:64] FLAG: --rotate-server-certificates="false"
May 25 13:05:00.133910 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.128670    2580 flags.go:64] FLAG: --runonce="false"
May 25 13:05:00.133910 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.128675    2580 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
May 25 13:05:00.133910 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.128681    2580 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
May 25 13:05:00.133910 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.128686    2580 flags.go:64] FLAG: --seccomp-default="false"
May 25 13:05:00.133910 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.128690    2580 flags.go:64] FLAG: --serialize-image-pulls="true"
May 25 13:05:00.133910 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.128695    2580 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
May 25 13:05:00.133910 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.128701    2580 flags.go:64] FLAG: --storage-driver-db="cadvisor"
May 25 13:05:00.133910 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.128706    2580 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
May 25 13:05:00.133910 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.128710    2580 flags.go:64] FLAG: --storage-driver-password="root"
May 25 13:05:00.133910 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.128715    2580 flags.go:64] FLAG: --storage-driver-secure="false"
May 25 13:05:00.134550 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.128720    2580 flags.go:64] FLAG: --storage-driver-table="stats"
May 25 13:05:00.134550 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.128724    2580 flags.go:64] FLAG: --storage-driver-user="root"
May 25 13:05:00.134550 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.128729    2580 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
May 25 13:05:00.134550 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.128734    2580 flags.go:64] FLAG: --sync-frequency="1m0s"
May 25 13:05:00.134550 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.128738    2580 flags.go:64] FLAG: --system-cgroups=""
May 25 13:05:00.134550 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.128743    2580 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
May 25 13:05:00.134550 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.128751    2580 flags.go:64] FLAG: --system-reserved-cgroup=""
May 25 13:05:00.134550 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.128756    2580 flags.go:64] FLAG: --tls-cert-file=""
May 25 13:05:00.134550 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.128760    2580 flags.go:64] FLAG: --tls-cipher-suites="[]"
May 25 13:05:00.134550 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.128768    2580 flags.go:64] FLAG: --tls-min-version=""
May 25 13:05:00.134550 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.128772    2580 flags.go:64] FLAG: --tls-private-key-file=""
May 25 13:05:00.134550 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.128776    2580 flags.go:64] FLAG: --topology-manager-policy="none"
May 25 13:05:00.134550 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.128781    2580 flags.go:64] FLAG: --topology-manager-policy-options=""
May 25 13:05:00.134550 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.128786    2580 flags.go:64] FLAG: --topology-manager-scope="container"
May 25 13:05:00.134550 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.128791    2580 flags.go:64] FLAG: --v="2"
May 25 13:05:00.134550 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.128798    2580 flags.go:64] FLAG: --version="false"
May 25 13:05:00.134550 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.128804    2580 flags.go:64] FLAG: --vmodule=""
May 25 13:05:00.134550 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.128811    2580 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
May 25 13:05:00.134550 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.128817    2580 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
May 25 13:05:00.134550 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.128981    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 25 13:05:00.134550 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.128990    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 25 13:05:00.134550 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.128994    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 25 13:05:00.134550 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.128998    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 25 13:05:00.134550 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.129002    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 25 13:05:00.135130 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.129006    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 25 13:05:00.135130 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.129010    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 25 13:05:00.135130 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.129014    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 25 13:05:00.135130 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.129018    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 25 13:05:00.135130 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.129022    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 25 13:05:00.135130 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.129026    2580 feature_gate.go:328] unrecognized feature gate: Example
May 25 13:05:00.135130 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.129030    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 25 13:05:00.135130 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.129034    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 25 13:05:00.135130 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.129038    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 25 13:05:00.135130 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.129041    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 25 13:05:00.135130 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.129046    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 25 13:05:00.135130 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.129050    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 25 13:05:00.135130 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.129055    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 25 13:05:00.135130 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.129059    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 25 13:05:00.135130 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.129063    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 25 13:05:00.135130 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.129067    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 25 13:05:00.135130 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.129071    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 25 13:05:00.135130 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.129074    2580 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 25 13:05:00.135130 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.129078    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 25 13:05:00.135130 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.129082    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 25 13:05:00.135742 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.129086    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 25 13:05:00.135742 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.129090    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
May 25 13:05:00.135742 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.129096    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 25 13:05:00.135742 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.129102    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 25 13:05:00.135742 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.129107    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 25 13:05:00.135742 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.129111    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 25 13:05:00.135742 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.129115    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 25 13:05:00.135742 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.129121    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 25 13:05:00.135742 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.129131    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 25 13:05:00.135742 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.129135    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 25 13:05:00.135742 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.129140    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 25 13:05:00.135742 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.129144    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 25 13:05:00.135742 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.129148    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 25 13:05:00.135742 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.129152    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 25 13:05:00.135742 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.129156    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 25 13:05:00.135742 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.129161    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 25 13:05:00.135742 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.129165    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 25 13:05:00.135742 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.129169    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 25 13:05:00.135742 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.129178    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 25 13:05:00.136223 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.129183    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 25 13:05:00.136223 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.129187    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 25 13:05:00.136223 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.129191    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 25 13:05:00.136223 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.129195    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 25 13:05:00.136223 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.129199    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 25 13:05:00.136223 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.129203    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 25 13:05:00.136223 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.129208    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 25 13:05:00.136223 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.129212    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 25 13:05:00.136223 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.129215    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 25 13:05:00.136223 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.129219    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 25 13:05:00.136223 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.129223    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 25 13:05:00.136223 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.129228    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 25 13:05:00.136223 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.129232    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 25 13:05:00.136223 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.129236    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 25 13:05:00.136223 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.129240    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 25 13:05:00.136223 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.129244    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 25 13:05:00.136223 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.129248    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 25 13:05:00.136223 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.129252    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 25 13:05:00.136223 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.129256    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 25 13:05:00.136727 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.129260    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 25 13:05:00.136727 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.129264    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 25 13:05:00.136727 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.129271    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 25 13:05:00.136727 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.129275    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 25 13:05:00.136727 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.129279    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 25 13:05:00.136727 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.129283    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 25 13:05:00.136727 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.129287    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 25 13:05:00.136727 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.129291    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 25 13:05:00.136727 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.129295    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 25 13:05:00.136727 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.129299    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 25 13:05:00.136727 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.129304    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 25 13:05:00.136727 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.129308    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 25 13:05:00.136727 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.129311    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 25 13:05:00.136727 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.129316    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 25 13:05:00.136727 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.129321    2580 feature_gate.go:328] unrecognized feature gate: Example2
May 25 13:05:00.136727 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.129326    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 25 13:05:00.136727 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.129330    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 25 13:05:00.136727 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.129334    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 25 13:05:00.136727 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.129338    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 25 13:05:00.136727 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.129342    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 25 13:05:00.137220 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.129346    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
May 25 13:05:00.137220 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.129350    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 25 13:05:00.137220 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.129353    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 25 13:05:00.137220 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.130081    2580 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]}
May 25 13:05:00.137332 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.137231    2580 server.go:530] "Kubelet version" kubeletVersion="v1.33.11"
May 25 13:05:00.137332 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.137247    2580 server.go:532] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
May 25 13:05:00.137332 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.137298    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 25 13:05:00.137332 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.137304    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 25 13:05:00.137332 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.137308    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 25 13:05:00.137332 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.137311    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 25 13:05:00.137332 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.137314    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 25 13:05:00.137332 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.137317    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 25 13:05:00.137332 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.137320    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 25 13:05:00.137332 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.137322    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 25 13:05:00.137332 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.137325    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 25 13:05:00.137332 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.137327    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 25 13:05:00.137332 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.137330    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 25 13:05:00.137332 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.137333    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 25 13:05:00.137332 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.137335    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 25 13:05:00.137332 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.137338    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 25 13:05:00.137332 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.137340    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 25 13:05:00.137783 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.137343    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 25 13:05:00.137783 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.137346    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 25 13:05:00.137783 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.137349    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 25 13:05:00.137783 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.137351    2580 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 25 13:05:00.137783 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.137355    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 25 13:05:00.137783 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.137357    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 25 13:05:00.137783 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.137360    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 25 13:05:00.137783 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.137379    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 25 13:05:00.137783 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.137381    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 25 13:05:00.137783 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.137384    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 25 13:05:00.137783 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.137386    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 25 13:05:00.137783 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.137389    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 25 13:05:00.137783 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.137391    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 25 13:05:00.137783 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.137395    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 25 13:05:00.137783 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.137398    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 25 13:05:00.137783 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.137401    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
May 25 13:05:00.137783 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.137403    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 25 13:05:00.137783 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.137406    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 25 13:05:00.137783 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.137408    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 25 13:05:00.137783 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.137411    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 25 13:05:00.138276 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.137413    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 25 13:05:00.138276 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.137416    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 25 13:05:00.138276 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.137418    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 25 13:05:00.138276 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.137421    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 25 13:05:00.138276 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.137423    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 25 13:05:00.138276 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.137425    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 25 13:05:00.138276 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.137428    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 25 13:05:00.138276 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.137430    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 25 13:05:00.138276 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.137433    2580 feature_gate.go:328] unrecognized feature gate: Example
May 25 13:05:00.138276 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.137436    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
May 25 13:05:00.138276 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.137438    2580 feature_gate.go:328] unrecognized feature gate: Example2
May 25 13:05:00.138276 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.137441    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 25 13:05:00.138276 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.137443    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 25 13:05:00.138276 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.137447    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 25 13:05:00.138276 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.137449    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 25 13:05:00.138276 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.137452    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 25 13:05:00.138276 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.137454    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 25 13:05:00.138276 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.137457    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 25 13:05:00.138276 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.137460    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 25 13:05:00.138276 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.137462    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 25 13:05:00.138804 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.137465    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 25 13:05:00.138804 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.137467    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 25 13:05:00.138804 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.137471    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 25 13:05:00.138804 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.137476    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 25 13:05:00.138804 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.137479    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 25 13:05:00.138804 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.137482    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 25 13:05:00.138804 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.137485    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 25 13:05:00.138804 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.137488    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 25 13:05:00.138804 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.137491    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 25 13:05:00.138804 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.137494    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 25 13:05:00.138804 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.137496    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 25 13:05:00.138804 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.137499    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 25 13:05:00.138804 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.137501    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 25 13:05:00.138804 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.137504    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 25 13:05:00.138804 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.137506    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 25 13:05:00.138804 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.137508    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 25 13:05:00.138804 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.137511    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 25 13:05:00.138804 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.137513    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 25 13:05:00.138804 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.137516    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 25 13:05:00.139265 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.137518    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 25 13:05:00.139265 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.137521    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 25 13:05:00.139265 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.137523    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 25 13:05:00.139265 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.137526    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 25 13:05:00.139265 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.137528    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 25 13:05:00.139265 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.137531    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 25 13:05:00.139265 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.137535    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 25 13:05:00.139265 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.137538    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 25 13:05:00.139265 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.137540    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 25 13:05:00.139265 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.137542    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 25 13:05:00.139265 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.137545    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 25 13:05:00.139265 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.137548    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 25 13:05:00.139265 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.137553    2580 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]}
May 25 13:05:00.139265 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.137651    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 25 13:05:00.139265 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.137657    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 25 13:05:00.139666 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.137660    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 25 13:05:00.139666 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.137663    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 25 13:05:00.139666 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.137666    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 25 13:05:00.139666 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.137669    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 25 13:05:00.139666 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.137672    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 25 13:05:00.139666 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.137675    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 25 13:05:00.139666 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.137677    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 25 13:05:00.139666 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.137681    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 25 13:05:00.139666 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.137685    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 25 13:05:00.139666 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.137687    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 25 13:05:00.139666 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.137690    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 25 13:05:00.139666 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.137692    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 25 13:05:00.139666 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.137695    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 25 13:05:00.139666 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.137697    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 25 13:05:00.139666 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.137700    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 25 13:05:00.139666 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.137702    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 25 13:05:00.139666 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.137705    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 25 13:05:00.139666 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.137707    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 25 13:05:00.139666 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.137710    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 25 13:05:00.140138 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.137712    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 25 13:05:00.140138 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.137714    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
May 25 13:05:00.140138 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.137717    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 25 13:05:00.140138 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.137719    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 25 13:05:00.140138 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.137722    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 25 13:05:00.140138 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.137724    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 25 13:05:00.140138 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.137727    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 25 13:05:00.140138 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.137730    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 25 13:05:00.140138 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.137732    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 25 13:05:00.140138 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.137735    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 25 13:05:00.140138 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.137738    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 25 13:05:00.140138 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.137740    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 25 13:05:00.140138 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.137743    2580 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 25 13:05:00.140138 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.137745    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 25 13:05:00.140138 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.137748    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 25 13:05:00.140138 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.137750    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 25 13:05:00.140138 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.137753    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 25 13:05:00.140138 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.137755    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 25 13:05:00.140138 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.137757    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 25 13:05:00.140138 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.137760    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 25 13:05:00.140639 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.137762    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 25 13:05:00.140639 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.137765    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 25 13:05:00.140639 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.137767    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 25 13:05:00.140639 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.137769    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 25 13:05:00.140639 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.137772    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 25 13:05:00.140639 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.137774    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 25 13:05:00.140639 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.137776    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 25 13:05:00.140639 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.137779    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 25 13:05:00.140639 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.137781    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 25 13:05:00.140639 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.137784    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 25 13:05:00.140639 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.137786    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 25 13:05:00.140639 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.137789    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 25 13:05:00.140639 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.137791    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 25 13:05:00.140639 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.137794    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 25 13:05:00.140639 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.137797    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 25 13:05:00.140639 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.137799    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 25 13:05:00.140639 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.137802    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 25 13:05:00.140639 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.137805    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 25 13:05:00.140639 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.137808    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 25 13:05:00.141109 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.137810    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 25 13:05:00.141109 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.137812    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 25 13:05:00.141109 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.137815    2580 feature_gate.go:328] unrecognized feature gate: Example2
May 25 13:05:00.141109 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.137817    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 25 13:05:00.141109 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.137821    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 25 13:05:00.141109 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.137823    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 25 13:05:00.141109 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.137826    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 25 13:05:00.141109 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.137828    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 25 13:05:00.141109 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.137830    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
May 25 13:05:00.141109 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.137833    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 25 13:05:00.141109 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.137835    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 25 13:05:00.141109 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.137838    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 25 13:05:00.141109 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.137840    2580 feature_gate.go:328] unrecognized feature gate: Example
May 25 13:05:00.141109 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.137842    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 25 13:05:00.141109 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.137845    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 25 13:05:00.141109 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.137847    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 25 13:05:00.141109 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.137849    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 25 13:05:00.141109 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.137852    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 25 13:05:00.141109 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.137854    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 25 13:05:00.141109 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.137856    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 25 13:05:00.141608 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.137859    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 25 13:05:00.141608 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.137862    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 25 13:05:00.141608 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.137864    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 25 13:05:00.141608 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.137866    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 25 13:05:00.141608 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.137869    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 25 13:05:00.141608 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:00.137871    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 25 13:05:00.141608 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.137876    2580 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]}
May 25 13:05:00.141608 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.138493    2580 server.go:962] "Client rotation is on, will bootstrap in background"
May 25 13:05:00.141608 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.140306    2580 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
May 25 13:05:00.141608 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.141460    2580 server.go:1019] "Starting client certificate rotation"
May 25 13:05:00.141608 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.141559    2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
May 25 13:05:00.141608 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.141599    2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
May 25 13:05:00.168194 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.168168    2580 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
May 25 13:05:00.170094 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.170024    2580 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
May 25 13:05:00.182345 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.182322    2580 log.go:25] "Validated CRI v1 runtime API"
May 25 13:05:00.187166 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.187147    2580 log.go:25] "Validated CRI v1 image API"
May 25 13:05:00.188398 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.188380    2580 server.go:1452] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
May 25 13:05:00.192776 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.192753    2580 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 f26333cb-52b2-4633-a900-b69e413aabe7:/dev/nvme0n1p4 ffccf4fb-64e4-494c-8c79-adf26f2dfcd3:/dev/nvme0n1p3]
May 25 13:05:00.192842 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.192775    2580 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
May 25 13:05:00.197251 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.197148    2580 manager.go:217] Machine: {Timestamp:2026-05-25 13:05:00.196633957 +0000 UTC m=+0.388928308 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3180760 MemoryCapacity:32812179456 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2bf68e0d0d86a220a0c368f980ec99 SystemUUID:ec2bf68e-0d0d-86a2-20a0-c368f980ec99 BootID:c1c76947-8143-4ac9-b3c8-74610939189b Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406089728 Type:vfs Inodes:4005393 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562439168 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6103040 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406089728 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:db:39:20:cd:6b Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:db:39:20:cd:6b Speed:0 Mtu:9001} {Name:ovs-system MacAddress:62:0c:c6:6f:59:03 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812179456 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
May 25 13:05:00.198101 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.198089    2580 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
May 25 13:05:00.198255 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.198229    2580 manager.go:233] Version: {KernelVersion:5.14.0-570.114.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260512-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
May 25 13:05:00.198361 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.198347    2580 reflector.go:430] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
May 25 13:05:00.199957 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.199929    2580 container_manager_linux.go:270] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
May 25 13:05:00.200131 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.199958    2580 container_manager_linux.go:275] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-135-103.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
May 25 13:05:00.200190 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.200141    2580 topology_manager.go:138] "Creating topology manager with none policy"
May 25 13:05:00.200190 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.200150    2580 container_manager_linux.go:306] "Creating device plugin manager"
May 25 13:05:00.200190 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.200158    2580 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
May 25 13:05:00.201762 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.201750    2580 server.go:72] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
May 25 13:05:00.204260 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.204248    2580 state_mem.go:36] "Initialized new in-memory state store"
May 25 13:05:00.204434 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.204424    2580 server.go:1267] "Using root directory" path="/var/lib/kubelet"
May 25 13:05:00.209089 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.209076    2580 kubelet.go:491] "Attempting to sync node with API server"
May 25 13:05:00.209148 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.209092    2580 kubelet.go:386] "Adding static pod path" path="/etc/kubernetes/manifests"
May 25 13:05:00.209148 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.209106    2580 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
May 25 13:05:00.209148 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.209116    2580 kubelet.go:397] "Adding apiserver pod source"
May 25 13:05:00.209148 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.209125    2580 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
May 25 13:05:00.210319 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.210306    2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
May 25 13:05:00.210388 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.210325    2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
May 25 13:05:00.213666 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.213645    2580 kuberuntime_manager.go:279] "Container runtime initialized" containerRuntime="cri-o" version="1.33.11-2.rhaos4.20.gitb2a8320.el9" apiVersion="v1"
May 25 13:05:00.215686 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.215673    2580 kubelet.go:953] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
May 25 13:05:00.216687 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.216676    2580 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
May 25 13:05:00.216729 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.216692    2580 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
May 25 13:05:00.216729 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.216698    2580 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
May 25 13:05:00.216729 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.216703    2580 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
May 25 13:05:00.216729 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.216711    2580 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
May 25 13:05:00.216729 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.216717    2580 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/secret"
May 25 13:05:00.216729 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.216723    2580 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
May 25 13:05:00.216729 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.216729    2580 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
May 25 13:05:00.216906 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.216735    2580 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/fc"
May 25 13:05:00.216906 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.216741    2580 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
May 25 13:05:00.216906 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.216754    2580 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/projected"
May 25 13:05:00.217158 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.217150    2580 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
May 25 13:05:00.218066 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.218023    2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-pzcqs"
May 25 13:05:00.218885 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.218868    2580 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/csi"
May 25 13:05:00.218924 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.218890    2580 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/image"
May 25 13:05:00.222163 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:05:00.220640    2580 reflector.go:200] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
May 25 13:05:00.222163 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:05:00.220641    2580 reflector.go:200] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-135-103.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
May 25 13:05:00.222919 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.222900    2580 watchdog_linux.go:99] "Systemd watchdog is not enabled"
May 25 13:05:00.222971 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.222955    2580 server.go:1295] "Started kubelet"
May 25 13:05:00.223091 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.223013    2580 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
May 25 13:05:00.223146 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.223073    2580 ratelimit.go:55] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
May 25 13:05:00.223192 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.223166    2580 server_v1.go:47] "podresources" method="list" useActivePods=true
May 25 13:05:00.223828 ip-10-0-135-103 systemd[1]: Started Kubernetes Kubelet.
May 25 13:05:00.224303 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.224197    2580 server.go:255] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
May 25 13:05:00.225921 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.225905    2580 server.go:317] "Adding debug handlers to kubelet server"
May 25 13:05:00.227396 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.227358    2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-pzcqs"
May 25 13:05:00.228949 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.228929    2580 csi_plugin.go:988] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-135-103.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
May 25 13:05:00.229995 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:05:00.228964    2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-103.ec2.internal.18b2d0ba0a7854e1  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-103.ec2.internal,UID:ip-10-0-135-103.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-135-103.ec2.internal,},FirstTimestamp:2026-05-25 13:05:00.222919905 +0000 UTC m=+0.415214258,LastTimestamp:2026-05-25 13:05:00.222919905 +0000 UTC m=+0.415214258,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-103.ec2.internal,}"
May 25 13:05:00.231688 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.231667    2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
May 25 13:05:00.232148 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.232124    2580 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
May 25 13:05:00.232225 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.232197    2580 volume_manager.go:295] "The desired_state_of_world populator starts"
May 25 13:05:00.232225 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.232208    2580 volume_manager.go:297] "Starting Kubelet Volume Manager"
May 25 13:05:00.232312 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.232286    2580 desired_state_of_world_populator.go:150] "Desired state populator starts to run"
May 25 13:05:00.232355 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.232336    2580 reconstruct.go:97] "Volume reconstruction finished"
May 25 13:05:00.232355 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.232341    2580 reconciler.go:26] "Reconciler: start to sync state"
May 25 13:05:00.233654 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:05:00.233489    2580 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-135-103.ec2.internal\" not found"
May 25 13:05:00.236011 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.235991    2580 factory.go:153] Registering CRI-O factory
May 25 13:05:00.236117 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.236051    2580 factory.go:223] Registration of the crio container factory successfully
May 25 13:05:00.236176 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.236143    2580 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
May 25 13:05:00.236176 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.236154    2580 factory.go:55] Registering systemd factory
May 25 13:05:00.236176 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.236162    2580 factory.go:223] Registration of the systemd container factory successfully
May 25 13:05:00.236308 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.236182    2580 factory.go:103] Registering Raw factory
May 25 13:05:00.236308 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.236195    2580 manager.go:1196] Started watching for new ooms in manager
May 25 13:05:00.236657 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.236640    2580 manager.go:319] Starting recovery of all containers
May 25 13:05:00.238134 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:05:00.238109    2580 kubelet.go:1618] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
May 25 13:05:00.239625 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.239600    2580 reflector.go:430] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
May 25 13:05:00.243532 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:05:00.243507    2580 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-135-103.ec2.internal\" not found" node="ip-10-0-135-103.ec2.internal"
May 25 13:05:00.248627 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.248604    2580 manager.go:324] Recovery completed
May 25 13:05:00.252601 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.252587    2580 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 25 13:05:00.254546 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.254530    2580 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-135-103.ec2.internal" event="NodeHasSufficientMemory"
May 25 13:05:00.254625 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.254559    2580 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-135-103.ec2.internal" event="NodeHasNoDiskPressure"
May 25 13:05:00.254625 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.254569    2580 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-135-103.ec2.internal" event="NodeHasSufficientPID"
May 25 13:05:00.255053 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.255041    2580 cpu_manager.go:222] "Starting CPU manager" policy="none"
May 25 13:05:00.255053 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.255053    2580 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
May 25 13:05:00.255138 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.255069    2580 state_mem.go:36] "Initialized new in-memory state store"
May 25 13:05:00.257443 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.257432    2580 policy_none.go:49] "None policy: Start"
May 25 13:05:00.257483 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.257449    2580 memory_manager.go:186] "Starting memorymanager" policy="None"
May 25 13:05:00.257483 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.257459    2580 state_mem.go:35] "Initializing new in-memory state store"
May 25 13:05:00.310574 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.296868    2580 manager.go:341] "Starting Device Plugin manager"
May 25 13:05:00.310574 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:05:00.296939    2580 manager.go:517] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
May 25 13:05:00.310574 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.296953    2580 server.go:85] "Starting device plugin registration server"
May 25 13:05:00.310574 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.297203    2580 eviction_manager.go:189] "Eviction manager: starting control loop"
May 25 13:05:00.310574 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.297214    2580 container_log_manager.go:189] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
May 25 13:05:00.310574 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.297303    2580 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
May 25 13:05:00.310574 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.297395    2580 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
May 25 13:05:00.310574 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.297403    2580 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
May 25 13:05:00.310574 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:05:00.298031    2580 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
May 25 13:05:00.310574 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:05:00.298066    2580 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-135-103.ec2.internal\" not found"
May 25 13:05:00.370780 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.370700    2580 kubelet_network_linux.go:49] "Initialized iptables rules." protocol="IPv4"
May 25 13:05:00.372109 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.372075    2580 kubelet_network_linux.go:49] "Initialized iptables rules." protocol="IPv6"
May 25 13:05:00.372109 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.372099    2580 status_manager.go:230] "Starting to sync pod status with apiserver"
May 25 13:05:00.372259 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.372140    2580 watchdog_linux.go:127] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
May 25 13:05:00.372259 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.372162    2580 kubelet.go:2451] "Starting kubelet main sync loop"
May 25 13:05:00.372259 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:05:00.372209    2580 kubelet.go:2475] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
May 25 13:05:00.376064 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.376047    2580 reflector.go:430] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
May 25 13:05:00.398275 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.398254    2580 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 25 13:05:00.399111 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.399096    2580 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-135-103.ec2.internal" event="NodeHasSufficientMemory"
May 25 13:05:00.399183 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.399124    2580 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-135-103.ec2.internal" event="NodeHasNoDiskPressure"
May 25 13:05:00.399183 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.399135    2580 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-135-103.ec2.internal" event="NodeHasSufficientPID"
May 25 13:05:00.399183 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.399158    2580 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-135-103.ec2.internal"
May 25 13:05:00.407863 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.407849    2580 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-135-103.ec2.internal"
May 25 13:05:00.407921 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:05:00.407872    2580 kubelet_node_status.go:597] "Error updating node status, will retry" err="error getting node \"ip-10-0-135-103.ec2.internal\": node \"ip-10-0-135-103.ec2.internal\" not found"
May 25 13:05:00.424824 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:05:00.424800    2580 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-135-103.ec2.internal\" not found"
May 25 13:05:00.472294 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.472262    2580 kubelet.go:2537] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-103.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-135-103.ec2.internal"]
May 25 13:05:00.472384 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.472343    2580 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 25 13:05:00.473302 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.473286    2580 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-135-103.ec2.internal" event="NodeHasSufficientMemory"
May 25 13:05:00.473392 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.473315    2580 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-135-103.ec2.internal" event="NodeHasNoDiskPressure"
May 25 13:05:00.473392 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.473325    2580 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-135-103.ec2.internal" event="NodeHasSufficientPID"
May 25 13:05:00.474509 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.474497    2580 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 25 13:05:00.474703 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.474690    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-103.ec2.internal"
May 25 13:05:00.474747 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.474729    2580 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 25 13:05:00.475223 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.475208    2580 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-135-103.ec2.internal" event="NodeHasSufficientMemory"
May 25 13:05:00.475273 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.475225    2580 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-135-103.ec2.internal" event="NodeHasSufficientMemory"
May 25 13:05:00.475273 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.475235    2580 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-135-103.ec2.internal" event="NodeHasNoDiskPressure"
May 25 13:05:00.475273 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.475248    2580 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-135-103.ec2.internal" event="NodeHasSufficientPID"
May 25 13:05:00.475273 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.475262    2580 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-135-103.ec2.internal" event="NodeHasNoDiskPressure"
May 25 13:05:00.475273 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.475274    2580 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-135-103.ec2.internal" event="NodeHasSufficientPID"
May 25 13:05:00.476288 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.476263    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-103.ec2.internal"
May 25 13:05:00.476393 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.476297    2580 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 25 13:05:00.477103 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.477088    2580 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-135-103.ec2.internal" event="NodeHasSufficientMemory"
May 25 13:05:00.477178 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.477120    2580 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-135-103.ec2.internal" event="NodeHasNoDiskPressure"
May 25 13:05:00.477178 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.477131    2580 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-135-103.ec2.internal" event="NodeHasSufficientPID"
May 25 13:05:00.507060 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:05:00.507043    2580 kubelet.go:3336] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-103.ec2.internal\" not found" node="ip-10-0-135-103.ec2.internal"
May 25 13:05:00.511245 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:05:00.511227    2580 kubelet.go:3336] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-103.ec2.internal\" not found" node="ip-10-0-135-103.ec2.internal"
May 25 13:05:00.525145 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:05:00.525106    2580 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-135-103.ec2.internal\" not found"
May 25 13:05:00.533712 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.533681    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d8b0c93f69930a962fcb9912400f9315-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-103.ec2.internal\" (UID: \"d8b0c93f69930a962fcb9912400f9315\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-103.ec2.internal"
May 25 13:05:00.533811 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.533717    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d8b0c93f69930a962fcb9912400f9315-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-103.ec2.internal\" (UID: \"d8b0c93f69930a962fcb9912400f9315\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-103.ec2.internal"
May 25 13:05:00.533811 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.533744    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/06181dfa378829b92cc5ea772451a452-config\") pod \"kube-apiserver-proxy-ip-10-0-135-103.ec2.internal\" (UID: \"06181dfa378829b92cc5ea772451a452\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-103.ec2.internal"
May 25 13:05:00.625507 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:05:00.625419    2580 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-135-103.ec2.internal\" not found"
May 25 13:05:00.634837 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.634798    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d8b0c93f69930a962fcb9912400f9315-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-103.ec2.internal\" (UID: \"d8b0c93f69930a962fcb9912400f9315\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-103.ec2.internal"
May 25 13:05:00.634920 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.634848    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d8b0c93f69930a962fcb9912400f9315-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-103.ec2.internal\" (UID: \"d8b0c93f69930a962fcb9912400f9315\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-103.ec2.internal"
May 25 13:05:00.634920 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.634895    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/06181dfa378829b92cc5ea772451a452-config\") pod \"kube-apiserver-proxy-ip-10-0-135-103.ec2.internal\" (UID: \"06181dfa378829b92cc5ea772451a452\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-103.ec2.internal"
May 25 13:05:00.634920 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.634899    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d8b0c93f69930a962fcb9912400f9315-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-103.ec2.internal\" (UID: \"d8b0c93f69930a962fcb9912400f9315\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-103.ec2.internal"
May 25 13:05:00.635048 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.634953    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/06181dfa378829b92cc5ea772451a452-config\") pod \"kube-apiserver-proxy-ip-10-0-135-103.ec2.internal\" (UID: \"06181dfa378829b92cc5ea772451a452\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-103.ec2.internal"
May 25 13:05:00.635048 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.635000    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d8b0c93f69930a962fcb9912400f9315-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-103.ec2.internal\" (UID: \"d8b0c93f69930a962fcb9912400f9315\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-103.ec2.internal"
May 25 13:05:00.726268 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:05:00.726222    2580 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-135-103.ec2.internal\" not found"
May 25 13:05:00.809734 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.809699    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-103.ec2.internal"
May 25 13:05:00.813523 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:00.813505    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-103.ec2.internal"
May 25 13:05:00.827162 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:05:00.827123    2580 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-135-103.ec2.internal\" not found"
May 25 13:05:00.927698 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:05:00.927607    2580 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-135-103.ec2.internal\" not found"
May 25 13:05:01.028130 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:05:01.028096    2580 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-135-103.ec2.internal\" not found"
May 25 13:05:01.128642 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:05:01.128609    2580 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-135-103.ec2.internal\" not found"
May 25 13:05:01.141117 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:01.141091    2580 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
May 25 13:05:01.141282 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:01.141261    2580 reflector.go:556] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
May 25 13:05:01.141351 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:01.141312    2580 reflector.go:556] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
May 25 13:05:01.228772 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:05:01.228695    2580 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-135-103.ec2.internal\" not found"
May 25 13:05:01.230459 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:01.230428    2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-05-24 13:00:00 +0000 UTC" deadline="2027-12-04 17:39:21.556716518 +0000 UTC"
May 25 13:05:01.230459 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:01.230456    2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13396h34m20.326262939s"
May 25 13:05:01.232689 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:01.232675    2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
May 25 13:05:01.248078 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:01.248056    2580 reflector.go:430] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
May 25 13:05:01.276589 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:01.276566    2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-4b49k"
May 25 13:05:01.284402 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:01.284350    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06181dfa378829b92cc5ea772451a452.slice/crio-8a64be40ce1c366a9ae0df4e4f3e49634cc476dff6e7911ede6124afc284ddde WatchSource:0}: Error finding container 8a64be40ce1c366a9ae0df4e4f3e49634cc476dff6e7911ede6124afc284ddde: Status 404 returned error can't find the container with id 8a64be40ce1c366a9ae0df4e4f3e49634cc476dff6e7911ede6124afc284ddde
May 25 13:05:01.284595 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:01.284571    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8b0c93f69930a962fcb9912400f9315.slice/crio-1ba3c4de8d35f378b34f2bb4911228d1d0665b6963bfdde3eb4dd970d737d242 WatchSource:0}: Error finding container 1ba3c4de8d35f378b34f2bb4911228d1d0665b6963bfdde3eb4dd970d737d242: Status 404 returned error can't find the container with id 1ba3c4de8d35f378b34f2bb4911228d1d0665b6963bfdde3eb4dd970d737d242
May 25 13:05:01.284719 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:01.284696    2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-4b49k"
May 25 13:05:01.287799 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:01.287785    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 25 13:05:01.329822 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:05:01.329793    2580 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-135-103.ec2.internal\" not found"
May 25 13:05:01.374939 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:01.374890    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-103.ec2.internal" event={"ID":"06181dfa378829b92cc5ea772451a452","Type":"ContainerStarted","Data":"8a64be40ce1c366a9ae0df4e4f3e49634cc476dff6e7911ede6124afc284ddde"}
May 25 13:05:01.375708 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:01.375679    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-103.ec2.internal" event={"ID":"d8b0c93f69930a962fcb9912400f9315","Type":"ContainerStarted","Data":"1ba3c4de8d35f378b34f2bb4911228d1d0665b6963bfdde3eb4dd970d737d242"}
May 25 13:05:01.430894 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:05:01.430863    2580 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-135-103.ec2.internal\" not found"
May 25 13:05:01.531480 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:05:01.531384    2580 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-135-103.ec2.internal\" not found"
May 25 13:05:01.574182 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:01.574144    2580 reflector.go:430] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
May 25 13:05:01.631518 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:05:01.631469    2580 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-135-103.ec2.internal\" not found"
May 25 13:05:01.643158 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:01.643133    2580 reflector.go:430] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
May 25 13:05:01.732301 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:01.732268    2580 kubelet.go:3340] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-103.ec2.internal"
May 25 13:05:01.741146 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:01.741111    2580 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
May 25 13:05:01.742125 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:01.742107    2580 kubelet.go:3340] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-103.ec2.internal"
May 25 13:05:01.750943 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:01.750926    2580 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
May 25 13:05:02.210052 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.210018    2580 apiserver.go:52] "Watching apiserver"
May 25 13:05:02.215812 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.215786    2580 reflector.go:430] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
May 25 13:05:02.217925 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.217887    2580 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-rqkvc","kube-system/kube-apiserver-proxy-ip-10-0-135-103.ec2.internal","openshift-dns/node-resolver-vlk4j","openshift-multus/multus-additional-cni-plugins-hj74k","openshift-multus/network-metrics-daemon-svlzs","openshift-network-diagnostics/network-check-target-25spc","openshift-ovn-kubernetes/ovnkube-node-646r6","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t2jp2","openshift-cluster-node-tuning-operator/tuned-lpfd5","openshift-image-registry/node-ca-wlq4q","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-103.ec2.internal","openshift-multus/multus-8xdhg","openshift-network-operator/iptables-alerter-nd955"]
May 25 13:05:02.220029 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.220010    2580 reflector.go:430] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
May 25 13:05:02.220731 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.220702    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-646r6"
May 25 13:05:02.222856 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.222838    2580 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
May 25 13:05:02.223706 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.223319    2580 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
May 25 13:05:02.223706 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.223383    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-hsbxd\""
May 25 13:05:02.223706 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.223538    2580 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
May 25 13:05:02.223706 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.223571    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
May 25 13:05:02.223956 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.223929    2580 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
May 25 13:05:02.224159 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.224142    2580 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
May 25 13:05:02.227675 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.225219    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-vlk4j"
May 25 13:05:02.227675 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.225411    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-hj74k"
May 25 13:05:02.227675 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.227585    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-svlzs"
May 25 13:05:02.227675 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:05:02.227663    2580 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-svlzs" podUID="e445efe1-4ac4-4df5-881f-748bcf8c5e0f"
May 25 13:05:02.228003 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.227982    2580 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
May 25 13:05:02.228077 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.228011    2580 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
May 25 13:05:02.228128 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.228072    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-k4hbq\""
May 25 13:05:02.228128 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.228076    2580 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
May 25 13:05:02.228227 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.228151    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-nhngb\""
May 25 13:05:02.228286 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.228228    2580 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
May 25 13:05:02.228332 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.228302    2580 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
May 25 13:05:02.228405 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.228348    2580 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
May 25 13:05:02.228405 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.228382    2580 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
May 25 13:05:02.231429 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.231402    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-25spc"
May 25 13:05:02.231505 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:05:02.231471    2580 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-25spc" podUID="fdd5eb93-deb6-441f-b1fe-3511c4eee77e"
May 25 13:05:02.236301 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.236280    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t2jp2"
May 25 13:05:02.238510 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.238488    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
May 25 13:05:02.238606 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.238510    2580 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
May 25 13:05:02.238606 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.238493    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-vkvjl\""
May 25 13:05:02.238750 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.238726    2580 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
May 25 13:05:02.239836 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.239388    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-lpfd5"
May 25 13:05:02.241475 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.241457    2580 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
May 25 13:05:02.241989 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.241689    2580 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
May 25 13:05:02.241989 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.241691    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-pbglf\""
May 25 13:05:02.242730 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.242708    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-rqkvc"
May 25 13:05:02.242838 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.242822    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-wlq4q"
May 25 13:05:02.244774 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.244753    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/33d153fd-31f6-4481-855e-728e06ce70f5-cni-binary-copy\") pod \"multus-additional-cni-plugins-hj74k\" (UID: \"33d153fd-31f6-4481-855e-728e06ce70f5\") " pod="openshift-multus/multus-additional-cni-plugins-hj74k"
May 25 13:05:02.244857 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.244796    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/59b905b8-5ce3-49f6-b47c-b5b3a525fe06-host-slash\") pod \"ovnkube-node-646r6\" (UID: \"59b905b8-5ce3-49f6-b47c-b5b3a525fe06\") " pod="openshift-ovn-kubernetes/ovnkube-node-646r6"
May 25 13:05:02.244857 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.244821    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/59b905b8-5ce3-49f6-b47c-b5b3a525fe06-host-run-netns\") pod \"ovnkube-node-646r6\" (UID: \"59b905b8-5ce3-49f6-b47c-b5b3a525fe06\") " pod="openshift-ovn-kubernetes/ovnkube-node-646r6"
May 25 13:05:02.244958 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.244874    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/59b905b8-5ce3-49f6-b47c-b5b3a525fe06-run-openvswitch\") pod \"ovnkube-node-646r6\" (UID: \"59b905b8-5ce3-49f6-b47c-b5b3a525fe06\") " pod="openshift-ovn-kubernetes/ovnkube-node-646r6"
May 25 13:05:02.244958 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.244910    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/59b905b8-5ce3-49f6-b47c-b5b3a525fe06-ovnkube-script-lib\") pod \"ovnkube-node-646r6\" (UID: \"59b905b8-5ce3-49f6-b47c-b5b3a525fe06\") " pod="openshift-ovn-kubernetes/ovnkube-node-646r6"
May 25 13:05:02.244958 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.244936    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/33d153fd-31f6-4481-855e-728e06ce70f5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-hj74k\" (UID: \"33d153fd-31f6-4481-855e-728e06ce70f5\") " pod="openshift-multus/multus-additional-cni-plugins-hj74k"
May 25 13:05:02.245089 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.244960    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/59b905b8-5ce3-49f6-b47c-b5b3a525fe06-run-ovn\") pod \"ovnkube-node-646r6\" (UID: \"59b905b8-5ce3-49f6-b47c-b5b3a525fe06\") " pod="openshift-ovn-kubernetes/ovnkube-node-646r6"
May 25 13:05:02.245089 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.244986    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/59b905b8-5ce3-49f6-b47c-b5b3a525fe06-node-log\") pod \"ovnkube-node-646r6\" (UID: \"59b905b8-5ce3-49f6-b47c-b5b3a525fe06\") " pod="openshift-ovn-kubernetes/ovnkube-node-646r6"
May 25 13:05:02.245089 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.245011    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e445efe1-4ac4-4df5-881f-748bcf8c5e0f-metrics-certs\") pod \"network-metrics-daemon-svlzs\" (UID: \"e445efe1-4ac4-4df5-881f-748bcf8c5e0f\") " pod="openshift-multus/network-metrics-daemon-svlzs"
May 25 13:05:02.245089 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.245035    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/59b905b8-5ce3-49f6-b47c-b5b3a525fe06-systemd-units\") pod \"ovnkube-node-646r6\" (UID: \"59b905b8-5ce3-49f6-b47c-b5b3a525fe06\") " pod="openshift-ovn-kubernetes/ovnkube-node-646r6"
May 25 13:05:02.245089 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.245059    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/59b905b8-5ce3-49f6-b47c-b5b3a525fe06-host-cni-netd\") pod \"ovnkube-node-646r6\" (UID: \"59b905b8-5ce3-49f6-b47c-b5b3a525fe06\") " pod="openshift-ovn-kubernetes/ovnkube-node-646r6"
May 25 13:05:02.245255 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.245095    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/59b905b8-5ce3-49f6-b47c-b5b3a525fe06-env-overrides\") pod \"ovnkube-node-646r6\" (UID: \"59b905b8-5ce3-49f6-b47c-b5b3a525fe06\") " pod="openshift-ovn-kubernetes/ovnkube-node-646r6"
May 25 13:05:02.245255 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.245120    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srj9q\" (UniqueName: \"kubernetes.io/projected/59b905b8-5ce3-49f6-b47c-b5b3a525fe06-kube-api-access-srj9q\") pod \"ovnkube-node-646r6\" (UID: \"59b905b8-5ce3-49f6-b47c-b5b3a525fe06\") " pod="openshift-ovn-kubernetes/ovnkube-node-646r6"
May 25 13:05:02.245255 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.245146    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/33d153fd-31f6-4481-855e-728e06ce70f5-os-release\") pod \"multus-additional-cni-plugins-hj74k\" (UID: \"33d153fd-31f6-4481-855e-728e06ce70f5\") " pod="openshift-multus/multus-additional-cni-plugins-hj74k"
May 25 13:05:02.245255 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.245173    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/33d153fd-31f6-4481-855e-728e06ce70f5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-hj74k\" (UID: \"33d153fd-31f6-4481-855e-728e06ce70f5\") " pod="openshift-multus/multus-additional-cni-plugins-hj74k"
May 25 13:05:02.245255 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.245199    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfgrd\" (UniqueName: \"kubernetes.io/projected/fdd5eb93-deb6-441f-b1fe-3511c4eee77e-kube-api-access-lfgrd\") pod \"network-check-target-25spc\" (UID: \"fdd5eb93-deb6-441f-b1fe-3511c4eee77e\") " pod="openshift-network-diagnostics/network-check-target-25spc"
May 25 13:05:02.245255 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.245223    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5a8fc898-888f-420d-a433-27e831c96721-agent-certs\") pod \"konnectivity-agent-rqkvc\" (UID: \"5a8fc898-888f-420d-a433-27e831c96721\") " pod="kube-system/konnectivity-agent-rqkvc"
May 25 13:05:02.245255 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.245245    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/59b905b8-5ce3-49f6-b47c-b5b3a525fe06-run-systemd\") pod \"ovnkube-node-646r6\" (UID: \"59b905b8-5ce3-49f6-b47c-b5b3a525fe06\") " pod="openshift-ovn-kubernetes/ovnkube-node-646r6"
May 25 13:05:02.245545 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.245269    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/59b905b8-5ce3-49f6-b47c-b5b3a525fe06-etc-openvswitch\") pod \"ovnkube-node-646r6\" (UID: \"59b905b8-5ce3-49f6-b47c-b5b3a525fe06\") " pod="openshift-ovn-kubernetes/ovnkube-node-646r6"
May 25 13:05:02.245545 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.245292    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/33d153fd-31f6-4481-855e-728e06ce70f5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-hj74k\" (UID: \"33d153fd-31f6-4481-855e-728e06ce70f5\") " pod="openshift-multus/multus-additional-cni-plugins-hj74k"
May 25 13:05:02.245545 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.245318    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/59b905b8-5ce3-49f6-b47c-b5b3a525fe06-host-kubelet\") pod \"ovnkube-node-646r6\" (UID: \"59b905b8-5ce3-49f6-b47c-b5b3a525fe06\") " pod="openshift-ovn-kubernetes/ovnkube-node-646r6"
May 25 13:05:02.245545 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.245342    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/59b905b8-5ce3-49f6-b47c-b5b3a525fe06-host-cni-bin\") pod \"ovnkube-node-646r6\" (UID: \"59b905b8-5ce3-49f6-b47c-b5b3a525fe06\") " pod="openshift-ovn-kubernetes/ovnkube-node-646r6"
May 25 13:05:02.245749 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.245615    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/59b905b8-5ce3-49f6-b47c-b5b3a525fe06-ovn-node-metrics-cert\") pod \"ovnkube-node-646r6\" (UID: \"59b905b8-5ce3-49f6-b47c-b5b3a525fe06\") " pod="openshift-ovn-kubernetes/ovnkube-node-646r6"
May 25 13:05:02.245749 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.245635    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/37b3e162-412d-4e21-8f2f-bb8398f30435-hosts-file\") pod \"node-resolver-vlk4j\" (UID: \"37b3e162-412d-4e21-8f2f-bb8398f30435\") " pod="openshift-dns/node-resolver-vlk4j"
May 25 13:05:02.245749 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.245680    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/33d153fd-31f6-4481-855e-728e06ce70f5-system-cni-dir\") pod \"multus-additional-cni-plugins-hj74k\" (UID: \"33d153fd-31f6-4481-855e-728e06ce70f5\") " pod="openshift-multus/multus-additional-cni-plugins-hj74k"
May 25 13:05:02.245749 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.245711    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5p2f5\" (UniqueName: \"kubernetes.io/projected/e445efe1-4ac4-4df5-881f-748bcf8c5e0f-kube-api-access-5p2f5\") pod \"network-metrics-daemon-svlzs\" (UID: \"e445efe1-4ac4-4df5-881f-748bcf8c5e0f\") " pod="openshift-multus/network-metrics-daemon-svlzs"
May 25 13:05:02.245749 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.245732    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5a8fc898-888f-420d-a433-27e831c96721-konnectivity-ca\") pod \"konnectivity-agent-rqkvc\" (UID: \"5a8fc898-888f-420d-a433-27e831c96721\") " pod="kube-system/konnectivity-agent-rqkvc"
May 25 13:05:02.245749 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.245746    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/59b905b8-5ce3-49f6-b47c-b5b3a525fe06-var-lib-openvswitch\") pod \"ovnkube-node-646r6\" (UID: \"59b905b8-5ce3-49f6-b47c-b5b3a525fe06\") " pod="openshift-ovn-kubernetes/ovnkube-node-646r6"
May 25 13:05:02.246018 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.245761    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/59b905b8-5ce3-49f6-b47c-b5b3a525fe06-host-run-ovn-kubernetes\") pod \"ovnkube-node-646r6\" (UID: \"59b905b8-5ce3-49f6-b47c-b5b3a525fe06\") " pod="openshift-ovn-kubernetes/ovnkube-node-646r6"
May 25 13:05:02.246018 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.245815    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/59b905b8-5ce3-49f6-b47c-b5b3a525fe06-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-646r6\" (UID: \"59b905b8-5ce3-49f6-b47c-b5b3a525fe06\") " pod="openshift-ovn-kubernetes/ovnkube-node-646r6"
May 25 13:05:02.246018 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.245851    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6t9p\" (UniqueName: \"kubernetes.io/projected/37b3e162-412d-4e21-8f2f-bb8398f30435-kube-api-access-g6t9p\") pod \"node-resolver-vlk4j\" (UID: \"37b3e162-412d-4e21-8f2f-bb8398f30435\") " pod="openshift-dns/node-resolver-vlk4j"
May 25 13:05:02.246018 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.245890    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/33d153fd-31f6-4481-855e-728e06ce70f5-cnibin\") pod \"multus-additional-cni-plugins-hj74k\" (UID: \"33d153fd-31f6-4481-855e-728e06ce70f5\") " pod="openshift-multus/multus-additional-cni-plugins-hj74k"
May 25 13:05:02.246018 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.245908    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvnnx\" (UniqueName: \"kubernetes.io/projected/33d153fd-31f6-4481-855e-728e06ce70f5-kube-api-access-wvnnx\") pod \"multus-additional-cni-plugins-hj74k\" (UID: \"33d153fd-31f6-4481-855e-728e06ce70f5\") " pod="openshift-multus/multus-additional-cni-plugins-hj74k"
May 25 13:05:02.246018 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.245924    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/59b905b8-5ce3-49f6-b47c-b5b3a525fe06-log-socket\") pod \"ovnkube-node-646r6\" (UID: \"59b905b8-5ce3-49f6-b47c-b5b3a525fe06\") " pod="openshift-ovn-kubernetes/ovnkube-node-646r6"
May 25 13:05:02.246018 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.245941    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/59b905b8-5ce3-49f6-b47c-b5b3a525fe06-ovnkube-config\") pod \"ovnkube-node-646r6\" (UID: \"59b905b8-5ce3-49f6-b47c-b5b3a525fe06\") " pod="openshift-ovn-kubernetes/ovnkube-node-646r6"
May 25 13:05:02.246018 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.245962    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37b3e162-412d-4e21-8f2f-bb8398f30435-tmp-dir\") pod \"node-resolver-vlk4j\" (UID: \"37b3e162-412d-4e21-8f2f-bb8398f30435\") " pod="openshift-dns/node-resolver-vlk4j"
May 25 13:05:02.246878 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.246859    2580 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
May 25 13:05:02.247095 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.247078    2580 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
May 25 13:05:02.247155 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.247101    2580 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
May 25 13:05:02.247265 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.247251    2580 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
May 25 13:05:02.247318 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.247287    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-2n6lm\""
May 25 13:05:02.247526 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.247433    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-d76l6\""
May 25 13:05:02.247526 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.247468    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
May 25 13:05:02.247627 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.247613    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-8xdhg"
May 25 13:05:02.247670 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.247636    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-nd955"
May 25 13:05:02.249544 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.249529    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-kgqpk\""
May 25 13:05:02.249739 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.249719    2580 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
May 25 13:05:02.249836 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.249758    2580 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
May 25 13:05:02.249903 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.249886    2580 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
May 25 13:05:02.249958 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.249931    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-s27wz\""
May 25 13:05:02.250140 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.250127    2580 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
May 25 13:05:02.285384 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.285339    2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-05-24 13:00:01 +0000 UTC" deadline="2027-12-08 15:35:11.400267774 +0000 UTC"
May 25 13:05:02.285384 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.285383    2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13490h30m9.114888254s"
May 25 13:05:02.318024 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.317988    2580 reflector.go:430] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
May 25 13:05:02.334409 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.334361    2580 desired_state_of_world_populator.go:158] "Finished populating initial desired state of world"
May 25 13:05:02.346320 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.346289    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/38b712ff-d8de-43d6-9262-4e460ecf7b32-etc-tuned\") pod \"tuned-lpfd5\" (UID: \"38b712ff-d8de-43d6-9262-4e460ecf7b32\") " pod="openshift-cluster-node-tuning-operator/tuned-lpfd5"
May 25 13:05:02.346493 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.346343    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/59b905b8-5ce3-49f6-b47c-b5b3a525fe06-node-log\") pod \"ovnkube-node-646r6\" (UID: \"59b905b8-5ce3-49f6-b47c-b5b3a525fe06\") " pod="openshift-ovn-kubernetes/ovnkube-node-646r6"
May 25 13:05:02.346493 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.346409    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8pwr5\" (UniqueName: \"kubernetes.io/projected/6ef8bb8d-98b5-4fe9-94a7-8e06d34ab971-kube-api-access-8pwr5\") pod \"node-ca-wlq4q\" (UID: \"6ef8bb8d-98b5-4fe9-94a7-8e06d34ab971\") " pod="openshift-image-registry/node-ca-wlq4q"
May 25 13:05:02.346493 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.346435    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c749c4da-8f53-4bb5-98c2-280f3cd403af-multus-daemon-config\") pod \"multus-8xdhg\" (UID: \"c749c4da-8f53-4bb5-98c2-280f3cd403af\") " pod="openshift-multus/multus-8xdhg"
May 25 13:05:02.346493 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.346439    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/59b905b8-5ce3-49f6-b47c-b5b3a525fe06-node-log\") pod \"ovnkube-node-646r6\" (UID: \"59b905b8-5ce3-49f6-b47c-b5b3a525fe06\") " pod="openshift-ovn-kubernetes/ovnkube-node-646r6"
May 25 13:05:02.346493 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.346457    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/10056893-23ca-4931-8142-3979d7611233-socket-dir\") pod \"aws-ebs-csi-driver-node-t2jp2\" (UID: \"10056893-23ca-4931-8142-3979d7611233\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t2jp2"
May 25 13:05:02.346493 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.346479    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/10056893-23ca-4931-8142-3979d7611233-device-dir\") pod \"aws-ebs-csi-driver-node-t2jp2\" (UID: \"10056893-23ca-4931-8142-3979d7611233\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t2jp2"
May 25 13:05:02.346725 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.346502    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/10056893-23ca-4931-8142-3979d7611233-sys-fs\") pod \"aws-ebs-csi-driver-node-t2jp2\" (UID: \"10056893-23ca-4931-8142-3979d7611233\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t2jp2"
May 25 13:05:02.346725 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.346524    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/38b712ff-d8de-43d6-9262-4e460ecf7b32-run\") pod \"tuned-lpfd5\" (UID: \"38b712ff-d8de-43d6-9262-4e460ecf7b32\") " pod="openshift-cluster-node-tuning-operator/tuned-lpfd5"
May 25 13:05:02.346725 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.346551    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e445efe1-4ac4-4df5-881f-748bcf8c5e0f-metrics-certs\") pod \"network-metrics-daemon-svlzs\" (UID: \"e445efe1-4ac4-4df5-881f-748bcf8c5e0f\") " pod="openshift-multus/network-metrics-daemon-svlzs"
May 25 13:05:02.346725 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.346573    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/59b905b8-5ce3-49f6-b47c-b5b3a525fe06-host-cni-netd\") pod \"ovnkube-node-646r6\" (UID: \"59b905b8-5ce3-49f6-b47c-b5b3a525fe06\") " pod="openshift-ovn-kubernetes/ovnkube-node-646r6"
May 25 13:05:02.346725 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.346625    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/59b905b8-5ce3-49f6-b47c-b5b3a525fe06-host-cni-netd\") pod \"ovnkube-node-646r6\" (UID: \"59b905b8-5ce3-49f6-b47c-b5b3a525fe06\") " pod="openshift-ovn-kubernetes/ovnkube-node-646r6"
May 25 13:05:02.346725 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:05:02.346666    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 13:05:02.346938 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.346740    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6ef8bb8d-98b5-4fe9-94a7-8e06d34ab971-host\") pod \"node-ca-wlq4q\" (UID: \"6ef8bb8d-98b5-4fe9-94a7-8e06d34ab971\") " pod="openshift-image-registry/node-ca-wlq4q"
May 25 13:05:02.346938 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:05:02.346777    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e445efe1-4ac4-4df5-881f-748bcf8c5e0f-metrics-certs podName:e445efe1-4ac4-4df5-881f-748bcf8c5e0f nodeName:}" failed. No retries permitted until 2026-05-25 13:05:02.846714047 +0000 UTC m=+3.039008399 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e445efe1-4ac4-4df5-881f-748bcf8c5e0f-metrics-certs") pod "network-metrics-daemon-svlzs" (UID: "e445efe1-4ac4-4df5-881f-748bcf8c5e0f") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 13:05:02.346938 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.346805    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6ef8bb8d-98b5-4fe9-94a7-8e06d34ab971-serviceca\") pod \"node-ca-wlq4q\" (UID: \"6ef8bb8d-98b5-4fe9-94a7-8e06d34ab971\") " pod="openshift-image-registry/node-ca-wlq4q"
May 25 13:05:02.346938 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.346825    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c749c4da-8f53-4bb5-98c2-280f3cd403af-cni-binary-copy\") pod \"multus-8xdhg\" (UID: \"c749c4da-8f53-4bb5-98c2-280f3cd403af\") " pod="openshift-multus/multus-8xdhg"
May 25 13:05:02.346938 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.346847    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/38b712ff-d8de-43d6-9262-4e460ecf7b32-etc-kubernetes\") pod \"tuned-lpfd5\" (UID: \"38b712ff-d8de-43d6-9262-4e460ecf7b32\") " pod="openshift-cluster-node-tuning-operator/tuned-lpfd5"
May 25 13:05:02.346938 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.346869    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/38b712ff-d8de-43d6-9262-4e460ecf7b32-var-lib-kubelet\") pod \"tuned-lpfd5\" (UID: \"38b712ff-d8de-43d6-9262-4e460ecf7b32\") " pod="openshift-cluster-node-tuning-operator/tuned-lpfd5"
May 25 13:05:02.346938 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.346885    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/38b712ff-d8de-43d6-9262-4e460ecf7b32-tmp\") pod \"tuned-lpfd5\" (UID: \"38b712ff-d8de-43d6-9262-4e460ecf7b32\") " pod="openshift-cluster-node-tuning-operator/tuned-lpfd5"
May 25 13:05:02.346938 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.346906    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/33d153fd-31f6-4481-855e-728e06ce70f5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-hj74k\" (UID: \"33d153fd-31f6-4481-855e-728e06ce70f5\") " pod="openshift-multus/multus-additional-cni-plugins-hj74k"
May 25 13:05:02.346938 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.346935    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-lfgrd\" (UniqueName: \"kubernetes.io/projected/fdd5eb93-deb6-441f-b1fe-3511c4eee77e-kube-api-access-lfgrd\") pod \"network-check-target-25spc\" (UID: \"fdd5eb93-deb6-441f-b1fe-3511c4eee77e\") " pod="openshift-network-diagnostics/network-check-target-25spc"
May 25 13:05:02.347275 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.346959    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5a8fc898-888f-420d-a433-27e831c96721-agent-certs\") pod \"konnectivity-agent-rqkvc\" (UID: \"5a8fc898-888f-420d-a433-27e831c96721\") " pod="kube-system/konnectivity-agent-rqkvc"
May 25 13:05:02.347275 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.346984    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/59b905b8-5ce3-49f6-b47c-b5b3a525fe06-run-systemd\") pod \"ovnkube-node-646r6\" (UID: \"59b905b8-5ce3-49f6-b47c-b5b3a525fe06\") " pod="openshift-ovn-kubernetes/ovnkube-node-646r6"
May 25 13:05:02.347275 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.347005    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/59b905b8-5ce3-49f6-b47c-b5b3a525fe06-etc-openvswitch\") pod \"ovnkube-node-646r6\" (UID: \"59b905b8-5ce3-49f6-b47c-b5b3a525fe06\") " pod="openshift-ovn-kubernetes/ovnkube-node-646r6"
May 25 13:05:02.347275 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.347031    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2prr\" (UniqueName: \"kubernetes.io/projected/47afaed9-536d-4fc6-ab45-e56c19bb2ea6-kube-api-access-q2prr\") pod \"iptables-alerter-nd955\" (UID: \"47afaed9-536d-4fc6-ab45-e56c19bb2ea6\") " pod="openshift-network-operator/iptables-alerter-nd955"
May 25 13:05:02.347275 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.347056    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/33d153fd-31f6-4481-855e-728e06ce70f5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-hj74k\" (UID: \"33d153fd-31f6-4481-855e-728e06ce70f5\") " pod="openshift-multus/multus-additional-cni-plugins-hj74k"
May 25 13:05:02.347275 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.347076    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/38b712ff-d8de-43d6-9262-4e460ecf7b32-sys\") pod \"tuned-lpfd5\" (UID: \"38b712ff-d8de-43d6-9262-4e460ecf7b32\") " pod="openshift-cluster-node-tuning-operator/tuned-lpfd5"
May 25 13:05:02.347275 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.347098    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/37b3e162-412d-4e21-8f2f-bb8398f30435-hosts-file\") pod \"node-resolver-vlk4j\" (UID: \"37b3e162-412d-4e21-8f2f-bb8398f30435\") " pod="openshift-dns/node-resolver-vlk4j"
May 25 13:05:02.347275 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.347124    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/33d153fd-31f6-4481-855e-728e06ce70f5-system-cni-dir\") pod \"multus-additional-cni-plugins-hj74k\" (UID: \"33d153fd-31f6-4481-855e-728e06ce70f5\") " pod="openshift-multus/multus-additional-cni-plugins-hj74k"
May 25 13:05:02.347275 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.347148    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5a8fc898-888f-420d-a433-27e831c96721-konnectivity-ca\") pod \"konnectivity-agent-rqkvc\" (UID: \"5a8fc898-888f-420d-a433-27e831c96721\") " pod="kube-system/konnectivity-agent-rqkvc"
May 25 13:05:02.347275 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.347173    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/59b905b8-5ce3-49f6-b47c-b5b3a525fe06-var-lib-openvswitch\") pod \"ovnkube-node-646r6\" (UID: \"59b905b8-5ce3-49f6-b47c-b5b3a525fe06\") " pod="openshift-ovn-kubernetes/ovnkube-node-646r6"
May 25 13:05:02.347275 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.347199    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c749c4da-8f53-4bb5-98c2-280f3cd403af-host-var-lib-kubelet\") pod \"multus-8xdhg\" (UID: \"c749c4da-8f53-4bb5-98c2-280f3cd403af\") " pod="openshift-multus/multus-8xdhg"
May 25 13:05:02.347275 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.347239    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c749c4da-8f53-4bb5-98c2-280f3cd403af-host-run-multus-certs\") pod \"multus-8xdhg\" (UID: \"c749c4da-8f53-4bb5-98c2-280f3cd403af\") " pod="openshift-multus/multus-8xdhg"
May 25 13:05:02.347275 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.347268    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-wvnnx\" (UniqueName: \"kubernetes.io/projected/33d153fd-31f6-4481-855e-728e06ce70f5-kube-api-access-wvnnx\") pod \"multus-additional-cni-plugins-hj74k\" (UID: \"33d153fd-31f6-4481-855e-728e06ce70f5\") " pod="openshift-multus/multus-additional-cni-plugins-hj74k"
May 25 13:05:02.347774 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.347291    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/59b905b8-5ce3-49f6-b47c-b5b3a525fe06-log-socket\") pod \"ovnkube-node-646r6\" (UID: \"59b905b8-5ce3-49f6-b47c-b5b3a525fe06\") " pod="openshift-ovn-kubernetes/ovnkube-node-646r6"
May 25 13:05:02.347774 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.347315    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/59b905b8-5ce3-49f6-b47c-b5b3a525fe06-ovnkube-config\") pod \"ovnkube-node-646r6\" (UID: \"59b905b8-5ce3-49f6-b47c-b5b3a525fe06\") " pod="openshift-ovn-kubernetes/ovnkube-node-646r6"
May 25 13:05:02.347774 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.347341    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c749c4da-8f53-4bb5-98c2-280f3cd403af-host-run-netns\") pod \"multus-8xdhg\" (UID: \"c749c4da-8f53-4bb5-98c2-280f3cd403af\") " pod="openshift-multus/multus-8xdhg"
May 25 13:05:02.347774 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.347391    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/33d153fd-31f6-4481-855e-728e06ce70f5-cni-binary-copy\") pod \"multus-additional-cni-plugins-hj74k\" (UID: \"33d153fd-31f6-4481-855e-728e06ce70f5\") " pod="openshift-multus/multus-additional-cni-plugins-hj74k"
May 25 13:05:02.347774 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.347418    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c749c4da-8f53-4bb5-98c2-280f3cd403af-multus-cni-dir\") pod \"multus-8xdhg\" (UID: \"c749c4da-8f53-4bb5-98c2-280f3cd403af\") " pod="openshift-multus/multus-8xdhg"
May 25 13:05:02.347774 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.347445    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/33d153fd-31f6-4481-855e-728e06ce70f5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-hj74k\" (UID: \"33d153fd-31f6-4481-855e-728e06ce70f5\") " pod="openshift-multus/multus-additional-cni-plugins-hj74k"
May 25 13:05:02.347774 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.347468    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c749c4da-8f53-4bb5-98c2-280f3cd403af-system-cni-dir\") pod \"multus-8xdhg\" (UID: \"c749c4da-8f53-4bb5-98c2-280f3cd403af\") " pod="openshift-multus/multus-8xdhg"
May 25 13:05:02.347774 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.347488    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c749c4da-8f53-4bb5-98c2-280f3cd403af-hostroot\") pod \"multus-8xdhg\" (UID: \"c749c4da-8f53-4bb5-98c2-280f3cd403af\") " pod="openshift-multus/multus-8xdhg"
May 25 13:05:02.347774 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.347512    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/10056893-23ca-4931-8142-3979d7611233-registration-dir\") pod \"aws-ebs-csi-driver-node-t2jp2\" (UID: \"10056893-23ca-4931-8142-3979d7611233\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t2jp2"
May 25 13:05:02.347774 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.347545    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c749c4da-8f53-4bb5-98c2-280f3cd403af-host-run-k8s-cni-cncf-io\") pod \"multus-8xdhg\" (UID: \"c749c4da-8f53-4bb5-98c2-280f3cd403af\") " pod="openshift-multus/multus-8xdhg"
May 25 13:05:02.347774 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.347572    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c749c4da-8f53-4bb5-98c2-280f3cd403af-multus-socket-dir-parent\") pod \"multus-8xdhg\" (UID: \"c749c4da-8f53-4bb5-98c2-280f3cd403af\") " pod="openshift-multus/multus-8xdhg"
May 25 13:05:02.347774 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.347590    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/33d153fd-31f6-4481-855e-728e06ce70f5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-hj74k\" (UID: \"33d153fd-31f6-4481-855e-728e06ce70f5\") " pod="openshift-multus/multus-additional-cni-plugins-hj74k"
May 25 13:05:02.347774 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.347597    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c749c4da-8f53-4bb5-98c2-280f3cd403af-host-var-lib-cni-bin\") pod \"multus-8xdhg\" (UID: \"c749c4da-8f53-4bb5-98c2-280f3cd403af\") " pod="openshift-multus/multus-8xdhg"
May 25 13:05:02.347774 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.347625    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/59b905b8-5ce3-49f6-b47c-b5b3a525fe06-systemd-units\") pod \"ovnkube-node-646r6\" (UID: \"59b905b8-5ce3-49f6-b47c-b5b3a525fe06\") " pod="openshift-ovn-kubernetes/ovnkube-node-646r6"
May 25 13:05:02.347774 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.347651    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/59b905b8-5ce3-49f6-b47c-b5b3a525fe06-env-overrides\") pod \"ovnkube-node-646r6\" (UID: \"59b905b8-5ce3-49f6-b47c-b5b3a525fe06\") " pod="openshift-ovn-kubernetes/ovnkube-node-646r6"
May 25 13:05:02.347774 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.347675    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-srj9q\" (UniqueName: \"kubernetes.io/projected/59b905b8-5ce3-49f6-b47c-b5b3a525fe06-kube-api-access-srj9q\") pod \"ovnkube-node-646r6\" (UID: \"59b905b8-5ce3-49f6-b47c-b5b3a525fe06\") " pod="openshift-ovn-kubernetes/ovnkube-node-646r6"
May 25 13:05:02.347774 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.347698    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c749c4da-8f53-4bb5-98c2-280f3cd403af-os-release\") pod \"multus-8xdhg\" (UID: \"c749c4da-8f53-4bb5-98c2-280f3cd403af\") " pod="openshift-multus/multus-8xdhg"
May 25 13:05:02.348452 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.347720    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/10056893-23ca-4931-8142-3979d7611233-kubelet-dir\") pod \"aws-ebs-csi-driver-node-t2jp2\" (UID: \"10056893-23ca-4931-8142-3979d7611233\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t2jp2"
May 25 13:05:02.348452 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.347749    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2fp9\" (UniqueName: \"kubernetes.io/projected/10056893-23ca-4931-8142-3979d7611233-kube-api-access-z2fp9\") pod \"aws-ebs-csi-driver-node-t2jp2\" (UID: \"10056893-23ca-4931-8142-3979d7611233\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t2jp2"
May 25 13:05:02.348452 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.347777    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/33d153fd-31f6-4481-855e-728e06ce70f5-os-release\") pod \"multus-additional-cni-plugins-hj74k\" (UID: \"33d153fd-31f6-4481-855e-728e06ce70f5\") " pod="openshift-multus/multus-additional-cni-plugins-hj74k"
May 25 13:05:02.348452 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.347802    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/59b905b8-5ce3-49f6-b47c-b5b3a525fe06-ovn-node-metrics-cert\") pod \"ovnkube-node-646r6\" (UID: \"59b905b8-5ce3-49f6-b47c-b5b3a525fe06\") " pod="openshift-ovn-kubernetes/ovnkube-node-646r6"
May 25 13:05:02.348452 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.347830    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c749c4da-8f53-4bb5-98c2-280f3cd403af-cnibin\") pod \"multus-8xdhg\" (UID: \"c749c4da-8f53-4bb5-98c2-280f3cd403af\") " pod="openshift-multus/multus-8xdhg"
May 25 13:05:02.348452 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.347854    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c749c4da-8f53-4bb5-98c2-280f3cd403af-host-var-lib-cni-multus\") pod \"multus-8xdhg\" (UID: \"c749c4da-8f53-4bb5-98c2-280f3cd403af\") " pod="openshift-multus/multus-8xdhg"
May 25 13:05:02.348452 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.347884    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/38b712ff-d8de-43d6-9262-4e460ecf7b32-etc-sysconfig\") pod \"tuned-lpfd5\" (UID: \"38b712ff-d8de-43d6-9262-4e460ecf7b32\") " pod="openshift-cluster-node-tuning-operator/tuned-lpfd5"
May 25 13:05:02.348452 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.347908    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/38b712ff-d8de-43d6-9262-4e460ecf7b32-etc-sysctl-d\") pod \"tuned-lpfd5\" (UID: \"38b712ff-d8de-43d6-9262-4e460ecf7b32\") " pod="openshift-cluster-node-tuning-operator/tuned-lpfd5"
May 25 13:05:02.348452 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.347931    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/38b712ff-d8de-43d6-9262-4e460ecf7b32-etc-systemd\") pod \"tuned-lpfd5\" (UID: \"38b712ff-d8de-43d6-9262-4e460ecf7b32\") " pod="openshift-cluster-node-tuning-operator/tuned-lpfd5"
May 25 13:05:02.348452 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.347958    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/59b905b8-5ce3-49f6-b47c-b5b3a525fe06-host-kubelet\") pod \"ovnkube-node-646r6\" (UID: \"59b905b8-5ce3-49f6-b47c-b5b3a525fe06\") " pod="openshift-ovn-kubernetes/ovnkube-node-646r6"
May 25 13:05:02.348452 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.347983    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/59b905b8-5ce3-49f6-b47c-b5b3a525fe06-host-cni-bin\") pod \"ovnkube-node-646r6\" (UID: \"59b905b8-5ce3-49f6-b47c-b5b3a525fe06\") " pod="openshift-ovn-kubernetes/ovnkube-node-646r6"
May 25 13:05:02.348452 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.348008    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c749c4da-8f53-4bb5-98c2-280f3cd403af-multus-conf-dir\") pod \"multus-8xdhg\" (UID: \"c749c4da-8f53-4bb5-98c2-280f3cd403af\") " pod="openshift-multus/multus-8xdhg"
May 25 13:05:02.348452 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.348034    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/10056893-23ca-4931-8142-3979d7611233-etc-selinux\") pod \"aws-ebs-csi-driver-node-t2jp2\" (UID: \"10056893-23ca-4931-8142-3979d7611233\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t2jp2"
May 25 13:05:02.348452 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.348056    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/38b712ff-d8de-43d6-9262-4e460ecf7b32-lib-modules\") pod \"tuned-lpfd5\" (UID: \"38b712ff-d8de-43d6-9262-4e460ecf7b32\") " pod="openshift-cluster-node-tuning-operator/tuned-lpfd5"
May 25 13:05:02.348452 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.348079    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/38b712ff-d8de-43d6-9262-4e460ecf7b32-host\") pod \"tuned-lpfd5\" (UID: \"38b712ff-d8de-43d6-9262-4e460ecf7b32\") " pod="openshift-cluster-node-tuning-operator/tuned-lpfd5"
May 25 13:05:02.348452 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.348108    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qg64\" (UniqueName: \"kubernetes.io/projected/38b712ff-d8de-43d6-9262-4e460ecf7b32-kube-api-access-2qg64\") pod \"tuned-lpfd5\" (UID: \"38b712ff-d8de-43d6-9262-4e460ecf7b32\") " pod="openshift-cluster-node-tuning-operator/tuned-lpfd5"
May 25 13:05:02.348452 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.348136    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-5p2f5\" (UniqueName: \"kubernetes.io/projected/e445efe1-4ac4-4df5-881f-748bcf8c5e0f-kube-api-access-5p2f5\") pod \"network-metrics-daemon-svlzs\" (UID: \"e445efe1-4ac4-4df5-881f-748bcf8c5e0f\") " pod="openshift-multus/network-metrics-daemon-svlzs"
May 25 13:05:02.349127 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.348165    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/59b905b8-5ce3-49f6-b47c-b5b3a525fe06-host-run-ovn-kubernetes\") pod \"ovnkube-node-646r6\" (UID: \"59b905b8-5ce3-49f6-b47c-b5b3a525fe06\") " pod="openshift-ovn-kubernetes/ovnkube-node-646r6"
May 25 13:05:02.349127 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.348196    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/59b905b8-5ce3-49f6-b47c-b5b3a525fe06-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-646r6\" (UID: \"59b905b8-5ce3-49f6-b47c-b5b3a525fe06\") " pod="openshift-ovn-kubernetes/ovnkube-node-646r6"
May 25 13:05:02.349127 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.348215    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/33d153fd-31f6-4481-855e-728e06ce70f5-system-cni-dir\") pod \"multus-additional-cni-plugins-hj74k\" (UID: \"33d153fd-31f6-4481-855e-728e06ce70f5\") " pod="openshift-multus/multus-additional-cni-plugins-hj74k"
May 25 13:05:02.349127 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.348224    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-g6t9p\" (UniqueName: \"kubernetes.io/projected/37b3e162-412d-4e21-8f2f-bb8398f30435-kube-api-access-g6t9p\") pod \"node-resolver-vlk4j\" (UID: \"37b3e162-412d-4e21-8f2f-bb8398f30435\") " pod="openshift-dns/node-resolver-vlk4j"
May 25 13:05:02.349127 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.348251    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/33d153fd-31f6-4481-855e-728e06ce70f5-cnibin\") pod \"multus-additional-cni-plugins-hj74k\" (UID: \"33d153fd-31f6-4481-855e-728e06ce70f5\") " pod="openshift-multus/multus-additional-cni-plugins-hj74k"
May 25 13:05:02.349127 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.348279    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/38b712ff-d8de-43d6-9262-4e460ecf7b32-etc-modprobe-d\") pod \"tuned-lpfd5\" (UID: \"38b712ff-d8de-43d6-9262-4e460ecf7b32\") " pod="openshift-cluster-node-tuning-operator/tuned-lpfd5"
May 25 13:05:02.349127 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.348305    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/38b712ff-d8de-43d6-9262-4e460ecf7b32-etc-sysctl-conf\") pod \"tuned-lpfd5\" (UID: \"38b712ff-d8de-43d6-9262-4e460ecf7b32\") " pod="openshift-cluster-node-tuning-operator/tuned-lpfd5"
May 25 13:05:02.349127 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.348330    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/47afaed9-536d-4fc6-ab45-e56c19bb2ea6-iptables-alerter-script\") pod \"iptables-alerter-nd955\" (UID: \"47afaed9-536d-4fc6-ab45-e56c19bb2ea6\") " pod="openshift-network-operator/iptables-alerter-nd955"
May 25 13:05:02.349127 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.348350    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/33d153fd-31f6-4481-855e-728e06ce70f5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-hj74k\" (UID: \"33d153fd-31f6-4481-855e-728e06ce70f5\") " pod="openshift-multus/multus-additional-cni-plugins-hj74k"
May 25 13:05:02.349127 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.348357    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37b3e162-412d-4e21-8f2f-bb8398f30435-tmp-dir\") pod \"node-resolver-vlk4j\" (UID: \"37b3e162-412d-4e21-8f2f-bb8398f30435\") " pod="openshift-dns/node-resolver-vlk4j"
May 25 13:05:02.349127 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.348407    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/59b905b8-5ce3-49f6-b47c-b5b3a525fe06-host-slash\") pod \"ovnkube-node-646r6\" (UID: \"59b905b8-5ce3-49f6-b47c-b5b3a525fe06\") " pod="openshift-ovn-kubernetes/ovnkube-node-646r6"
May 25 13:05:02.349127 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.348426    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/59b905b8-5ce3-49f6-b47c-b5b3a525fe06-var-lib-openvswitch\") pod \"ovnkube-node-646r6\" (UID: \"59b905b8-5ce3-49f6-b47c-b5b3a525fe06\") " pod="openshift-ovn-kubernetes/ovnkube-node-646r6"
May 25 13:05:02.349127 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.348433    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/59b905b8-5ce3-49f6-b47c-b5b3a525fe06-host-run-netns\") pod \"ovnkube-node-646r6\" (UID: \"59b905b8-5ce3-49f6-b47c-b5b3a525fe06\") " pod="openshift-ovn-kubernetes/ovnkube-node-646r6"
May 25 13:05:02.349127 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.348461    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/59b905b8-5ce3-49f6-b47c-b5b3a525fe06-run-openvswitch\") pod \"ovnkube-node-646r6\" (UID: \"59b905b8-5ce3-49f6-b47c-b5b3a525fe06\") " pod="openshift-ovn-kubernetes/ovnkube-node-646r6"
May 25 13:05:02.349127 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.348495    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/59b905b8-5ce3-49f6-b47c-b5b3a525fe06-ovnkube-script-lib\") pod \"ovnkube-node-646r6\" (UID: \"59b905b8-5ce3-49f6-b47c-b5b3a525fe06\") " pod="openshift-ovn-kubernetes/ovnkube-node-646r6"
May 25 13:05:02.349127 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.348523    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c749c4da-8f53-4bb5-98c2-280f3cd403af-etc-kubernetes\") pod \"multus-8xdhg\" (UID: \"c749c4da-8f53-4bb5-98c2-280f3cd403af\") " pod="openshift-multus/multus-8xdhg"
May 25 13:05:02.349127 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.348549    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/47afaed9-536d-4fc6-ab45-e56c19bb2ea6-host-slash\") pod \"iptables-alerter-nd955\" (UID: \"47afaed9-536d-4fc6-ab45-e56c19bb2ea6\") " pod="openshift-network-operator/iptables-alerter-nd955"
May 25 13:05:02.349856 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.348577    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/59b905b8-5ce3-49f6-b47c-b5b3a525fe06-run-ovn\") pod \"ovnkube-node-646r6\" (UID: \"59b905b8-5ce3-49f6-b47c-b5b3a525fe06\") " pod="openshift-ovn-kubernetes/ovnkube-node-646r6"
May 25 13:05:02.349856 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.348604    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jxbs\" (UniqueName: \"kubernetes.io/projected/c749c4da-8f53-4bb5-98c2-280f3cd403af-kube-api-access-4jxbs\") pod \"multus-8xdhg\" (UID: \"c749c4da-8f53-4bb5-98c2-280f3cd403af\") " pod="openshift-multus/multus-8xdhg"
May 25 13:05:02.349856 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.348690    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5a8fc898-888f-420d-a433-27e831c96721-konnectivity-ca\") pod \"konnectivity-agent-rqkvc\" (UID: \"5a8fc898-888f-420d-a433-27e831c96721\") " pod="kube-system/konnectivity-agent-rqkvc"
May 25 13:05:02.349856 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.348717    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/59b905b8-5ce3-49f6-b47c-b5b3a525fe06-host-kubelet\") pod \"ovnkube-node-646r6\" (UID: \"59b905b8-5ce3-49f6-b47c-b5b3a525fe06\") " pod="openshift-ovn-kubernetes/ovnkube-node-646r6"
May 25 13:05:02.349856 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.348816    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/59b905b8-5ce3-49f6-b47c-b5b3a525fe06-systemd-units\") pod \"ovnkube-node-646r6\" (UID: \"59b905b8-5ce3-49f6-b47c-b5b3a525fe06\") " pod="openshift-ovn-kubernetes/ovnkube-node-646r6"
May 25 13:05:02.349856 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.349125    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/33d153fd-31f6-4481-855e-728e06ce70f5-cni-binary-copy\") pod \"multus-additional-cni-plugins-hj74k\" (UID: \"33d153fd-31f6-4481-855e-728e06ce70f5\") " pod="openshift-multus/multus-additional-cni-plugins-hj74k"
May 25 13:05:02.349856 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.349204    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/59b905b8-5ce3-49f6-b47c-b5b3a525fe06-env-overrides\") pod \"ovnkube-node-646r6\" (UID: \"59b905b8-5ce3-49f6-b47c-b5b3a525fe06\") " pod="openshift-ovn-kubernetes/ovnkube-node-646r6"
May 25 13:05:02.349856 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.349212    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/59b905b8-5ce3-49f6-b47c-b5b3a525fe06-ovnkube-config\") pod \"ovnkube-node-646r6\" (UID: \"59b905b8-5ce3-49f6-b47c-b5b3a525fe06\") " pod="openshift-ovn-kubernetes/ovnkube-node-646r6"
May 25 13:05:02.349856 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.349267    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/59b905b8-5ce3-49f6-b47c-b5b3a525fe06-host-cni-bin\") pod \"ovnkube-node-646r6\" (UID: \"59b905b8-5ce3-49f6-b47c-b5b3a525fe06\") " pod="openshift-ovn-kubernetes/ovnkube-node-646r6"
May 25 13:05:02.349856 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.349527    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/33d153fd-31f6-4481-855e-728e06ce70f5-os-release\") pod \"multus-additional-cni-plugins-hj74k\" (UID: \"33d153fd-31f6-4481-855e-728e06ce70f5\") " pod="openshift-multus/multus-additional-cni-plugins-hj74k"
May 25 13:05:02.349856 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.349563    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/59b905b8-5ce3-49f6-b47c-b5b3a525fe06-run-systemd\") pod \"ovnkube-node-646r6\" (UID: \"59b905b8-5ce3-49f6-b47c-b5b3a525fe06\") " pod="openshift-ovn-kubernetes/ovnkube-node-646r6"
May 25 13:05:02.349856 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.349573    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/33d153fd-31f6-4481-855e-728e06ce70f5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-hj74k\" (UID: \"33d153fd-31f6-4481-855e-728e06ce70f5\") " pod="openshift-multus/multus-additional-cni-plugins-hj74k"
May 25 13:05:02.349856 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.348162    2580 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
May 25 13:05:02.349856 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.349611    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/59b905b8-5ce3-49f6-b47c-b5b3a525fe06-host-run-ovn-kubernetes\") pod \"ovnkube-node-646r6\" (UID: \"59b905b8-5ce3-49f6-b47c-b5b3a525fe06\") " pod="openshift-ovn-kubernetes/ovnkube-node-646r6"
May 25 13:05:02.350432 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.349930    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37b3e162-412d-4e21-8f2f-bb8398f30435-tmp-dir\") pod \"node-resolver-vlk4j\" (UID: \"37b3e162-412d-4e21-8f2f-bb8398f30435\") " pod="openshift-dns/node-resolver-vlk4j"
May 25 13:05:02.350432 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.350022    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/59b905b8-5ce3-49f6-b47c-b5b3a525fe06-host-slash\") pod \"ovnkube-node-646r6\" (UID: \"59b905b8-5ce3-49f6-b47c-b5b3a525fe06\") " pod="openshift-ovn-kubernetes/ovnkube-node-646r6"
May 25 13:05:02.350432 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.350066    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/59b905b8-5ce3-49f6-b47c-b5b3a525fe06-host-run-netns\") pod \"ovnkube-node-646r6\" (UID: \"59b905b8-5ce3-49f6-b47c-b5b3a525fe06\") " pod="openshift-ovn-kubernetes/ovnkube-node-646r6"
May 25 13:05:02.350432 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.350105    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/59b905b8-5ce3-49f6-b47c-b5b3a525fe06-run-openvswitch\") pod \"ovnkube-node-646r6\" (UID: \"59b905b8-5ce3-49f6-b47c-b5b3a525fe06\") " pod="openshift-ovn-kubernetes/ovnkube-node-646r6"
May 25 13:05:02.350591 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.350512    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/59b905b8-5ce3-49f6-b47c-b5b3a525fe06-ovnkube-script-lib\") pod \"ovnkube-node-646r6\" (UID: \"59b905b8-5ce3-49f6-b47c-b5b3a525fe06\") " pod="openshift-ovn-kubernetes/ovnkube-node-646r6"
May 25 13:05:02.350810 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.350587    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/59b905b8-5ce3-49f6-b47c-b5b3a525fe06-run-ovn\") pod \"ovnkube-node-646r6\" (UID: \"59b905b8-5ce3-49f6-b47c-b5b3a525fe06\") " pod="openshift-ovn-kubernetes/ovnkube-node-646r6"
May 25 13:05:02.350810 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.350740    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/59b905b8-5ce3-49f6-b47c-b5b3a525fe06-log-socket\") pod \"ovnkube-node-646r6\" (UID: \"59b905b8-5ce3-49f6-b47c-b5b3a525fe06\") " pod="openshift-ovn-kubernetes/ovnkube-node-646r6"
May 25 13:05:02.351142 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.351073    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/33d153fd-31f6-4481-855e-728e06ce70f5-cnibin\") pod \"multus-additional-cni-plugins-hj74k\" (UID: \"33d153fd-31f6-4481-855e-728e06ce70f5\") " pod="openshift-multus/multus-additional-cni-plugins-hj74k"
May 25 13:05:02.351142 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.351092    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/59b905b8-5ce3-49f6-b47c-b5b3a525fe06-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-646r6\" (UID: \"59b905b8-5ce3-49f6-b47c-b5b3a525fe06\") " pod="openshift-ovn-kubernetes/ovnkube-node-646r6"
May 25 13:05:02.351142 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.348196    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/59b905b8-5ce3-49f6-b47c-b5b3a525fe06-etc-openvswitch\") pod \"ovnkube-node-646r6\" (UID: \"59b905b8-5ce3-49f6-b47c-b5b3a525fe06\") " pod="openshift-ovn-kubernetes/ovnkube-node-646r6"
May 25 13:05:02.351347 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.351158    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/37b3e162-412d-4e21-8f2f-bb8398f30435-hosts-file\") pod \"node-resolver-vlk4j\" (UID: \"37b3e162-412d-4e21-8f2f-bb8398f30435\") " pod="openshift-dns/node-resolver-vlk4j"
May 25 13:05:02.352944 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:05:02.352913    2580 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 25 13:05:02.352944 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:05:02.352938    2580 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 25 13:05:02.353098 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:05:02.352951    2580 projected.go:194] Error preparing data for projected volume kube-api-access-lfgrd for pod openshift-network-diagnostics/network-check-target-25spc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 13:05:02.353098 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:05:02.353018    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fdd5eb93-deb6-441f-b1fe-3511c4eee77e-kube-api-access-lfgrd podName:fdd5eb93-deb6-441f-b1fe-3511c4eee77e nodeName:}" failed. No retries permitted until 2026-05-25 13:05:02.853001538 +0000 UTC m=+3.045295891 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-lfgrd" (UniqueName: "kubernetes.io/projected/fdd5eb93-deb6-441f-b1fe-3511c4eee77e-kube-api-access-lfgrd") pod "network-check-target-25spc" (UID: "fdd5eb93-deb6-441f-b1fe-3511c4eee77e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 13:05:02.353098 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.353051    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/59b905b8-5ce3-49f6-b47c-b5b3a525fe06-ovn-node-metrics-cert\") pod \"ovnkube-node-646r6\" (UID: \"59b905b8-5ce3-49f6-b47c-b5b3a525fe06\") " pod="openshift-ovn-kubernetes/ovnkube-node-646r6"
May 25 13:05:02.353256 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.353189    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5a8fc898-888f-420d-a433-27e831c96721-agent-certs\") pod \"konnectivity-agent-rqkvc\" (UID: \"5a8fc898-888f-420d-a433-27e831c96721\") " pod="kube-system/konnectivity-agent-rqkvc"
May 25 13:05:02.355301 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.355274    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvnnx\" (UniqueName: \"kubernetes.io/projected/33d153fd-31f6-4481-855e-728e06ce70f5-kube-api-access-wvnnx\") pod \"multus-additional-cni-plugins-hj74k\" (UID: \"33d153fd-31f6-4481-855e-728e06ce70f5\") " pod="openshift-multus/multus-additional-cni-plugins-hj74k"
May 25 13:05:02.356154 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.356134    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5p2f5\" (UniqueName: \"kubernetes.io/projected/e445efe1-4ac4-4df5-881f-748bcf8c5e0f-kube-api-access-5p2f5\") pod \"network-metrics-daemon-svlzs\" (UID: \"e445efe1-4ac4-4df5-881f-748bcf8c5e0f\") " pod="openshift-multus/network-metrics-daemon-svlzs"
May 25 13:05:02.356269 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.356257    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-srj9q\" (UniqueName: \"kubernetes.io/projected/59b905b8-5ce3-49f6-b47c-b5b3a525fe06-kube-api-access-srj9q\") pod \"ovnkube-node-646r6\" (UID: \"59b905b8-5ce3-49f6-b47c-b5b3a525fe06\") " pod="openshift-ovn-kubernetes/ovnkube-node-646r6"
May 25 13:05:02.364611 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.364589    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6t9p\" (UniqueName: \"kubernetes.io/projected/37b3e162-412d-4e21-8f2f-bb8398f30435-kube-api-access-g6t9p\") pod \"node-resolver-vlk4j\" (UID: \"37b3e162-412d-4e21-8f2f-bb8398f30435\") " pod="openshift-dns/node-resolver-vlk4j"
May 25 13:05:02.449032 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.448999    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/38b712ff-d8de-43d6-9262-4e460ecf7b32-sys\") pod \"tuned-lpfd5\" (UID: \"38b712ff-d8de-43d6-9262-4e460ecf7b32\") " pod="openshift-cluster-node-tuning-operator/tuned-lpfd5"
May 25 13:05:02.449032 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.449036    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c749c4da-8f53-4bb5-98c2-280f3cd403af-host-var-lib-kubelet\") pod \"multus-8xdhg\" (UID: \"c749c4da-8f53-4bb5-98c2-280f3cd403af\") " pod="openshift-multus/multus-8xdhg"
May 25 13:05:02.449032 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.449052    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c749c4da-8f53-4bb5-98c2-280f3cd403af-host-run-multus-certs\") pod \"multus-8xdhg\" (UID: \"c749c4da-8f53-4bb5-98c2-280f3cd403af\") " pod="openshift-multus/multus-8xdhg"
May 25 13:05:02.449288 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.449077    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c749c4da-8f53-4bb5-98c2-280f3cd403af-host-run-netns\") pod \"multus-8xdhg\" (UID: \"c749c4da-8f53-4bb5-98c2-280f3cd403af\") " pod="openshift-multus/multus-8xdhg"
May 25 13:05:02.449288 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.449100    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c749c4da-8f53-4bb5-98c2-280f3cd403af-multus-cni-dir\") pod \"multus-8xdhg\" (UID: \"c749c4da-8f53-4bb5-98c2-280f3cd403af\") " pod="openshift-multus/multus-8xdhg"
May 25 13:05:02.449288 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.449126    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c749c4da-8f53-4bb5-98c2-280f3cd403af-system-cni-dir\") pod \"multus-8xdhg\" (UID: \"c749c4da-8f53-4bb5-98c2-280f3cd403af\") " pod="openshift-multus/multus-8xdhg"
May 25 13:05:02.449288 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.449131    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/38b712ff-d8de-43d6-9262-4e460ecf7b32-sys\") pod \"tuned-lpfd5\" (UID: \"38b712ff-d8de-43d6-9262-4e460ecf7b32\") " pod="openshift-cluster-node-tuning-operator/tuned-lpfd5"
May 25 13:05:02.449288 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.449138    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c749c4da-8f53-4bb5-98c2-280f3cd403af-host-var-lib-kubelet\") pod \"multus-8xdhg\" (UID: \"c749c4da-8f53-4bb5-98c2-280f3cd403af\") " pod="openshift-multus/multus-8xdhg"
May 25 13:05:02.449288 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.449146    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c749c4da-8f53-4bb5-98c2-280f3cd403af-host-run-netns\") pod \"multus-8xdhg\" (UID: \"c749c4da-8f53-4bb5-98c2-280f3cd403af\") " pod="openshift-multus/multus-8xdhg"
May 25 13:05:02.449288 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.449149    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c749c4da-8f53-4bb5-98c2-280f3cd403af-hostroot\") pod \"multus-8xdhg\" (UID: \"c749c4da-8f53-4bb5-98c2-280f3cd403af\") " pod="openshift-multus/multus-8xdhg"
May 25 13:05:02.449288 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.449184    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c749c4da-8f53-4bb5-98c2-280f3cd403af-hostroot\") pod \"multus-8xdhg\" (UID: \"c749c4da-8f53-4bb5-98c2-280f3cd403af\") " pod="openshift-multus/multus-8xdhg"
May 25 13:05:02.449288 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.449192    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c749c4da-8f53-4bb5-98c2-280f3cd403af-multus-cni-dir\") pod \"multus-8xdhg\" (UID: \"c749c4da-8f53-4bb5-98c2-280f3cd403af\") " pod="openshift-multus/multus-8xdhg"
May 25 13:05:02.449288 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.449210    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/10056893-23ca-4931-8142-3979d7611233-registration-dir\") pod \"aws-ebs-csi-driver-node-t2jp2\" (UID: \"10056893-23ca-4931-8142-3979d7611233\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t2jp2"
May 25 13:05:02.449288 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.449199    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c749c4da-8f53-4bb5-98c2-280f3cd403af-system-cni-dir\") pod \"multus-8xdhg\" (UID: \"c749c4da-8f53-4bb5-98c2-280f3cd403af\") " pod="openshift-multus/multus-8xdhg"
May 25 13:05:02.449288 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.449146    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c749c4da-8f53-4bb5-98c2-280f3cd403af-host-run-multus-certs\") pod \"multus-8xdhg\" (UID: \"c749c4da-8f53-4bb5-98c2-280f3cd403af\") " pod="openshift-multus/multus-8xdhg"
May 25 13:05:02.449288 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.449234    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c749c4da-8f53-4bb5-98c2-280f3cd403af-host-run-k8s-cni-cncf-io\") pod \"multus-8xdhg\" (UID: \"c749c4da-8f53-4bb5-98c2-280f3cd403af\") " pod="openshift-multus/multus-8xdhg"
May 25 13:05:02.449288 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.449255    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c749c4da-8f53-4bb5-98c2-280f3cd403af-host-run-k8s-cni-cncf-io\") pod \"multus-8xdhg\" (UID: \"c749c4da-8f53-4bb5-98c2-280f3cd403af\") " pod="openshift-multus/multus-8xdhg"
May 25 13:05:02.449288 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.449265    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c749c4da-8f53-4bb5-98c2-280f3cd403af-multus-socket-dir-parent\") pod \"multus-8xdhg\" (UID: \"c749c4da-8f53-4bb5-98c2-280f3cd403af\") " pod="openshift-multus/multus-8xdhg"
May 25 13:05:02.449288 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.449273    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/10056893-23ca-4931-8142-3979d7611233-registration-dir\") pod \"aws-ebs-csi-driver-node-t2jp2\" (UID: \"10056893-23ca-4931-8142-3979d7611233\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t2jp2"
May 25 13:05:02.449288 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.449290    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c749c4da-8f53-4bb5-98c2-280f3cd403af-host-var-lib-cni-bin\") pod \"multus-8xdhg\" (UID: \"c749c4da-8f53-4bb5-98c2-280f3cd403af\") " pod="openshift-multus/multus-8xdhg"
May 25 13:05:02.449971 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.449309    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c749c4da-8f53-4bb5-98c2-280f3cd403af-os-release\") pod \"multus-8xdhg\" (UID: \"c749c4da-8f53-4bb5-98c2-280f3cd403af\") " pod="openshift-multus/multus-8xdhg"
May 25 13:05:02.449971 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.449327    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/10056893-23ca-4931-8142-3979d7611233-kubelet-dir\") pod \"aws-ebs-csi-driver-node-t2jp2\" (UID: \"10056893-23ca-4931-8142-3979d7611233\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t2jp2"
May 25 13:05:02.449971 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.449340    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c749c4da-8f53-4bb5-98c2-280f3cd403af-host-var-lib-cni-bin\") pod \"multus-8xdhg\" (UID: \"c749c4da-8f53-4bb5-98c2-280f3cd403af\") " pod="openshift-multus/multus-8xdhg"
May 25 13:05:02.449971 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.449349    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-z2fp9\" (UniqueName: \"kubernetes.io/projected/10056893-23ca-4931-8142-3979d7611233-kube-api-access-z2fp9\") pod \"aws-ebs-csi-driver-node-t2jp2\" (UID: \"10056893-23ca-4931-8142-3979d7611233\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t2jp2"
May 25 13:05:02.449971 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.449360    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c749c4da-8f53-4bb5-98c2-280f3cd403af-os-release\") pod \"multus-8xdhg\" (UID: \"c749c4da-8f53-4bb5-98c2-280f3cd403af\") " pod="openshift-multus/multus-8xdhg"
May 25 13:05:02.449971 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.449386    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c749c4da-8f53-4bb5-98c2-280f3cd403af-cnibin\") pod \"multus-8xdhg\" (UID: \"c749c4da-8f53-4bb5-98c2-280f3cd403af\") " pod="openshift-multus/multus-8xdhg"
May 25 13:05:02.449971 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.449389    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c749c4da-8f53-4bb5-98c2-280f3cd403af-multus-socket-dir-parent\") pod \"multus-8xdhg\" (UID: \"c749c4da-8f53-4bb5-98c2-280f3cd403af\") " pod="openshift-multus/multus-8xdhg"
May 25 13:05:02.449971 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.449410    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c749c4da-8f53-4bb5-98c2-280f3cd403af-host-var-lib-cni-multus\") pod \"multus-8xdhg\" (UID: \"c749c4da-8f53-4bb5-98c2-280f3cd403af\") " pod="openshift-multus/multus-8xdhg"
May 25 13:05:02.449971 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.449431    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c749c4da-8f53-4bb5-98c2-280f3cd403af-cnibin\") pod \"multus-8xdhg\" (UID: \"c749c4da-8f53-4bb5-98c2-280f3cd403af\") " pod="openshift-multus/multus-8xdhg"
May 25 13:05:02.449971 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.449435    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/38b712ff-d8de-43d6-9262-4e460ecf7b32-etc-sysconfig\") pod \"tuned-lpfd5\" (UID: \"38b712ff-d8de-43d6-9262-4e460ecf7b32\") " pod="openshift-cluster-node-tuning-operator/tuned-lpfd5"
May 25 13:05:02.449971 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.449458    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/38b712ff-d8de-43d6-9262-4e460ecf7b32-etc-sysctl-d\") pod \"tuned-lpfd5\" (UID: \"38b712ff-d8de-43d6-9262-4e460ecf7b32\") " pod="openshift-cluster-node-tuning-operator/tuned-lpfd5"
May 25 13:05:02.449971 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.449433    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c749c4da-8f53-4bb5-98c2-280f3cd403af-host-var-lib-cni-multus\") pod \"multus-8xdhg\" (UID: \"c749c4da-8f53-4bb5-98c2-280f3cd403af\") " pod="openshift-multus/multus-8xdhg"
May 25 13:05:02.449971 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.449413    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/10056893-23ca-4931-8142-3979d7611233-kubelet-dir\") pod \"aws-ebs-csi-driver-node-t2jp2\" (UID: \"10056893-23ca-4931-8142-3979d7611233\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t2jp2"
May 25 13:05:02.449971 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.449483    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/38b712ff-d8de-43d6-9262-4e460ecf7b32-etc-systemd\") pod \"tuned-lpfd5\" (UID: \"38b712ff-d8de-43d6-9262-4e460ecf7b32\") " pod="openshift-cluster-node-tuning-operator/tuned-lpfd5"
May 25 13:05:02.449971 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.449523    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c749c4da-8f53-4bb5-98c2-280f3cd403af-multus-conf-dir\") pod \"multus-8xdhg\" (UID: \"c749c4da-8f53-4bb5-98c2-280f3cd403af\") " pod="openshift-multus/multus-8xdhg"
May 25 13:05:02.449971 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.449523    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/38b712ff-d8de-43d6-9262-4e460ecf7b32-etc-sysconfig\") pod \"tuned-lpfd5\" (UID: \"38b712ff-d8de-43d6-9262-4e460ecf7b32\") " pod="openshift-cluster-node-tuning-operator/tuned-lpfd5"
May 25 13:05:02.449971 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.449544    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/10056893-23ca-4931-8142-3979d7611233-etc-selinux\") pod \"aws-ebs-csi-driver-node-t2jp2\" (UID: \"10056893-23ca-4931-8142-3979d7611233\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t2jp2"
May 25 13:05:02.449971 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.449548    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/38b712ff-d8de-43d6-9262-4e460ecf7b32-etc-systemd\") pod \"tuned-lpfd5\" (UID: \"38b712ff-d8de-43d6-9262-4e460ecf7b32\") " pod="openshift-cluster-node-tuning-operator/tuned-lpfd5"
May 25 13:05:02.450745 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.449565    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c749c4da-8f53-4bb5-98c2-280f3cd403af-multus-conf-dir\") pod \"multus-8xdhg\" (UID: \"c749c4da-8f53-4bb5-98c2-280f3cd403af\") " pod="openshift-multus/multus-8xdhg"
May 25 13:05:02.450745 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.449569    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/38b712ff-d8de-43d6-9262-4e460ecf7b32-lib-modules\") pod \"tuned-lpfd5\" (UID: \"38b712ff-d8de-43d6-9262-4e460ecf7b32\") " pod="openshift-cluster-node-tuning-operator/tuned-lpfd5"
May 25 13:05:02.450745 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.449580    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/38b712ff-d8de-43d6-9262-4e460ecf7b32-etc-sysctl-d\") pod \"tuned-lpfd5\" (UID: \"38b712ff-d8de-43d6-9262-4e460ecf7b32\") " pod="openshift-cluster-node-tuning-operator/tuned-lpfd5"
May 25 13:05:02.450745 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.449592    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/38b712ff-d8de-43d6-9262-4e460ecf7b32-host\") pod \"tuned-lpfd5\" (UID: \"38b712ff-d8de-43d6-9262-4e460ecf7b32\") " pod="openshift-cluster-node-tuning-operator/tuned-lpfd5"
May 25 13:05:02.450745 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.449615    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-2qg64\" (UniqueName: \"kubernetes.io/projected/38b712ff-d8de-43d6-9262-4e460ecf7b32-kube-api-access-2qg64\") pod \"tuned-lpfd5\" (UID: \"38b712ff-d8de-43d6-9262-4e460ecf7b32\") " pod="openshift-cluster-node-tuning-operator/tuned-lpfd5"
May 25 13:05:02.450745 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.449619    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/38b712ff-d8de-43d6-9262-4e460ecf7b32-host\") pod \"tuned-lpfd5\" (UID: \"38b712ff-d8de-43d6-9262-4e460ecf7b32\") " pod="openshift-cluster-node-tuning-operator/tuned-lpfd5"
May 25 13:05:02.450745 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.449595    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/10056893-23ca-4931-8142-3979d7611233-etc-selinux\") pod \"aws-ebs-csi-driver-node-t2jp2\" (UID: \"10056893-23ca-4931-8142-3979d7611233\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t2jp2"
May 25 13:05:02.450745 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.449641    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/38b712ff-d8de-43d6-9262-4e460ecf7b32-lib-modules\") pod \"tuned-lpfd5\" (UID: \"38b712ff-d8de-43d6-9262-4e460ecf7b32\") " pod="openshift-cluster-node-tuning-operator/tuned-lpfd5"
May 25 13:05:02.450745 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.449642    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/38b712ff-d8de-43d6-9262-4e460ecf7b32-etc-modprobe-d\") pod \"tuned-lpfd5\" (UID: \"38b712ff-d8de-43d6-9262-4e460ecf7b32\") " pod="openshift-cluster-node-tuning-operator/tuned-lpfd5"
May 25 13:05:02.450745 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.449688    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/38b712ff-d8de-43d6-9262-4e460ecf7b32-etc-sysctl-conf\") pod \"tuned-lpfd5\" (UID: \"38b712ff-d8de-43d6-9262-4e460ecf7b32\") " pod="openshift-cluster-node-tuning-operator/tuned-lpfd5"
May 25 13:05:02.450745 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.449713    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/47afaed9-536d-4fc6-ab45-e56c19bb2ea6-iptables-alerter-script\") pod \"iptables-alerter-nd955\" (UID: \"47afaed9-536d-4fc6-ab45-e56c19bb2ea6\") " pod="openshift-network-operator/iptables-alerter-nd955"
May 25 13:05:02.450745 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.449722    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/38b712ff-d8de-43d6-9262-4e460ecf7b32-etc-modprobe-d\") pod \"tuned-lpfd5\" (UID: \"38b712ff-d8de-43d6-9262-4e460ecf7b32\") " pod="openshift-cluster-node-tuning-operator/tuned-lpfd5"
May 25 13:05:02.450745 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.449748    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c749c4da-8f53-4bb5-98c2-280f3cd403af-etc-kubernetes\") pod \"multus-8xdhg\" (UID: \"c749c4da-8f53-4bb5-98c2-280f3cd403af\") " pod="openshift-multus/multus-8xdhg"
May 25 13:05:02.450745 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.449772    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/47afaed9-536d-4fc6-ab45-e56c19bb2ea6-host-slash\") pod \"iptables-alerter-nd955\" (UID: \"47afaed9-536d-4fc6-ab45-e56c19bb2ea6\") " pod="openshift-network-operator/iptables-alerter-nd955"
May 25 13:05:02.450745 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.449806    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-4jxbs\" (UniqueName: \"kubernetes.io/projected/c749c4da-8f53-4bb5-98c2-280f3cd403af-kube-api-access-4jxbs\") pod \"multus-8xdhg\" (UID: \"c749c4da-8f53-4bb5-98c2-280f3cd403af\") " pod="openshift-multus/multus-8xdhg"
May 25 13:05:02.450745 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.449830    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/38b712ff-d8de-43d6-9262-4e460ecf7b32-etc-tuned\") pod \"tuned-lpfd5\" (UID: \"38b712ff-d8de-43d6-9262-4e460ecf7b32\") " pod="openshift-cluster-node-tuning-operator/tuned-lpfd5"
May 25 13:05:02.450745 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.449855    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-8pwr5\" (UniqueName: \"kubernetes.io/projected/6ef8bb8d-98b5-4fe9-94a7-8e06d34ab971-kube-api-access-8pwr5\") pod \"node-ca-wlq4q\" (UID: \"6ef8bb8d-98b5-4fe9-94a7-8e06d34ab971\") " pod="openshift-image-registry/node-ca-wlq4q"
May 25 13:05:02.450745 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.449773    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/38b712ff-d8de-43d6-9262-4e460ecf7b32-etc-sysctl-conf\") pod \"tuned-lpfd5\" (UID: \"38b712ff-d8de-43d6-9262-4e460ecf7b32\") " pod="openshift-cluster-node-tuning-operator/tuned-lpfd5"
May 25 13:05:02.451500 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.449804    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c749c4da-8f53-4bb5-98c2-280f3cd403af-etc-kubernetes\") pod \"multus-8xdhg\" (UID: \"c749c4da-8f53-4bb5-98c2-280f3cd403af\") " pod="openshift-multus/multus-8xdhg"
May 25 13:05:02.451500 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.449809    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/47afaed9-536d-4fc6-ab45-e56c19bb2ea6-host-slash\") pod \"iptables-alerter-nd955\" (UID: \"47afaed9-536d-4fc6-ab45-e56c19bb2ea6\") " pod="openshift-network-operator/iptables-alerter-nd955"
May 25 13:05:02.451500 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.449881    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c749c4da-8f53-4bb5-98c2-280f3cd403af-multus-daemon-config\") pod \"multus-8xdhg\" (UID: \"c749c4da-8f53-4bb5-98c2-280f3cd403af\") " pod="openshift-multus/multus-8xdhg"
May 25 13:05:02.451500 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.449922    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/10056893-23ca-4931-8142-3979d7611233-socket-dir\") pod \"aws-ebs-csi-driver-node-t2jp2\" (UID: \"10056893-23ca-4931-8142-3979d7611233\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t2jp2"
May 25 13:05:02.451500 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.449959    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/10056893-23ca-4931-8142-3979d7611233-device-dir\") pod \"aws-ebs-csi-driver-node-t2jp2\" (UID: \"10056893-23ca-4931-8142-3979d7611233\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t2jp2"
May 25 13:05:02.451500 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.449982    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/10056893-23ca-4931-8142-3979d7611233-sys-fs\") pod \"aws-ebs-csi-driver-node-t2jp2\" (UID: \"10056893-23ca-4931-8142-3979d7611233\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t2jp2"
May 25 13:05:02.451500 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.450004    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/38b712ff-d8de-43d6-9262-4e460ecf7b32-run\") pod \"tuned-lpfd5\" (UID: \"38b712ff-d8de-43d6-9262-4e460ecf7b32\") " pod="openshift-cluster-node-tuning-operator/tuned-lpfd5"
May 25 13:05:02.451500 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.450037    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/10056893-23ca-4931-8142-3979d7611233-device-dir\") pod \"aws-ebs-csi-driver-node-t2jp2\" (UID: \"10056893-23ca-4931-8142-3979d7611233\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t2jp2"
May 25 13:05:02.451500 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.450043    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6ef8bb8d-98b5-4fe9-94a7-8e06d34ab971-host\") pod \"node-ca-wlq4q\" (UID: \"6ef8bb8d-98b5-4fe9-94a7-8e06d34ab971\") " pod="openshift-image-registry/node-ca-wlq4q"
May 25 13:05:02.451500 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.450074    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6ef8bb8d-98b5-4fe9-94a7-8e06d34ab971-host\") pod \"node-ca-wlq4q\" (UID: \"6ef8bb8d-98b5-4fe9-94a7-8e06d34ab971\") " pod="openshift-image-registry/node-ca-wlq4q"
May 25 13:05:02.451500 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.450080    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6ef8bb8d-98b5-4fe9-94a7-8e06d34ab971-serviceca\") pod \"node-ca-wlq4q\" (UID: \"6ef8bb8d-98b5-4fe9-94a7-8e06d34ab971\") " pod="openshift-image-registry/node-ca-wlq4q"
May 25 13:05:02.451500 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.450109    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c749c4da-8f53-4bb5-98c2-280f3cd403af-cni-binary-copy\") pod \"multus-8xdhg\" (UID: \"c749c4da-8f53-4bb5-98c2-280f3cd403af\") " pod="openshift-multus/multus-8xdhg"
May 25 13:05:02.451500 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.450134    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/38b712ff-d8de-43d6-9262-4e460ecf7b32-etc-kubernetes\") pod \"tuned-lpfd5\" (UID: \"38b712ff-d8de-43d6-9262-4e460ecf7b32\") " pod="openshift-cluster-node-tuning-operator/tuned-lpfd5"
May 25 13:05:02.451500 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.450157    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/38b712ff-d8de-43d6-9262-4e460ecf7b32-var-lib-kubelet\") pod \"tuned-lpfd5\" (UID: \"38b712ff-d8de-43d6-9262-4e460ecf7b32\") " pod="openshift-cluster-node-tuning-operator/tuned-lpfd5"
May 25 13:05:02.451500 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.450156    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/10056893-23ca-4931-8142-3979d7611233-socket-dir\") pod \"aws-ebs-csi-driver-node-t2jp2\" (UID: \"10056893-23ca-4931-8142-3979d7611233\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t2jp2"
May 25 13:05:02.451500 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.450181    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/38b712ff-d8de-43d6-9262-4e460ecf7b32-tmp\") pod \"tuned-lpfd5\" (UID: \"38b712ff-d8de-43d6-9262-4e460ecf7b32\") " pod="openshift-cluster-node-tuning-operator/tuned-lpfd5"
May 25 13:05:02.451500 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.450217    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/10056893-23ca-4931-8142-3979d7611233-sys-fs\") pod \"aws-ebs-csi-driver-node-t2jp2\" (UID: \"10056893-23ca-4931-8142-3979d7611233\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t2jp2"
May 25 13:05:02.451500 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.450229    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-q2prr\" (UniqueName: \"kubernetes.io/projected/47afaed9-536d-4fc6-ab45-e56c19bb2ea6-kube-api-access-q2prr\") pod \"iptables-alerter-nd955\" (UID: \"47afaed9-536d-4fc6-ab45-e56c19bb2ea6\") " pod="openshift-network-operator/iptables-alerter-nd955"
May 25 13:05:02.452048 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.450247    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/47afaed9-536d-4fc6-ab45-e56c19bb2ea6-iptables-alerter-script\") pod \"iptables-alerter-nd955\" (UID: \"47afaed9-536d-4fc6-ab45-e56c19bb2ea6\") " pod="openshift-network-operator/iptables-alerter-nd955"
May 25 13:05:02.452048 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.450261    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/38b712ff-d8de-43d6-9262-4e460ecf7b32-etc-kubernetes\") pod \"tuned-lpfd5\" (UID: \"38b712ff-d8de-43d6-9262-4e460ecf7b32\") " pod="openshift-cluster-node-tuning-operator/tuned-lpfd5"
May 25 13:05:02.452048 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.450307    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/38b712ff-d8de-43d6-9262-4e460ecf7b32-run\") pod \"tuned-lpfd5\" (UID: \"38b712ff-d8de-43d6-9262-4e460ecf7b32\") " pod="openshift-cluster-node-tuning-operator/tuned-lpfd5"
May 25 13:05:02.452048 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.450335    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/38b712ff-d8de-43d6-9262-4e460ecf7b32-var-lib-kubelet\") pod \"tuned-lpfd5\" (UID: \"38b712ff-d8de-43d6-9262-4e460ecf7b32\") " pod="openshift-cluster-node-tuning-operator/tuned-lpfd5"
May 25 13:05:02.452048 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.450447    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c749c4da-8f53-4bb5-98c2-280f3cd403af-multus-daemon-config\") pod \"multus-8xdhg\" (UID: \"c749c4da-8f53-4bb5-98c2-280f3cd403af\") " pod="openshift-multus/multus-8xdhg"
May 25 13:05:02.452048 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.450509    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6ef8bb8d-98b5-4fe9-94a7-8e06d34ab971-serviceca\") pod \"node-ca-wlq4q\" (UID: \"6ef8bb8d-98b5-4fe9-94a7-8e06d34ab971\") " pod="openshift-image-registry/node-ca-wlq4q"
May 25 13:05:02.452048 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.450769    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c749c4da-8f53-4bb5-98c2-280f3cd403af-cni-binary-copy\") pod \"multus-8xdhg\" (UID: \"c749c4da-8f53-4bb5-98c2-280f3cd403af\") " pod="openshift-multus/multus-8xdhg"
May 25 13:05:02.452815 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.452790    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/38b712ff-d8de-43d6-9262-4e460ecf7b32-tmp\") pod \"tuned-lpfd5\" (UID: \"38b712ff-d8de-43d6-9262-4e460ecf7b32\") " pod="openshift-cluster-node-tuning-operator/tuned-lpfd5"
May 25 13:05:02.452921 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.452848    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/38b712ff-d8de-43d6-9262-4e460ecf7b32-etc-tuned\") pod \"tuned-lpfd5\" (UID: \"38b712ff-d8de-43d6-9262-4e460ecf7b32\") " pod="openshift-cluster-node-tuning-operator/tuned-lpfd5"
May 25 13:05:02.457739 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.457715    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2prr\" (UniqueName: \"kubernetes.io/projected/47afaed9-536d-4fc6-ab45-e56c19bb2ea6-kube-api-access-q2prr\") pod \"iptables-alerter-nd955\" (UID: \"47afaed9-536d-4fc6-ab45-e56c19bb2ea6\") " pod="openshift-network-operator/iptables-alerter-nd955"
May 25 13:05:02.458153 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.458132    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qg64\" (UniqueName: \"kubernetes.io/projected/38b712ff-d8de-43d6-9262-4e460ecf7b32-kube-api-access-2qg64\") pod \"tuned-lpfd5\" (UID: \"38b712ff-d8de-43d6-9262-4e460ecf7b32\") " pod="openshift-cluster-node-tuning-operator/tuned-lpfd5"
May 25 13:05:02.458577 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.458559    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2fp9\" (UniqueName: \"kubernetes.io/projected/10056893-23ca-4931-8142-3979d7611233-kube-api-access-z2fp9\") pod \"aws-ebs-csi-driver-node-t2jp2\" (UID: \"10056893-23ca-4931-8142-3979d7611233\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t2jp2"
May 25 13:05:02.458731 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.458715    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jxbs\" (UniqueName: \"kubernetes.io/projected/c749c4da-8f53-4bb5-98c2-280f3cd403af-kube-api-access-4jxbs\") pod \"multus-8xdhg\" (UID: \"c749c4da-8f53-4bb5-98c2-280f3cd403af\") " pod="openshift-multus/multus-8xdhg"
May 25 13:05:02.458842 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.458824    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pwr5\" (UniqueName: \"kubernetes.io/projected/6ef8bb8d-98b5-4fe9-94a7-8e06d34ab971-kube-api-access-8pwr5\") pod \"node-ca-wlq4q\" (UID: \"6ef8bb8d-98b5-4fe9-94a7-8e06d34ab971\") " pod="openshift-image-registry/node-ca-wlq4q"
May 25 13:05:02.533939 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.533863    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-646r6"
May 25 13:05:02.543713 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.543681    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-vlk4j"
May 25 13:05:02.552317 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.552290    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-hj74k"
May 25 13:05:02.558991 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.558971    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t2jp2"
May 25 13:05:02.566545 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.566520    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-lpfd5"
May 25 13:05:02.573056 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.573030    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-rqkvc"
May 25 13:05:02.581525 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.581503    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-wlq4q"
May 25 13:05:02.589144 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.589123    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-8xdhg"
May 25 13:05:02.596716 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.596693    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-nd955"
May 25 13:05:02.853675 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.853564    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e445efe1-4ac4-4df5-881f-748bcf8c5e0f-metrics-certs\") pod \"network-metrics-daemon-svlzs\" (UID: \"e445efe1-4ac4-4df5-881f-748bcf8c5e0f\") " pod="openshift-multus/network-metrics-daemon-svlzs"
May 25 13:05:02.853675 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:02.853639    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-lfgrd\" (UniqueName: \"kubernetes.io/projected/fdd5eb93-deb6-441f-b1fe-3511c4eee77e-kube-api-access-lfgrd\") pod \"network-check-target-25spc\" (UID: \"fdd5eb93-deb6-441f-b1fe-3511c4eee77e\") " pod="openshift-network-diagnostics/network-check-target-25spc"
May 25 13:05:02.853874 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:05:02.853738    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 13:05:02.853874 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:05:02.853810    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e445efe1-4ac4-4df5-881f-748bcf8c5e0f-metrics-certs podName:e445efe1-4ac4-4df5-881f-748bcf8c5e0f nodeName:}" failed. No retries permitted until 2026-05-25 13:05:03.853789874 +0000 UTC m=+4.046084232 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e445efe1-4ac4-4df5-881f-748bcf8c5e0f-metrics-certs") pod "network-metrics-daemon-svlzs" (UID: "e445efe1-4ac4-4df5-881f-748bcf8c5e0f") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 13:05:02.853874 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:05:02.853743    2580 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 25 13:05:02.853874 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:05:02.853840    2580 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 25 13:05:02.853874 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:05:02.853853    2580 projected.go:194] Error preparing data for projected volume kube-api-access-lfgrd for pod openshift-network-diagnostics/network-check-target-25spc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 13:05:02.854095 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:05:02.853916    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fdd5eb93-deb6-441f-b1fe-3511c4eee77e-kube-api-access-lfgrd podName:fdd5eb93-deb6-441f-b1fe-3511c4eee77e nodeName:}" failed. No retries permitted until 2026-05-25 13:05:03.853899007 +0000 UTC m=+4.046193353 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-lfgrd" (UniqueName: "kubernetes.io/projected/fdd5eb93-deb6-441f-b1fe-3511c4eee77e-kube-api-access-lfgrd") pod "network-check-target-25spc" (UID: "fdd5eb93-deb6-441f-b1fe-3511c4eee77e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 13:05:02.919802 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:02.919767    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ef8bb8d_98b5_4fe9_94a7_8e06d34ab971.slice/crio-7e93c5a216f1c7151a123ff1b53118ee9846732e989a13e180a0b4ac83c4abdd WatchSource:0}: Error finding container 7e93c5a216f1c7151a123ff1b53118ee9846732e989a13e180a0b4ac83c4abdd: Status 404 returned error can't find the container with id 7e93c5a216f1c7151a123ff1b53118ee9846732e989a13e180a0b4ac83c4abdd
May 25 13:05:02.921162 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:02.921044    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33d153fd_31f6_4481_855e_728e06ce70f5.slice/crio-ad123037e2e88cb4819baa1975b306efe06fe5d34cba219cb711ee2574e73bf1 WatchSource:0}: Error finding container ad123037e2e88cb4819baa1975b306efe06fe5d34cba219cb711ee2574e73bf1: Status 404 returned error can't find the container with id ad123037e2e88cb4819baa1975b306efe06fe5d34cba219cb711ee2574e73bf1
May 25 13:05:02.924396 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:02.924232    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc749c4da_8f53_4bb5_98c2_280f3cd403af.slice/crio-f430eb81856984d6b3351fe3459616f9f29a7d509cf731cabe06c14791ff8653 WatchSource:0}: Error finding container f430eb81856984d6b3351fe3459616f9f29a7d509cf731cabe06c14791ff8653: Status 404 returned error can't find the container with id f430eb81856984d6b3351fe3459616f9f29a7d509cf731cabe06c14791ff8653
May 25 13:05:02.931459 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:02.931431    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a8fc898_888f_420d_a433_27e831c96721.slice/crio-ee32d29007fd4077e4439982c66a4a77671456b5e4cdd523bed8a5f1e11af326 WatchSource:0}: Error finding container ee32d29007fd4077e4439982c66a4a77671456b5e4cdd523bed8a5f1e11af326: Status 404 returned error can't find the container with id ee32d29007fd4077e4439982c66a4a77671456b5e4cdd523bed8a5f1e11af326
May 25 13:05:02.932333 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:02.932310    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37b3e162_412d_4e21_8f2f_bb8398f30435.slice/crio-3c56f05e705620985619322e0c1ff38ac61135f0611311b4da5351a0dc3c5f9b WatchSource:0}: Error finding container 3c56f05e705620985619322e0c1ff38ac61135f0611311b4da5351a0dc3c5f9b: Status 404 returned error can't find the container with id 3c56f05e705620985619322e0c1ff38ac61135f0611311b4da5351a0dc3c5f9b
May 25 13:05:02.932531 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:02.932512    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59b905b8_5ce3_49f6_b47c_b5b3a525fe06.slice/crio-f0dedc2bd937aae2542586dca9704a072b7fdff4f1b85a094eb957382574bbac WatchSource:0}: Error finding container f0dedc2bd937aae2542586dca9704a072b7fdff4f1b85a094eb957382574bbac: Status 404 returned error can't find the container with id f0dedc2bd937aae2542586dca9704a072b7fdff4f1b85a094eb957382574bbac
May 25 13:05:03.286220 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:03.286172    2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-05-24 13:00:01 +0000 UTC" deadline="2028-01-08 10:52:13.570127523 +0000 UTC"
May 25 13:05:03.286220 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:03.286214    2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14229h47m10.283917264s"
May 25 13:05:03.385149 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:03.385107    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-103.ec2.internal" event={"ID":"06181dfa378829b92cc5ea772451a452","Type":"ContainerStarted","Data":"d94edebc1a58da61aef66d13af5da4a1b2cabf27833d0e845a5738ddba9df9d1"}
May 25 13:05:03.389787 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:03.389154    2580 generic.go:358] "Generic (PLEG): container finished" podID="d8b0c93f69930a962fcb9912400f9315" containerID="c5de4f6b698649ce3454c96d9c1f194f2d8887298279ee505de628c56700281d" exitCode=0
May 25 13:05:03.389787 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:03.389228    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-103.ec2.internal" event={"ID":"d8b0c93f69930a962fcb9912400f9315","Type":"ContainerDied","Data":"c5de4f6b698649ce3454c96d9c1f194f2d8887298279ee505de628c56700281d"}
May 25 13:05:03.398556 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:03.398488    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-646r6" event={"ID":"59b905b8-5ce3-49f6-b47c-b5b3a525fe06","Type":"ContainerStarted","Data":"f0dedc2bd937aae2542586dca9704a072b7fdff4f1b85a094eb957382574bbac"}
May 25 13:05:03.407459 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:03.407434    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-vlk4j" event={"ID":"37b3e162-412d-4e21-8f2f-bb8398f30435","Type":"ContainerStarted","Data":"3c56f05e705620985619322e0c1ff38ac61135f0611311b4da5351a0dc3c5f9b"}
May 25 13:05:03.416328 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:03.416277    2580 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-103.ec2.internal" podStartSLOduration=2.416260114 podStartE2EDuration="2.416260114s" podCreationTimestamp="2026-05-25 13:05:01 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 13:05:03.403061532 +0000 UTC m=+3.595355895" watchObservedRunningTime="2026-05-25 13:05:03.416260114 +0000 UTC m=+3.608554476"
May 25 13:05:03.417139 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:03.417112    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-rqkvc" event={"ID":"5a8fc898-888f-420d-a433-27e831c96721","Type":"ContainerStarted","Data":"ee32d29007fd4077e4439982c66a4a77671456b5e4cdd523bed8a5f1e11af326"}
May 25 13:05:03.427382 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:03.427298    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-nd955" event={"ID":"47afaed9-536d-4fc6-ab45-e56c19bb2ea6","Type":"ContainerStarted","Data":"ab0230e2942d2afdbe0a99368bc04e4bdafc3eaa3edc6a9af874fcf0bcf5bc72"}
May 25 13:05:03.431032 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:03.430983    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-lpfd5" event={"ID":"38b712ff-d8de-43d6-9262-4e460ecf7b32","Type":"ContainerStarted","Data":"93437e5f92e229221eba863869a3c5da37653ec6facddb0b077012c9943c0628"}
May 25 13:05:03.441636 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:03.441569    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-8xdhg" event={"ID":"c749c4da-8f53-4bb5-98c2-280f3cd403af","Type":"ContainerStarted","Data":"f430eb81856984d6b3351fe3459616f9f29a7d509cf731cabe06c14791ff8653"}
May 25 13:05:03.464775 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:03.464712    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t2jp2" event={"ID":"10056893-23ca-4931-8142-3979d7611233","Type":"ContainerStarted","Data":"3ba26d742aa24046c3c8aa329ad9e0117cfd5af57fa7c134f8c5cdeff21dacdb"}
May 25 13:05:03.478924 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:03.478876    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hj74k" event={"ID":"33d153fd-31f6-4481-855e-728e06ce70f5","Type":"ContainerStarted","Data":"ad123037e2e88cb4819baa1975b306efe06fe5d34cba219cb711ee2574e73bf1"}
May 25 13:05:03.480664 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:03.480610    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-wlq4q" event={"ID":"6ef8bb8d-98b5-4fe9-94a7-8e06d34ab971","Type":"ContainerStarted","Data":"7e93c5a216f1c7151a123ff1b53118ee9846732e989a13e180a0b4ac83c4abdd"}
May 25 13:05:03.862620 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:03.862535    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e445efe1-4ac4-4df5-881f-748bcf8c5e0f-metrics-certs\") pod \"network-metrics-daemon-svlzs\" (UID: \"e445efe1-4ac4-4df5-881f-748bcf8c5e0f\") " pod="openshift-multus/network-metrics-daemon-svlzs"
May 25 13:05:03.862620 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:03.862596    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-lfgrd\" (UniqueName: \"kubernetes.io/projected/fdd5eb93-deb6-441f-b1fe-3511c4eee77e-kube-api-access-lfgrd\") pod \"network-check-target-25spc\" (UID: \"fdd5eb93-deb6-441f-b1fe-3511c4eee77e\") " pod="openshift-network-diagnostics/network-check-target-25spc"
May 25 13:05:03.862861 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:05:03.862782    2580 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 25 13:05:03.862861 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:05:03.862800    2580 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 25 13:05:03.862861 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:05:03.862813    2580 projected.go:194] Error preparing data for projected volume kube-api-access-lfgrd for pod openshift-network-diagnostics/network-check-target-25spc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 13:05:03.863007 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:05:03.862872    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fdd5eb93-deb6-441f-b1fe-3511c4eee77e-kube-api-access-lfgrd podName:fdd5eb93-deb6-441f-b1fe-3511c4eee77e nodeName:}" failed. No retries permitted until 2026-05-25 13:05:05.862853214 +0000 UTC m=+6.055147563 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-lfgrd" (UniqueName: "kubernetes.io/projected/fdd5eb93-deb6-441f-b1fe-3511c4eee77e-kube-api-access-lfgrd") pod "network-check-target-25spc" (UID: "fdd5eb93-deb6-441f-b1fe-3511c4eee77e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 13:05:03.863277 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:05:03.863260    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 13:05:03.863352 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:05:03.863310    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e445efe1-4ac4-4df5-881f-748bcf8c5e0f-metrics-certs podName:e445efe1-4ac4-4df5-881f-748bcf8c5e0f nodeName:}" failed. No retries permitted until 2026-05-25 13:05:05.863294217 +0000 UTC m=+6.055588559 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e445efe1-4ac4-4df5-881f-748bcf8c5e0f-metrics-certs") pod "network-metrics-daemon-svlzs" (UID: "e445efe1-4ac4-4df5-881f-748bcf8c5e0f") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 13:05:04.375684 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:04.374967    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-svlzs"
May 25 13:05:04.375684 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:05:04.375116    2580 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-svlzs" podUID="e445efe1-4ac4-4df5-881f-748bcf8c5e0f"
May 25 13:05:04.375684 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:04.375544    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-25spc"
May 25 13:05:04.375684 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:05:04.375630    2580 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-25spc" podUID="fdd5eb93-deb6-441f-b1fe-3511c4eee77e"
May 25 13:05:04.496891 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:04.496856    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-103.ec2.internal" event={"ID":"d8b0c93f69930a962fcb9912400f9315","Type":"ContainerStarted","Data":"8f3fbb9c19d7b8ff20f431b845a4d25861049d9c64443a6c2ad1884fad21599e"}
May 25 13:05:05.885920 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:05.885080    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e445efe1-4ac4-4df5-881f-748bcf8c5e0f-metrics-certs\") pod \"network-metrics-daemon-svlzs\" (UID: \"e445efe1-4ac4-4df5-881f-748bcf8c5e0f\") " pod="openshift-multus/network-metrics-daemon-svlzs"
May 25 13:05:05.885920 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:05.885136    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-lfgrd\" (UniqueName: \"kubernetes.io/projected/fdd5eb93-deb6-441f-b1fe-3511c4eee77e-kube-api-access-lfgrd\") pod \"network-check-target-25spc\" (UID: \"fdd5eb93-deb6-441f-b1fe-3511c4eee77e\") " pod="openshift-network-diagnostics/network-check-target-25spc"
May 25 13:05:05.885920 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:05:05.885292    2580 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 25 13:05:05.885920 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:05:05.885312    2580 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 25 13:05:05.885920 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:05:05.885326    2580 projected.go:194] Error preparing data for projected volume kube-api-access-lfgrd for pod openshift-network-diagnostics/network-check-target-25spc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 13:05:05.885920 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:05:05.885404    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fdd5eb93-deb6-441f-b1fe-3511c4eee77e-kube-api-access-lfgrd podName:fdd5eb93-deb6-441f-b1fe-3511c4eee77e nodeName:}" failed. No retries permitted until 2026-05-25 13:05:09.885384975 +0000 UTC m=+10.077679328 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-lfgrd" (UniqueName: "kubernetes.io/projected/fdd5eb93-deb6-441f-b1fe-3511c4eee77e-kube-api-access-lfgrd") pod "network-check-target-25spc" (UID: "fdd5eb93-deb6-441f-b1fe-3511c4eee77e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 13:05:05.885920 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:05:05.885791    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 13:05:05.885920 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:05:05.885837    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e445efe1-4ac4-4df5-881f-748bcf8c5e0f-metrics-certs podName:e445efe1-4ac4-4df5-881f-748bcf8c5e0f nodeName:}" failed. No retries permitted until 2026-05-25 13:05:09.885823482 +0000 UTC m=+10.078117825 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e445efe1-4ac4-4df5-881f-748bcf8c5e0f-metrics-certs") pod "network-metrics-daemon-svlzs" (UID: "e445efe1-4ac4-4df5-881f-748bcf8c5e0f") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 13:05:06.373298 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:06.372583    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-svlzs"
May 25 13:05:06.373298 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:05:06.372742    2580 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-svlzs" podUID="e445efe1-4ac4-4df5-881f-748bcf8c5e0f"
May 25 13:05:06.373298 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:06.373146    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-25spc"
May 25 13:05:06.373298 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:05:06.373222    2580 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-25spc" podUID="fdd5eb93-deb6-441f-b1fe-3511c4eee77e"
May 25 13:05:08.372616 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:08.372577    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-25spc"
May 25 13:05:08.373009 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:08.372577    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-svlzs"
May 25 13:05:08.373009 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:05:08.372855    2580 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-svlzs" podUID="e445efe1-4ac4-4df5-881f-748bcf8c5e0f"
May 25 13:05:08.373009 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:05:08.372716    2580 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-25spc" podUID="fdd5eb93-deb6-441f-b1fe-3511c4eee77e"
May 25 13:05:09.921522 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:09.920753    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-lfgrd\" (UniqueName: \"kubernetes.io/projected/fdd5eb93-deb6-441f-b1fe-3511c4eee77e-kube-api-access-lfgrd\") pod \"network-check-target-25spc\" (UID: \"fdd5eb93-deb6-441f-b1fe-3511c4eee77e\") " pod="openshift-network-diagnostics/network-check-target-25spc"
May 25 13:05:09.921522 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:09.920839    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e445efe1-4ac4-4df5-881f-748bcf8c5e0f-metrics-certs\") pod \"network-metrics-daemon-svlzs\" (UID: \"e445efe1-4ac4-4df5-881f-748bcf8c5e0f\") " pod="openshift-multus/network-metrics-daemon-svlzs"
May 25 13:05:09.921522 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:05:09.920954    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 13:05:09.921522 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:05:09.921014    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e445efe1-4ac4-4df5-881f-748bcf8c5e0f-metrics-certs podName:e445efe1-4ac4-4df5-881f-748bcf8c5e0f nodeName:}" failed. No retries permitted until 2026-05-25 13:05:17.920994619 +0000 UTC m=+18.113288980 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e445efe1-4ac4-4df5-881f-748bcf8c5e0f-metrics-certs") pod "network-metrics-daemon-svlzs" (UID: "e445efe1-4ac4-4df5-881f-748bcf8c5e0f") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 13:05:09.921522 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:05:09.921427    2580 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 25 13:05:09.921522 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:05:09.921443    2580 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 25 13:05:09.921522 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:05:09.921455    2580 projected.go:194] Error preparing data for projected volume kube-api-access-lfgrd for pod openshift-network-diagnostics/network-check-target-25spc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 13:05:09.921522 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:05:09.921488    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fdd5eb93-deb6-441f-b1fe-3511c4eee77e-kube-api-access-lfgrd podName:fdd5eb93-deb6-441f-b1fe-3511c4eee77e nodeName:}" failed. No retries permitted until 2026-05-25 13:05:17.921478568 +0000 UTC m=+18.113772907 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-lfgrd" (UniqueName: "kubernetes.io/projected/fdd5eb93-deb6-441f-b1fe-3511c4eee77e-kube-api-access-lfgrd") pod "network-check-target-25spc" (UID: "fdd5eb93-deb6-441f-b1fe-3511c4eee77e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 13:05:10.374164 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:10.374127    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-svlzs"
May 25 13:05:10.374164 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:10.374158    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-25spc"
May 25 13:05:10.374361 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:05:10.374252    2580 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-svlzs" podUID="e445efe1-4ac4-4df5-881f-748bcf8c5e0f"
May 25 13:05:10.374361 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:05:10.374321    2580 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-25spc" podUID="fdd5eb93-deb6-441f-b1fe-3511c4eee77e"
May 25 13:05:12.372554 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:12.372518    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-25spc"
May 25 13:05:12.372985 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:05:12.372643    2580 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-25spc" podUID="fdd5eb93-deb6-441f-b1fe-3511c4eee77e"
May 25 13:05:12.372985 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:12.372676    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-svlzs"
May 25 13:05:12.372985 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:05:12.372802    2580 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-svlzs" podUID="e445efe1-4ac4-4df5-881f-748bcf8c5e0f"
May 25 13:05:14.372912 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:14.372883    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-svlzs"
May 25 13:05:14.373456 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:14.372883    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-25spc"
May 25 13:05:14.373456 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:05:14.372989    2580 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-svlzs" podUID="e445efe1-4ac4-4df5-881f-748bcf8c5e0f"
May 25 13:05:14.373456 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:05:14.373096    2580 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-25spc" podUID="fdd5eb93-deb6-441f-b1fe-3511c4eee77e"
May 25 13:05:16.372618 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:16.372579    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-svlzs"
May 25 13:05:16.373034 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:05:16.372696    2580 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-svlzs" podUID="e445efe1-4ac4-4df5-881f-748bcf8c5e0f"
May 25 13:05:16.373034 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:16.372763    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-25spc"
May 25 13:05:16.373034 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:05:16.372856    2580 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-25spc" podUID="fdd5eb93-deb6-441f-b1fe-3511c4eee77e"
May 25 13:05:17.979420 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:17.979380    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e445efe1-4ac4-4df5-881f-748bcf8c5e0f-metrics-certs\") pod \"network-metrics-daemon-svlzs\" (UID: \"e445efe1-4ac4-4df5-881f-748bcf8c5e0f\") " pod="openshift-multus/network-metrics-daemon-svlzs"
May 25 13:05:17.979420 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:17.979426    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-lfgrd\" (UniqueName: \"kubernetes.io/projected/fdd5eb93-deb6-441f-b1fe-3511c4eee77e-kube-api-access-lfgrd\") pod \"network-check-target-25spc\" (UID: \"fdd5eb93-deb6-441f-b1fe-3511c4eee77e\") " pod="openshift-network-diagnostics/network-check-target-25spc"
May 25 13:05:17.979861 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:05:17.979551    2580 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 25 13:05:17.979861 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:05:17.979550    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 13:05:17.979861 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:05:17.979571    2580 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 25 13:05:17.979861 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:05:17.979588    2580 projected.go:194] Error preparing data for projected volume kube-api-access-lfgrd for pod openshift-network-diagnostics/network-check-target-25spc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 13:05:17.979861 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:05:17.979637    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fdd5eb93-deb6-441f-b1fe-3511c4eee77e-kube-api-access-lfgrd podName:fdd5eb93-deb6-441f-b1fe-3511c4eee77e nodeName:}" failed. No retries permitted until 2026-05-25 13:05:33.979621904 +0000 UTC m=+34.171916242 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-lfgrd" (UniqueName: "kubernetes.io/projected/fdd5eb93-deb6-441f-b1fe-3511c4eee77e-kube-api-access-lfgrd") pod "network-check-target-25spc" (UID: "fdd5eb93-deb6-441f-b1fe-3511c4eee77e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 13:05:17.979861 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:05:17.979651    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e445efe1-4ac4-4df5-881f-748bcf8c5e0f-metrics-certs podName:e445efe1-4ac4-4df5-881f-748bcf8c5e0f nodeName:}" failed. No retries permitted until 2026-05-25 13:05:33.979644173 +0000 UTC m=+34.171938511 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e445efe1-4ac4-4df5-881f-748bcf8c5e0f-metrics-certs") pod "network-metrics-daemon-svlzs" (UID: "e445efe1-4ac4-4df5-881f-748bcf8c5e0f") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 13:05:18.373298 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:18.373262    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-svlzs"
May 25 13:05:18.373501 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:18.373264    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-25spc"
May 25 13:05:18.373501 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:05:18.373425    2580 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-svlzs" podUID="e445efe1-4ac4-4df5-881f-748bcf8c5e0f"
May 25 13:05:18.373501 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:05:18.373489    2580 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-25spc" podUID="fdd5eb93-deb6-441f-b1fe-3511c4eee77e"
May 25 13:05:20.373566 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:20.373486    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-svlzs"
May 25 13:05:20.373983 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:05:20.373624    2580 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-svlzs" podUID="e445efe1-4ac4-4df5-881f-748bcf8c5e0f"
May 25 13:05:20.373983 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:20.373662    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-25spc"
May 25 13:05:20.373983 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:05:20.373781    2580 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-25spc" podUID="fdd5eb93-deb6-441f-b1fe-3511c4eee77e"
May 25 13:05:21.528459 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:21.528028    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-vlk4j" event={"ID":"37b3e162-412d-4e21-8f2f-bb8398f30435","Type":"ContainerStarted","Data":"668944fd04f4e4c045f4202d4aa113b8c9c73ac8acd3550e8d15894dac20f8a2"}
May 25 13:05:21.530406 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:21.530007    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-rqkvc" event={"ID":"5a8fc898-888f-420d-a433-27e831c96721","Type":"ContainerStarted","Data":"534eadea289b9bdd1885de688f782833a42c67b09a321f6ccb79881e0f10a84d"}
May 25 13:05:21.533024 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:21.532653    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-lpfd5" event={"ID":"38b712ff-d8de-43d6-9262-4e460ecf7b32","Type":"ContainerStarted","Data":"a31c4bd40e8e40ba5da9c8514ea9463e18546fbd7449b229bb589151840b638f"}
May 25 13:05:21.536776 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:21.536752    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t2jp2" event={"ID":"10056893-23ca-4931-8142-3979d7611233","Type":"ContainerStarted","Data":"1c164f18b65ce99ce0de97be26b599dfb60bbb53c2454ff9b0d704aa7b754984"}
May 25 13:05:21.539191 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:21.539164    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-wlq4q" event={"ID":"6ef8bb8d-98b5-4fe9-94a7-8e06d34ab971","Type":"ContainerStarted","Data":"f6e4b33e383d4db0111bef7add46700adb45b840da04a85f1d97a32daa4851ac"}
May 25 13:05:21.543034 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:21.542990    2580 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-dns/node-resolver-vlk4j" podStartSLOduration=3.255080422 podStartE2EDuration="21.54298049s" podCreationTimestamp="2026-05-25 13:05:00 +0000 UTC" firstStartedPulling="2026-05-25 13:05:02.95568917 +0000 UTC m=+3.147983511" lastFinishedPulling="2026-05-25 13:05:21.24358924 +0000 UTC m=+21.435883579" observedRunningTime="2026-05-25 13:05:21.54281449 +0000 UTC m=+21.735108847" watchObservedRunningTime="2026-05-25 13:05:21.54298049 +0000 UTC m=+21.735274849"
May 25 13:05:21.544193 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:21.543605    2580 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-103.ec2.internal" podStartSLOduration=20.543595327 podStartE2EDuration="20.543595327s" podCreationTimestamp="2026-05-25 13:05:01 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 13:05:04.511728188 +0000 UTC m=+4.704022550" watchObservedRunningTime="2026-05-25 13:05:21.543595327 +0000 UTC m=+21.735889689"
May 25 13:05:21.582543 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:21.582504    2580 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/konnectivity-agent-rqkvc" podStartSLOduration=3.317007405 podStartE2EDuration="21.582492301s" podCreationTimestamp="2026-05-25 13:05:00 +0000 UTC" firstStartedPulling="2026-05-25 13:05:02.955746814 +0000 UTC m=+3.148041167" lastFinishedPulling="2026-05-25 13:05:21.221231726 +0000 UTC m=+21.413526063" observedRunningTime="2026-05-25 13:05:21.582340931 +0000 UTC m=+21.774635287" watchObservedRunningTime="2026-05-25 13:05:21.582492301 +0000 UTC m=+21.774786657"
May 25 13:05:21.582897 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:21.582874    2580 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-lpfd5" podStartSLOduration=3.266517867 podStartE2EDuration="21.582868685s" podCreationTimestamp="2026-05-25 13:05:00 +0000 UTC" firstStartedPulling="2026-05-25 13:05:02.929420686 +0000 UTC m=+3.121715027" lastFinishedPulling="2026-05-25 13:05:21.245771494 +0000 UTC m=+21.438065845" observedRunningTime="2026-05-25 13:05:21.567051331 +0000 UTC m=+21.759345687" watchObservedRunningTime="2026-05-25 13:05:21.582868685 +0000 UTC m=+21.775163045"
May 25 13:05:21.619560 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:21.619522    2580 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-image-registry/node-ca-wlq4q" podStartSLOduration=11.520480856 podStartE2EDuration="21.619508762s" podCreationTimestamp="2026-05-25 13:05:00 +0000 UTC" firstStartedPulling="2026-05-25 13:05:02.922708163 +0000 UTC m=+3.115002504" lastFinishedPulling="2026-05-25 13:05:13.021736058 +0000 UTC m=+13.214030410" observedRunningTime="2026-05-25 13:05:21.618988243 +0000 UTC m=+21.811282606" watchObservedRunningTime="2026-05-25 13:05:21.619508762 +0000 UTC m=+21.811803121"
May 25 13:05:22.372654 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:22.372480    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-svlzs"
May 25 13:05:22.372837 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:22.372526    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-25spc"
May 25 13:05:22.372837 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:05:22.372726    2580 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-svlzs" podUID="e445efe1-4ac4-4df5-881f-748bcf8c5e0f"
May 25 13:05:22.372837 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:05:22.372779    2580 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-25spc" podUID="fdd5eb93-deb6-441f-b1fe-3511c4eee77e"
May 25 13:05:22.543209 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:22.543181    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-646r6_59b905b8-5ce3-49f6-b47c-b5b3a525fe06/ovn-acl-logging/0.log"
May 25 13:05:22.543744 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:22.543509    2580 generic.go:358] "Generic (PLEG): container finished" podID="59b905b8-5ce3-49f6-b47c-b5b3a525fe06" containerID="15198629395d35bd4d929d466740577cfef5f41fe070fd2830bfc6c919a7f0d6" exitCode=1
May 25 13:05:22.543744 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:22.543579    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-646r6" event={"ID":"59b905b8-5ce3-49f6-b47c-b5b3a525fe06","Type":"ContainerStarted","Data":"05efc117fa89ecdadecd6491cd10927f9ed291f995bf32a72aab5d5b28c81eb3"}
May 25 13:05:22.543744 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:22.543616    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-646r6" event={"ID":"59b905b8-5ce3-49f6-b47c-b5b3a525fe06","Type":"ContainerStarted","Data":"a2b263cf9a4143aced53fa50baf04d65b61b63f3a540ddd46b8297dd5f7c13f8"}
May 25 13:05:22.543744 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:22.543625    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-646r6" event={"ID":"59b905b8-5ce3-49f6-b47c-b5b3a525fe06","Type":"ContainerStarted","Data":"8031399419d5849b7d202533306cb3b30fc391dfc898e084cbca5666840099d2"}
May 25 13:05:22.543744 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:22.543633    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-646r6" event={"ID":"59b905b8-5ce3-49f6-b47c-b5b3a525fe06","Type":"ContainerStarted","Data":"cf5dc9c43f01ead442bd6e326b644fa7f32262e9cc7815b3264dc66d13c1dbdb"}
May 25 13:05:22.543744 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:22.543640    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-646r6" event={"ID":"59b905b8-5ce3-49f6-b47c-b5b3a525fe06","Type":"ContainerDied","Data":"15198629395d35bd4d929d466740577cfef5f41fe070fd2830bfc6c919a7f0d6"}
May 25 13:05:22.543744 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:22.543651    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-646r6" event={"ID":"59b905b8-5ce3-49f6-b47c-b5b3a525fe06","Type":"ContainerStarted","Data":"5de423c857c011399217635e81dab28de45d9e7e80f76da1307ba4acea36c00e"}
May 25 13:05:22.544716 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:22.544688    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-nd955" event={"ID":"47afaed9-536d-4fc6-ab45-e56c19bb2ea6","Type":"ContainerStarted","Data":"4f9ebd7a8af9bc348ce4bed9502758c8d7c39555e218e2e859221981dc6ee713"}
May 25 13:05:22.545907 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:22.545885    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-8xdhg" event={"ID":"c749c4da-8f53-4bb5-98c2-280f3cd403af","Type":"ContainerStarted","Data":"95800be2f2139f11487121a1bbfadd21fd461a351370081a3e7b6d57d543c815"}
May 25 13:05:22.547092 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:22.547073    2580 generic.go:358] "Generic (PLEG): container finished" podID="33d153fd-31f6-4481-855e-728e06ce70f5" containerID="5ef57552e922d7a1a3d5c29641a0c19ee8c30f9fac169a11752a7fd386b0d013" exitCode=0
May 25 13:05:22.547186 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:22.547162    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hj74k" event={"ID":"33d153fd-31f6-4481-855e-728e06ce70f5","Type":"ContainerDied","Data":"5ef57552e922d7a1a3d5c29641a0c19ee8c30f9fac169a11752a7fd386b0d013"}
May 25 13:05:22.573634 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:22.573588    2580 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/multus-8xdhg" podStartSLOduration=4.2419312080000005 podStartE2EDuration="22.573577436s" podCreationTimestamp="2026-05-25 13:05:00 +0000 UTC" firstStartedPulling="2026-05-25 13:05:02.927005127 +0000 UTC m=+3.119299479" lastFinishedPulling="2026-05-25 13:05:21.258651369 +0000 UTC m=+21.450945707" observedRunningTime="2026-05-25 13:05:22.573035907 +0000 UTC m=+22.765330267" watchObservedRunningTime="2026-05-25 13:05:22.573577436 +0000 UTC m=+22.765871795"
May 25 13:05:22.573756 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:22.573709    2580 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-nd955" podStartSLOduration=4.259237977 podStartE2EDuration="22.573701854s" podCreationTimestamp="2026-05-25 13:05:00 +0000 UTC" firstStartedPulling="2026-05-25 13:05:02.930239919 +0000 UTC m=+3.122534271" lastFinishedPulling="2026-05-25 13:05:21.244703796 +0000 UTC m=+21.436998148" observedRunningTime="2026-05-25 13:05:22.558341204 +0000 UTC m=+22.750635563" watchObservedRunningTime="2026-05-25 13:05:22.573701854 +0000 UTC m=+22.765996214"
May 25 13:05:22.728095 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:22.728070    2580 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
May 25 13:05:23.260648 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:23.260436    2580 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-rqkvc"
May 25 13:05:23.261188 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:23.261169    2580 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-rqkvc"
May 25 13:05:23.308645 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:23.308514    2580 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-05-25T13:05:22.728091087Z","UUID":"09dc836c-d339-4151-b399-0853cf9ce009","Handler":null,"Name":"","Endpoint":""}
May 25 13:05:23.312298 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:23.312262    2580 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
May 25 13:05:23.312298 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:23.312295    2580 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
May 25 13:05:23.373172 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:23.373141    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-rqkvc"
May 25 13:05:23.373825 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:23.373806    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-rqkvc"
May 25 13:05:23.551411 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:23.551305    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t2jp2" event={"ID":"10056893-23ca-4931-8142-3979d7611233","Type":"ContainerStarted","Data":"44b4b07d75369c62bc9289e03c1930a17d53e79cc64eae5500c306027455fe77"}
May 25 13:05:24.373974 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:24.373485    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-svlzs"
May 25 13:05:24.373974 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:05:24.373637    2580 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-svlzs" podUID="e445efe1-4ac4-4df5-881f-748bcf8c5e0f"
May 25 13:05:24.374473 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:24.374310    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-25spc"
May 25 13:05:24.374473 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:05:24.374434    2580 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-25spc" podUID="fdd5eb93-deb6-441f-b1fe-3511c4eee77e"
May 25 13:05:24.556385 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:24.556345    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-646r6_59b905b8-5ce3-49f6-b47c-b5b3a525fe06/ovn-acl-logging/0.log"
May 25 13:05:24.556901 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:24.556716    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-646r6" event={"ID":"59b905b8-5ce3-49f6-b47c-b5b3a525fe06","Type":"ContainerStarted","Data":"752e485d16bee16fdaae1ddfc0a46be274d9d83ba24ea11ce657e286f95813e5"}
May 25 13:05:24.558852 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:24.558816    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t2jp2" event={"ID":"10056893-23ca-4931-8142-3979d7611233","Type":"ContainerStarted","Data":"4595a35d3c9cd10885d6ee185813e7b7e40922af07d9fd34ec0ce423998e6c51"}
May 25 13:05:26.372536 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:26.372506    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-svlzs"
May 25 13:05:26.372536 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:26.372522    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-25spc"
May 25 13:05:26.373259 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:05:26.372634    2580 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-svlzs" podUID="e445efe1-4ac4-4df5-881f-748bcf8c5e0f"
May 25 13:05:26.373259 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:05:26.372725    2580 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-25spc" podUID="fdd5eb93-deb6-441f-b1fe-3511c4eee77e"
May 25 13:05:27.565724 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:27.565557    2580 generic.go:358] "Generic (PLEG): container finished" podID="33d153fd-31f6-4481-855e-728e06ce70f5" containerID="40baf329f07c9f3ed8289639cecb3aec427c5b4b7605a659f6f62a36729f7159" exitCode=0
May 25 13:05:27.566212 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:27.565634    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hj74k" event={"ID":"33d153fd-31f6-4481-855e-728e06ce70f5","Type":"ContainerDied","Data":"40baf329f07c9f3ed8289639cecb3aec427c5b4b7605a659f6f62a36729f7159"}
May 25 13:05:27.568882 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:27.568866    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-646r6_59b905b8-5ce3-49f6-b47c-b5b3a525fe06/ovn-acl-logging/0.log"
May 25 13:05:27.569237 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:27.569210    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-646r6" event={"ID":"59b905b8-5ce3-49f6-b47c-b5b3a525fe06","Type":"ContainerStarted","Data":"172fc0d3f637533b83e80645b284b6e034ea78c6efe28440a7353ea3e0dd275f"}
May 25 13:05:27.569491 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:27.569473    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-646r6"
May 25 13:05:27.569578 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:27.569506    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-646r6"
May 25 13:05:27.569700 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:27.569684    2580 scope.go:117] "RemoveContainer" containerID="15198629395d35bd4d929d466740577cfef5f41fe070fd2830bfc6c919a7f0d6"
May 25 13:05:27.586856 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:27.586827    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-646r6"
May 25 13:05:27.586946 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:27.586911    2580 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-t2jp2" podStartSLOduration=6.458151365 podStartE2EDuration="27.586898847s" podCreationTimestamp="2026-05-25 13:05:00 +0000 UTC" firstStartedPulling="2026-05-25 13:05:02.928533694 +0000 UTC m=+3.120828039" lastFinishedPulling="2026-05-25 13:05:24.057281184 +0000 UTC m=+24.249575521" observedRunningTime="2026-05-25 13:05:24.578736661 +0000 UTC m=+24.771031022" watchObservedRunningTime="2026-05-25 13:05:27.586898847 +0000 UTC m=+27.779193206"
May 25 13:05:28.374681 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:28.374641    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-svlzs"
May 25 13:05:28.374881 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:05:28.374783    2580 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-svlzs" podUID="e445efe1-4ac4-4df5-881f-748bcf8c5e0f"
May 25 13:05:28.375232 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:28.375211    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-25spc"
May 25 13:05:28.375316 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:05:28.375302    2580 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-25spc" podUID="fdd5eb93-deb6-441f-b1fe-3511c4eee77e"
May 25 13:05:28.574780 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:28.574751    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-646r6_59b905b8-5ce3-49f6-b47c-b5b3a525fe06/ovn-acl-logging/0.log"
May 25 13:05:28.575237 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:28.575122    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-646r6" event={"ID":"59b905b8-5ce3-49f6-b47c-b5b3a525fe06","Type":"ContainerStarted","Data":"f74b8ab4c74dbc27a2caaa938c833624bd0b5f54173901687c83901005002e06"}
May 25 13:05:28.575503 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:28.575482    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-646r6"
May 25 13:05:28.590943 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:28.590912    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-646r6"
May 25 13:05:28.627730 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:28.627644    2580 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-646r6" podStartSLOduration=10.285709004 podStartE2EDuration="28.627630171s" podCreationTimestamp="2026-05-25 13:05:00 +0000 UTC" firstStartedPulling="2026-05-25 13:05:02.95569076 +0000 UTC m=+3.147985114" lastFinishedPulling="2026-05-25 13:05:21.297611936 +0000 UTC m=+21.489906281" observedRunningTime="2026-05-25 13:05:28.601340023 +0000 UTC m=+28.793634384" watchObservedRunningTime="2026-05-25 13:05:28.627630171 +0000 UTC m=+28.819924531"
May 25 13:05:29.083132 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:29.082902    2580 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-25spc"]
May 25 13:05:29.083323 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:29.083260    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-25spc"
May 25 13:05:29.083412 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:05:29.083350    2580 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-25spc" podUID="fdd5eb93-deb6-441f-b1fe-3511c4eee77e"
May 25 13:05:29.083775 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:29.083752    2580 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-svlzs"]
May 25 13:05:29.083871 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:29.083860    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-svlzs"
May 25 13:05:29.083990 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:05:29.083963    2580 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-svlzs" podUID="e445efe1-4ac4-4df5-881f-748bcf8c5e0f"
May 25 13:05:29.578857 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:29.578821    2580 generic.go:358] "Generic (PLEG): container finished" podID="33d153fd-31f6-4481-855e-728e06ce70f5" containerID="8667470c7195ac64524f4c6e19ff53f501c83b3ba8252dd27780a981653d64fd" exitCode=0
May 25 13:05:29.579246 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:29.578899    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hj74k" event={"ID":"33d153fd-31f6-4481-855e-728e06ce70f5","Type":"ContainerDied","Data":"8667470c7195ac64524f4c6e19ff53f501c83b3ba8252dd27780a981653d64fd"}
May 25 13:05:30.373332 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:30.373295    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-25spc"
May 25 13:05:30.373549 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:05:30.373413    2580 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-25spc" podUID="fdd5eb93-deb6-441f-b1fe-3511c4eee77e"
May 25 13:05:30.373549 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:30.373478    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-svlzs"
May 25 13:05:30.373647 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:05:30.373575    2580 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-svlzs" podUID="e445efe1-4ac4-4df5-881f-748bcf8c5e0f"
May 25 13:05:31.583737 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:31.583706    2580 generic.go:358] "Generic (PLEG): container finished" podID="33d153fd-31f6-4481-855e-728e06ce70f5" containerID="e6b4535ffcfd2ffefbcb2e27dc5db525f68f62c0ac1f32869de155d3f7e25a5d" exitCode=0
May 25 13:05:31.584282 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:31.583769    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hj74k" event={"ID":"33d153fd-31f6-4481-855e-728e06ce70f5","Type":"ContainerDied","Data":"e6b4535ffcfd2ffefbcb2e27dc5db525f68f62c0ac1f32869de155d3f7e25a5d"}
May 25 13:05:32.372997 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:32.372962    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-svlzs"
May 25 13:05:32.373168 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:32.372962    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-25spc"
May 25 13:05:32.373168 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:05:32.373111    2580 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-svlzs" podUID="e445efe1-4ac4-4df5-881f-748bcf8c5e0f"
May 25 13:05:32.373168 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:05:32.373153    2580 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-25spc" podUID="fdd5eb93-deb6-441f-b1fe-3511c4eee77e"
May 25 13:05:33.999356 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:33.999315    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e445efe1-4ac4-4df5-881f-748bcf8c5e0f-metrics-certs\") pod \"network-metrics-daemon-svlzs\" (UID: \"e445efe1-4ac4-4df5-881f-748bcf8c5e0f\") " pod="openshift-multus/network-metrics-daemon-svlzs"
May 25 13:05:33.999881 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:33.999388    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-lfgrd\" (UniqueName: \"kubernetes.io/projected/fdd5eb93-deb6-441f-b1fe-3511c4eee77e-kube-api-access-lfgrd\") pod \"network-check-target-25spc\" (UID: \"fdd5eb93-deb6-441f-b1fe-3511c4eee77e\") " pod="openshift-network-diagnostics/network-check-target-25spc"
May 25 13:05:33.999881 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:05:33.999481    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 13:05:33.999881 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:05:33.999521    2580 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 25 13:05:33.999881 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:05:33.999537    2580 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 25 13:05:33.999881 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:05:33.999540    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e445efe1-4ac4-4df5-881f-748bcf8c5e0f-metrics-certs podName:e445efe1-4ac4-4df5-881f-748bcf8c5e0f nodeName:}" failed. No retries permitted until 2026-05-25 13:06:05.999524442 +0000 UTC m=+66.191818780 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e445efe1-4ac4-4df5-881f-748bcf8c5e0f-metrics-certs") pod "network-metrics-daemon-svlzs" (UID: "e445efe1-4ac4-4df5-881f-748bcf8c5e0f") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 13:05:33.999881 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:05:33.999546    2580 projected.go:194] Error preparing data for projected volume kube-api-access-lfgrd for pod openshift-network-diagnostics/network-check-target-25spc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 13:05:33.999881 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:05:33.999584    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fdd5eb93-deb6-441f-b1fe-3511c4eee77e-kube-api-access-lfgrd podName:fdd5eb93-deb6-441f-b1fe-3511c4eee77e nodeName:}" failed. No retries permitted until 2026-05-25 13:06:05.999572547 +0000 UTC m=+66.191866885 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-lfgrd" (UniqueName: "kubernetes.io/projected/fdd5eb93-deb6-441f-b1fe-3511c4eee77e-kube-api-access-lfgrd") pod "network-check-target-25spc" (UID: "fdd5eb93-deb6-441f-b1fe-3511c4eee77e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 13:05:34.177440 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:34.177410    2580 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-135-103.ec2.internal" event="NodeReady"
May 25 13:05:34.177608 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:34.177566    2580 kubelet_node_status.go:550] "Fast updating node status as it just became ready"
May 25 13:05:34.228733 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:34.228660    2580 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-ltl6z"]
May 25 13:05:34.257477 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:34.257445    2580 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-8gtmh"]
May 25 13:05:34.257640 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:34.257623    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-ltl6z"
May 25 13:05:34.260767 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:34.260748    2580 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
May 25 13:05:34.261017 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:34.260987    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
May 25 13:05:34.261119 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:34.260988    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-9wl44\""
May 25 13:05:34.273611 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:34.273588    2580 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-ltl6z"]
May 25 13:05:34.273716 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:34.273604    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-8gtmh"
May 25 13:05:34.273716 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:34.273632    2580 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-8gtmh"]
May 25 13:05:34.276567 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:34.276549    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
May 25 13:05:34.276567 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:34.276563    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-zhtj8\""
May 25 13:05:34.276837 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:34.276818    2580 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
May 25 13:05:34.276837 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:34.276835    2580 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
May 25 13:05:34.373310 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:34.373276    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-svlzs"
May 25 13:05:34.373508 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:34.373323    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-25spc"
May 25 13:05:34.376116 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:34.376092    2580 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
May 25 13:05:34.376247 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:34.376137    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-cw8xk\""
May 25 13:05:34.376385 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:34.376353    2580 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
May 25 13:05:34.376575 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:34.376556    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
May 25 13:05:34.376575 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:34.376569    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-rx2c5\""
May 25 13:05:34.402753 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:34.402720    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c04c0bd2-e6c2-42a0-949b-8f818987dd6b-config-volume\") pod \"dns-default-ltl6z\" (UID: \"c04c0bd2-e6c2-42a0-949b-8f818987dd6b\") " pod="openshift-dns/dns-default-ltl6z"
May 25 13:05:34.402922 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:34.402762    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c04c0bd2-e6c2-42a0-949b-8f818987dd6b-tmp-dir\") pod \"dns-default-ltl6z\" (UID: \"c04c0bd2-e6c2-42a0-949b-8f818987dd6b\") " pod="openshift-dns/dns-default-ltl6z"
May 25 13:05:34.402922 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:34.402810    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2e9a6623-9ccb-484c-b669-38caefdf2976-cert\") pod \"ingress-canary-8gtmh\" (UID: \"2e9a6623-9ccb-484c-b669-38caefdf2976\") " pod="openshift-ingress-canary/ingress-canary-8gtmh"
May 25 13:05:34.402922 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:34.402865    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5v77\" (UniqueName: \"kubernetes.io/projected/2e9a6623-9ccb-484c-b669-38caefdf2976-kube-api-access-s5v77\") pod \"ingress-canary-8gtmh\" (UID: \"2e9a6623-9ccb-484c-b669-38caefdf2976\") " pod="openshift-ingress-canary/ingress-canary-8gtmh"
May 25 13:05:34.402922 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:34.402902    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c04c0bd2-e6c2-42a0-949b-8f818987dd6b-metrics-tls\") pod \"dns-default-ltl6z\" (UID: \"c04c0bd2-e6c2-42a0-949b-8f818987dd6b\") " pod="openshift-dns/dns-default-ltl6z"
May 25 13:05:34.403133 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:34.402926    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmmm2\" (UniqueName: \"kubernetes.io/projected/c04c0bd2-e6c2-42a0-949b-8f818987dd6b-kube-api-access-hmmm2\") pod \"dns-default-ltl6z\" (UID: \"c04c0bd2-e6c2-42a0-949b-8f818987dd6b\") " pod="openshift-dns/dns-default-ltl6z"
May 25 13:05:34.504196 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:34.504105    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c04c0bd2-e6c2-42a0-949b-8f818987dd6b-config-volume\") pod \"dns-default-ltl6z\" (UID: \"c04c0bd2-e6c2-42a0-949b-8f818987dd6b\") " pod="openshift-dns/dns-default-ltl6z"
May 25 13:05:34.504196 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:34.504141    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c04c0bd2-e6c2-42a0-949b-8f818987dd6b-tmp-dir\") pod \"dns-default-ltl6z\" (UID: \"c04c0bd2-e6c2-42a0-949b-8f818987dd6b\") " pod="openshift-dns/dns-default-ltl6z"
May 25 13:05:34.504196 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:34.504171    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2e9a6623-9ccb-484c-b669-38caefdf2976-cert\") pod \"ingress-canary-8gtmh\" (UID: \"2e9a6623-9ccb-484c-b669-38caefdf2976\") " pod="openshift-ingress-canary/ingress-canary-8gtmh"
May 25 13:05:34.504459 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:34.504207    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-s5v77\" (UniqueName: \"kubernetes.io/projected/2e9a6623-9ccb-484c-b669-38caefdf2976-kube-api-access-s5v77\") pod \"ingress-canary-8gtmh\" (UID: \"2e9a6623-9ccb-484c-b669-38caefdf2976\") " pod="openshift-ingress-canary/ingress-canary-8gtmh"
May 25 13:05:34.504459 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:05:34.504284    2580 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 25 13:05:34.504459 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:05:34.504345    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e9a6623-9ccb-484c-b669-38caefdf2976-cert podName:2e9a6623-9ccb-484c-b669-38caefdf2976 nodeName:}" failed. No retries permitted until 2026-05-25 13:05:35.004324651 +0000 UTC m=+35.196619006 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/2e9a6623-9ccb-484c-b669-38caefdf2976-cert") pod "ingress-canary-8gtmh" (UID: "2e9a6623-9ccb-484c-b669-38caefdf2976") : secret "canary-serving-cert" not found
May 25 13:05:34.504459 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:34.504397    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c04c0bd2-e6c2-42a0-949b-8f818987dd6b-metrics-tls\") pod \"dns-default-ltl6z\" (UID: \"c04c0bd2-e6c2-42a0-949b-8f818987dd6b\") " pod="openshift-dns/dns-default-ltl6z"
May 25 13:05:34.504459 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:34.504445    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-hmmm2\" (UniqueName: \"kubernetes.io/projected/c04c0bd2-e6c2-42a0-949b-8f818987dd6b-kube-api-access-hmmm2\") pod \"dns-default-ltl6z\" (UID: \"c04c0bd2-e6c2-42a0-949b-8f818987dd6b\") " pod="openshift-dns/dns-default-ltl6z"
May 25 13:05:34.504695 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:05:34.504503    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 25 13:05:34.504695 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:05:34.504544    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c04c0bd2-e6c2-42a0-949b-8f818987dd6b-metrics-tls podName:c04c0bd2-e6c2-42a0-949b-8f818987dd6b nodeName:}" failed. No retries permitted until 2026-05-25 13:05:35.004530836 +0000 UTC m=+35.196825180 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c04c0bd2-e6c2-42a0-949b-8f818987dd6b-metrics-tls") pod "dns-default-ltl6z" (UID: "c04c0bd2-e6c2-42a0-949b-8f818987dd6b") : secret "dns-default-metrics-tls" not found
May 25 13:05:34.504695 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:34.504591    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c04c0bd2-e6c2-42a0-949b-8f818987dd6b-tmp-dir\") pod \"dns-default-ltl6z\" (UID: \"c04c0bd2-e6c2-42a0-949b-8f818987dd6b\") " pod="openshift-dns/dns-default-ltl6z"
May 25 13:05:34.504854 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:34.504833    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c04c0bd2-e6c2-42a0-949b-8f818987dd6b-config-volume\") pod \"dns-default-ltl6z\" (UID: \"c04c0bd2-e6c2-42a0-949b-8f818987dd6b\") " pod="openshift-dns/dns-default-ltl6z"
May 25 13:05:34.515908 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:34.515780    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5v77\" (UniqueName: \"kubernetes.io/projected/2e9a6623-9ccb-484c-b669-38caefdf2976-kube-api-access-s5v77\") pod \"ingress-canary-8gtmh\" (UID: \"2e9a6623-9ccb-484c-b669-38caefdf2976\") " pod="openshift-ingress-canary/ingress-canary-8gtmh"
May 25 13:05:34.518087 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:34.518061    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmmm2\" (UniqueName: \"kubernetes.io/projected/c04c0bd2-e6c2-42a0-949b-8f818987dd6b-kube-api-access-hmmm2\") pod \"dns-default-ltl6z\" (UID: \"c04c0bd2-e6c2-42a0-949b-8f818987dd6b\") " pod="openshift-dns/dns-default-ltl6z"
May 25 13:05:35.008001 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:35.007959    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2e9a6623-9ccb-484c-b669-38caefdf2976-cert\") pod \"ingress-canary-8gtmh\" (UID: \"2e9a6623-9ccb-484c-b669-38caefdf2976\") " pod="openshift-ingress-canary/ingress-canary-8gtmh"
May 25 13:05:35.008001 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:35.008006    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c04c0bd2-e6c2-42a0-949b-8f818987dd6b-metrics-tls\") pod \"dns-default-ltl6z\" (UID: \"c04c0bd2-e6c2-42a0-949b-8f818987dd6b\") " pod="openshift-dns/dns-default-ltl6z"
May 25 13:05:35.008476 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:05:35.008112    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 25 13:05:35.008476 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:05:35.008122    2580 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 25 13:05:35.008476 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:05:35.008187    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c04c0bd2-e6c2-42a0-949b-8f818987dd6b-metrics-tls podName:c04c0bd2-e6c2-42a0-949b-8f818987dd6b nodeName:}" failed. No retries permitted until 2026-05-25 13:05:36.008173261 +0000 UTC m=+36.200467599 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c04c0bd2-e6c2-42a0-949b-8f818987dd6b-metrics-tls") pod "dns-default-ltl6z" (UID: "c04c0bd2-e6c2-42a0-949b-8f818987dd6b") : secret "dns-default-metrics-tls" not found
May 25 13:05:35.008476 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:05:35.008202    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e9a6623-9ccb-484c-b669-38caefdf2976-cert podName:2e9a6623-9ccb-484c-b669-38caefdf2976 nodeName:}" failed. No retries permitted until 2026-05-25 13:05:36.00819604 +0000 UTC m=+36.200490378 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/2e9a6623-9ccb-484c-b669-38caefdf2976-cert") pod "ingress-canary-8gtmh" (UID: "2e9a6623-9ccb-484c-b669-38caefdf2976") : secret "canary-serving-cert" not found
May 25 13:05:35.288458 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:35.288388    2580 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6f7545f5cb-h8cph"]
May 25 13:05:35.314207 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:35.314182    2580 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6f7545f5cb-h8cph"]
May 25 13:05:35.314402 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:35.314357    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6f7545f5cb-h8cph"
May 25 13:05:35.316730 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:35.316705    2580 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
May 25 13:05:35.316845 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:35.316710    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-mrk8m\""
May 25 13:05:35.317559 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:35.317529    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
May 25 13:05:35.317733 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:35.317633    2580 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
May 25 13:05:35.317733 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:35.317705    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
May 25 13:05:35.411742 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:35.411712    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1a680b18-bede-464c-a3ba-c5eb503d8166-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6f7545f5cb-h8cph\" (UID: \"1a680b18-bede-464c-a3ba-c5eb503d8166\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6f7545f5cb-h8cph"
May 25 13:05:35.411935 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:35.411761    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqnsn\" (UniqueName: \"kubernetes.io/projected/1a680b18-bede-464c-a3ba-c5eb503d8166-kube-api-access-vqnsn\") pod \"managed-serviceaccount-addon-agent-6f7545f5cb-h8cph\" (UID: \"1a680b18-bede-464c-a3ba-c5eb503d8166\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6f7545f5cb-h8cph"
May 25 13:05:35.512455 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:35.512415    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1a680b18-bede-464c-a3ba-c5eb503d8166-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6f7545f5cb-h8cph\" (UID: \"1a680b18-bede-464c-a3ba-c5eb503d8166\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6f7545f5cb-h8cph"
May 25 13:05:35.512629 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:35.512495    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-vqnsn\" (UniqueName: \"kubernetes.io/projected/1a680b18-bede-464c-a3ba-c5eb503d8166-kube-api-access-vqnsn\") pod \"managed-serviceaccount-addon-agent-6f7545f5cb-h8cph\" (UID: \"1a680b18-bede-464c-a3ba-c5eb503d8166\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6f7545f5cb-h8cph"
May 25 13:05:35.515898 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:35.515871    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1a680b18-bede-464c-a3ba-c5eb503d8166-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6f7545f5cb-h8cph\" (UID: \"1a680b18-bede-464c-a3ba-c5eb503d8166\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6f7545f5cb-h8cph"
May 25 13:05:35.525600 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:35.525575    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqnsn\" (UniqueName: \"kubernetes.io/projected/1a680b18-bede-464c-a3ba-c5eb503d8166-kube-api-access-vqnsn\") pod \"managed-serviceaccount-addon-agent-6f7545f5cb-h8cph\" (UID: \"1a680b18-bede-464c-a3ba-c5eb503d8166\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6f7545f5cb-h8cph"
May 25 13:05:35.630849 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:35.630811    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6f7545f5cb-h8cph"
May 25 13:05:35.798688 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:35.798652    2580 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6f7545f5cb-h8cph"]
May 25 13:05:36.016434 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:36.016294    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c04c0bd2-e6c2-42a0-949b-8f818987dd6b-metrics-tls\") pod \"dns-default-ltl6z\" (UID: \"c04c0bd2-e6c2-42a0-949b-8f818987dd6b\") " pod="openshift-dns/dns-default-ltl6z"
May 25 13:05:36.016434 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:36.016430    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2e9a6623-9ccb-484c-b669-38caefdf2976-cert\") pod \"ingress-canary-8gtmh\" (UID: \"2e9a6623-9ccb-484c-b669-38caefdf2976\") " pod="openshift-ingress-canary/ingress-canary-8gtmh"
May 25 13:05:36.016970 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:05:36.016501    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 25 13:05:36.016970 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:05:36.016548    2580 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 25 13:05:36.016970 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:05:36.016594    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c04c0bd2-e6c2-42a0-949b-8f818987dd6b-metrics-tls podName:c04c0bd2-e6c2-42a0-949b-8f818987dd6b nodeName:}" failed. No retries permitted until 2026-05-25 13:05:38.016570689 +0000 UTC m=+38.208865042 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c04c0bd2-e6c2-42a0-949b-8f818987dd6b-metrics-tls") pod "dns-default-ltl6z" (UID: "c04c0bd2-e6c2-42a0-949b-8f818987dd6b") : secret "dns-default-metrics-tls" not found
May 25 13:05:36.016970 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:05:36.016614    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e9a6623-9ccb-484c-b669-38caefdf2976-cert podName:2e9a6623-9ccb-484c-b669-38caefdf2976 nodeName:}" failed. No retries permitted until 2026-05-25 13:05:38.016604946 +0000 UTC m=+38.208899285 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/2e9a6623-9ccb-484c-b669-38caefdf2976-cert") pod "ingress-canary-8gtmh" (UID: "2e9a6623-9ccb-484c-b669-38caefdf2976") : secret "canary-serving-cert" not found
May 25 13:05:37.902511 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:05:37.902468    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1a680b18_bede_464c_a3ba_c5eb503d8166.slice/crio-c156198df3f7a9d74980f68a0854b9b09f9bd558ae5bd600f1ed6cd26fa42975 WatchSource:0}: Error finding container c156198df3f7a9d74980f68a0854b9b09f9bd558ae5bd600f1ed6cd26fa42975: Status 404 returned error can't find the container with id c156198df3f7a9d74980f68a0854b9b09f9bd558ae5bd600f1ed6cd26fa42975
May 25 13:05:38.031243 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:38.031219    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2e9a6623-9ccb-484c-b669-38caefdf2976-cert\") pod \"ingress-canary-8gtmh\" (UID: \"2e9a6623-9ccb-484c-b669-38caefdf2976\") " pod="openshift-ingress-canary/ingress-canary-8gtmh"
May 25 13:05:38.031351 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:38.031258    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c04c0bd2-e6c2-42a0-949b-8f818987dd6b-metrics-tls\") pod \"dns-default-ltl6z\" (UID: \"c04c0bd2-e6c2-42a0-949b-8f818987dd6b\") " pod="openshift-dns/dns-default-ltl6z"
May 25 13:05:38.031459 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:05:38.031398    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 25 13:05:38.031459 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:05:38.031411    2580 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 25 13:05:38.031459 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:05:38.031447    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c04c0bd2-e6c2-42a0-949b-8f818987dd6b-metrics-tls podName:c04c0bd2-e6c2-42a0-949b-8f818987dd6b nodeName:}" failed. No retries permitted until 2026-05-25 13:05:42.031434385 +0000 UTC m=+42.223728723 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c04c0bd2-e6c2-42a0-949b-8f818987dd6b-metrics-tls") pod "dns-default-ltl6z" (UID: "c04c0bd2-e6c2-42a0-949b-8f818987dd6b") : secret "dns-default-metrics-tls" not found
May 25 13:05:38.031568 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:05:38.031471    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e9a6623-9ccb-484c-b669-38caefdf2976-cert podName:2e9a6623-9ccb-484c-b669-38caefdf2976 nodeName:}" failed. No retries permitted until 2026-05-25 13:05:42.031453189 +0000 UTC m=+42.223747541 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/2e9a6623-9ccb-484c-b669-38caefdf2976-cert") pod "ingress-canary-8gtmh" (UID: "2e9a6623-9ccb-484c-b669-38caefdf2976") : secret "canary-serving-cert" not found
May 25 13:05:38.597439 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:38.597404    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6f7545f5cb-h8cph" event={"ID":"1a680b18-bede-464c-a3ba-c5eb503d8166","Type":"ContainerStarted","Data":"c156198df3f7a9d74980f68a0854b9b09f9bd558ae5bd600f1ed6cd26fa42975"}
May 25 13:05:38.599485 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:38.599459    2580 generic.go:358] "Generic (PLEG): container finished" podID="33d153fd-31f6-4481-855e-728e06ce70f5" containerID="b96e7e74ad278dbd71ac81930e879a62c5a4a6f2d9b9ba62666bdcf85b481491" exitCode=0
May 25 13:05:38.599609 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:38.599494    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hj74k" event={"ID":"33d153fd-31f6-4481-855e-728e06ce70f5","Type":"ContainerDied","Data":"b96e7e74ad278dbd71ac81930e879a62c5a4a6f2d9b9ba62666bdcf85b481491"}
May 25 13:05:39.604469 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:39.604435    2580 generic.go:358] "Generic (PLEG): container finished" podID="33d153fd-31f6-4481-855e-728e06ce70f5" containerID="47edf4d273e2d4d6b6b5555fae5a72dc00360d104fc7b945ecfa0fa4b375da34" exitCode=0
May 25 13:05:39.604827 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:39.604488    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hj74k" event={"ID":"33d153fd-31f6-4481-855e-728e06ce70f5","Type":"ContainerDied","Data":"47edf4d273e2d4d6b6b5555fae5a72dc00360d104fc7b945ecfa0fa4b375da34"}
May 25 13:05:40.608935 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:40.608896    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hj74k" event={"ID":"33d153fd-31f6-4481-855e-728e06ce70f5","Type":"ContainerStarted","Data":"2b3cc4c5eabd03c8260979b6b0f94c6ff5ea64dd79e8aedc09fa4087fa33a63b"}
May 25 13:05:40.635471 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:40.635321    2580 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-hj74k" podStartSLOduration=5.60844277 podStartE2EDuration="40.63530731s" podCreationTimestamp="2026-05-25 13:05:00 +0000 UTC" firstStartedPulling="2026-05-25 13:05:02.92308402 +0000 UTC m=+3.115378358" lastFinishedPulling="2026-05-25 13:05:37.949948543 +0000 UTC m=+38.142242898" observedRunningTime="2026-05-25 13:05:40.634108855 +0000 UTC m=+40.826403215" watchObservedRunningTime="2026-05-25 13:05:40.63530731 +0000 UTC m=+40.827601669"
May 25 13:05:41.612152 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:41.612112    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6f7545f5cb-h8cph" event={"ID":"1a680b18-bede-464c-a3ba-c5eb503d8166","Type":"ContainerStarted","Data":"e727e99294174dd1f1688b61ae6133d3ef09762850e5ea9dee333b3e51029337"}
May 25 13:05:41.627161 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:41.627111    2580 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6f7545f5cb-h8cph" podStartSLOduration=3.543385497 podStartE2EDuration="6.627097777s" podCreationTimestamp="2026-05-25 13:05:35 +0000 UTC" firstStartedPulling="2026-05-25 13:05:37.904297902 +0000 UTC m=+38.096592241" lastFinishedPulling="2026-05-25 13:05:40.988010183 +0000 UTC m=+41.180304521" observedRunningTime="2026-05-25 13:05:41.626723202 +0000 UTC m=+41.819017594" watchObservedRunningTime="2026-05-25 13:05:41.627097777 +0000 UTC m=+41.819392136"
May 25 13:05:42.060767 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:42.060662    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2e9a6623-9ccb-484c-b669-38caefdf2976-cert\") pod \"ingress-canary-8gtmh\" (UID: \"2e9a6623-9ccb-484c-b669-38caefdf2976\") " pod="openshift-ingress-canary/ingress-canary-8gtmh"
May 25 13:05:42.060767 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:42.060724    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c04c0bd2-e6c2-42a0-949b-8f818987dd6b-metrics-tls\") pod \"dns-default-ltl6z\" (UID: \"c04c0bd2-e6c2-42a0-949b-8f818987dd6b\") " pod="openshift-dns/dns-default-ltl6z"
May 25 13:05:42.060955 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:05:42.060821    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 25 13:05:42.060955 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:05:42.060822    2580 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 25 13:05:42.060955 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:05:42.060878    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c04c0bd2-e6c2-42a0-949b-8f818987dd6b-metrics-tls podName:c04c0bd2-e6c2-42a0-949b-8f818987dd6b nodeName:}" failed. No retries permitted until 2026-05-25 13:05:50.060861885 +0000 UTC m=+50.253156223 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c04c0bd2-e6c2-42a0-949b-8f818987dd6b-metrics-tls") pod "dns-default-ltl6z" (UID: "c04c0bd2-e6c2-42a0-949b-8f818987dd6b") : secret "dns-default-metrics-tls" not found
May 25 13:05:42.060955 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:05:42.060890    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e9a6623-9ccb-484c-b669-38caefdf2976-cert podName:2e9a6623-9ccb-484c-b669-38caefdf2976 nodeName:}" failed. No retries permitted until 2026-05-25 13:05:50.060884813 +0000 UTC m=+50.253179151 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/2e9a6623-9ccb-484c-b669-38caefdf2976-cert") pod "ingress-canary-8gtmh" (UID: "2e9a6623-9ccb-484c-b669-38caefdf2976") : secret "canary-serving-cert" not found
May 25 13:05:50.115687 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:50.115647    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2e9a6623-9ccb-484c-b669-38caefdf2976-cert\") pod \"ingress-canary-8gtmh\" (UID: \"2e9a6623-9ccb-484c-b669-38caefdf2976\") " pod="openshift-ingress-canary/ingress-canary-8gtmh"
May 25 13:05:50.115687 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:05:50.115692    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c04c0bd2-e6c2-42a0-949b-8f818987dd6b-metrics-tls\") pod \"dns-default-ltl6z\" (UID: \"c04c0bd2-e6c2-42a0-949b-8f818987dd6b\") " pod="openshift-dns/dns-default-ltl6z"
May 25 13:05:50.116180 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:05:50.115797    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 25 13:05:50.116180 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:05:50.115801    2580 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 25 13:05:50.116180 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:05:50.115849    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c04c0bd2-e6c2-42a0-949b-8f818987dd6b-metrics-tls podName:c04c0bd2-e6c2-42a0-949b-8f818987dd6b nodeName:}" failed. No retries permitted until 2026-05-25 13:06:06.115834982 +0000 UTC m=+66.308129324 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c04c0bd2-e6c2-42a0-949b-8f818987dd6b-metrics-tls") pod "dns-default-ltl6z" (UID: "c04c0bd2-e6c2-42a0-949b-8f818987dd6b") : secret "dns-default-metrics-tls" not found
May 25 13:05:50.116180 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:05:50.115862    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e9a6623-9ccb-484c-b669-38caefdf2976-cert podName:2e9a6623-9ccb-484c-b669-38caefdf2976 nodeName:}" failed. No retries permitted until 2026-05-25 13:06:06.115856521 +0000 UTC m=+66.308150858 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/2e9a6623-9ccb-484c-b669-38caefdf2976-cert") pod "ingress-canary-8gtmh" (UID: "2e9a6623-9ccb-484c-b669-38caefdf2976") : secret "canary-serving-cert" not found
May 25 13:06:00.591762 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:06:00.591728    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-646r6"
May 25 13:06:06.029659 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:06:06.029597    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-lfgrd\" (UniqueName: \"kubernetes.io/projected/fdd5eb93-deb6-441f-b1fe-3511c4eee77e-kube-api-access-lfgrd\") pod \"network-check-target-25spc\" (UID: \"fdd5eb93-deb6-441f-b1fe-3511c4eee77e\") " pod="openshift-network-diagnostics/network-check-target-25spc"
May 25 13:06:06.030067 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:06:06.029694    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e445efe1-4ac4-4df5-881f-748bcf8c5e0f-metrics-certs\") pod \"network-metrics-daemon-svlzs\" (UID: \"e445efe1-4ac4-4df5-881f-748bcf8c5e0f\") " pod="openshift-multus/network-metrics-daemon-svlzs"
May 25 13:06:06.032062 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:06:06.032042    2580 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
May 25 13:06:06.032116 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:06:06.032064    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
May 25 13:06:06.039865 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:06:06.039845    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
May 25 13:06:06.040039 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:06:06.039910    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e445efe1-4ac4-4df5-881f-748bcf8c5e0f-metrics-certs podName:e445efe1-4ac4-4df5-881f-748bcf8c5e0f nodeName:}" failed. No retries permitted until 2026-05-25 13:07:10.039892087 +0000 UTC m=+130.232186425 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e445efe1-4ac4-4df5-881f-748bcf8c5e0f-metrics-certs") pod "network-metrics-daemon-svlzs" (UID: "e445efe1-4ac4-4df5-881f-748bcf8c5e0f") : secret "metrics-daemon-secret" not found
May 25 13:06:06.042017 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:06:06.041998    2580 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
May 25 13:06:06.054380 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:06:06.054343    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfgrd\" (UniqueName: \"kubernetes.io/projected/fdd5eb93-deb6-441f-b1fe-3511c4eee77e-kube-api-access-lfgrd\") pod \"network-check-target-25spc\" (UID: \"fdd5eb93-deb6-441f-b1fe-3511c4eee77e\") " pod="openshift-network-diagnostics/network-check-target-25spc"
May 25 13:06:06.130689 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:06:06.130654    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2e9a6623-9ccb-484c-b669-38caefdf2976-cert\") pod \"ingress-canary-8gtmh\" (UID: \"2e9a6623-9ccb-484c-b669-38caefdf2976\") " pod="openshift-ingress-canary/ingress-canary-8gtmh"
May 25 13:06:06.130859 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:06:06.130699    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c04c0bd2-e6c2-42a0-949b-8f818987dd6b-metrics-tls\") pod \"dns-default-ltl6z\" (UID: \"c04c0bd2-e6c2-42a0-949b-8f818987dd6b\") " pod="openshift-dns/dns-default-ltl6z"
May 25 13:06:06.130859 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:06:06.130806    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 25 13:06:06.130859 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:06:06.130828    2580 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 25 13:06:06.130859 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:06:06.130855    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c04c0bd2-e6c2-42a0-949b-8f818987dd6b-metrics-tls podName:c04c0bd2-e6c2-42a0-949b-8f818987dd6b nodeName:}" failed. No retries permitted until 2026-05-25 13:06:38.130842573 +0000 UTC m=+98.323136911 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c04c0bd2-e6c2-42a0-949b-8f818987dd6b-metrics-tls") pod "dns-default-ltl6z" (UID: "c04c0bd2-e6c2-42a0-949b-8f818987dd6b") : secret "dns-default-metrics-tls" not found
May 25 13:06:06.131037 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:06:06.130892    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e9a6623-9ccb-484c-b669-38caefdf2976-cert podName:2e9a6623-9ccb-484c-b669-38caefdf2976 nodeName:}" failed. No retries permitted until 2026-05-25 13:06:38.130873136 +0000 UTC m=+98.323167476 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/2e9a6623-9ccb-484c-b669-38caefdf2976-cert") pod "ingress-canary-8gtmh" (UID: "2e9a6623-9ccb-484c-b669-38caefdf2976") : secret "canary-serving-cert" not found
May 25 13:06:06.193165 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:06:06.193141    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-cw8xk\""
May 25 13:06:06.201264 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:06:06.201243    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-25spc"
May 25 13:06:06.319273 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:06:06.319191    2580 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-25spc"]
May 25 13:06:06.322808 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:06:06.322780    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdd5eb93_deb6_441f_b1fe_3511c4eee77e.slice/crio-601a93818008cc7f4c03245fa64f9caf0cd913758c41488dda1cbcae52cb6453 WatchSource:0}: Error finding container 601a93818008cc7f4c03245fa64f9caf0cd913758c41488dda1cbcae52cb6453: Status 404 returned error can't find the container with id 601a93818008cc7f4c03245fa64f9caf0cd913758c41488dda1cbcae52cb6453
May 25 13:06:06.659149 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:06:06.659113    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-25spc" event={"ID":"fdd5eb93-deb6-441f-b1fe-3511c4eee77e","Type":"ContainerStarted","Data":"601a93818008cc7f4c03245fa64f9caf0cd913758c41488dda1cbcae52cb6453"}
May 25 13:06:09.666170 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:06:09.666134    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-25spc" event={"ID":"fdd5eb93-deb6-441f-b1fe-3511c4eee77e","Type":"ContainerStarted","Data":"bb4dd8dca587e8f2a405a838a75682aa8aa0ab11124bc12d88a16ae4f5bf3d21"}
May 25 13:06:09.666638 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:06:09.666338    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-25spc"
May 25 13:06:09.686838 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:06:09.686781    2580 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-25spc" podStartSLOduration=66.884903933 podStartE2EDuration="1m9.686766776s" podCreationTimestamp="2026-05-25 13:05:00 +0000 UTC" firstStartedPulling="2026-05-25 13:06:06.324559004 +0000 UTC m=+66.516853342" lastFinishedPulling="2026-05-25 13:06:09.126421843 +0000 UTC m=+69.318716185" observedRunningTime="2026-05-25 13:06:09.68662684 +0000 UTC m=+69.878921212" watchObservedRunningTime="2026-05-25 13:06:09.686766776 +0000 UTC m=+69.879061163"
May 25 13:06:35.665834 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:06:35.665699    2580 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-549b58d588-dnld2"]
May 25 13:06:35.668162 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:06:35.668142    2580 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-769864757b-hs9k5"]
May 25 13:06:35.668256 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:06:35.668238    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-549b58d588-dnld2"
May 25 13:06:35.669949 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:06:35.669928    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-769864757b-hs9k5"
May 25 13:06:35.670472 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:06:35.670443    2580 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
May 25 13:06:35.670584 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:06:35.670483    2580 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
May 25 13:06:35.670750 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:06:35.670735    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-8b5l2\""
May 25 13:06:35.671328 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:06:35.671313    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
May 25 13:06:35.671909 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:06:35.671890    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
May 25 13:06:35.671993 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:06:35.671968    2580 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
May 25 13:06:35.672321 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:06:35.672307    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
May 25 13:06:35.673995 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:06:35.673978    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
May 25 13:06:35.674590 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:06:35.674573    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
May 25 13:06:35.674679 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:06:35.674639    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-6tjlh\""
May 25 13:06:35.674861 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:06:35.674844    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
May 25 13:06:35.690893 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:06:35.690861    2580 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
May 25 13:06:35.693021 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:06:35.692996    2580 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-549b58d588-dnld2"]
May 25 13:06:35.693761 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:06:35.693733    2580 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-769864757b-hs9k5"]
May 25 13:06:35.729002 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:06:35.728972    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glbcm\" (UniqueName: \"kubernetes.io/projected/264a8628-6305-4840-9925-88aeafe69834-kube-api-access-glbcm\") pod \"router-default-549b58d588-dnld2\" (UID: \"264a8628-6305-4840-9925-88aeafe69834\") " pod="openshift-ingress/router-default-549b58d588-dnld2"
May 25 13:06:35.729138 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:06:35.729021    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/264a8628-6305-4840-9925-88aeafe69834-stats-auth\") pod \"router-default-549b58d588-dnld2\" (UID: \"264a8628-6305-4840-9925-88aeafe69834\") " pod="openshift-ingress/router-default-549b58d588-dnld2"
May 25 13:06:35.729138 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:06:35.729050    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f21589b0-1b80-46bb-92e8-1252201ed8f3-registry-tls\") pod \"image-registry-769864757b-hs9k5\" (UID: \"f21589b0-1b80-46bb-92e8-1252201ed8f3\") " pod="openshift-image-registry/image-registry-769864757b-hs9k5"
May 25 13:06:35.729138 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:06:35.729073    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f21589b0-1b80-46bb-92e8-1252201ed8f3-bound-sa-token\") pod \"image-registry-769864757b-hs9k5\" (UID: \"f21589b0-1b80-46bb-92e8-1252201ed8f3\") " pod="openshift-image-registry/image-registry-769864757b-hs9k5"
May 25 13:06:35.729138 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:06:35.729089    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwjmn\" (UniqueName: \"kubernetes.io/projected/f21589b0-1b80-46bb-92e8-1252201ed8f3-kube-api-access-vwjmn\") pod \"image-registry-769864757b-hs9k5\" (UID: \"f21589b0-1b80-46bb-92e8-1252201ed8f3\") " pod="openshift-image-registry/image-registry-769864757b-hs9k5"
May 25 13:06:35.729138 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:06:35.729110    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/264a8628-6305-4840-9925-88aeafe69834-metrics-certs\") pod \"router-default-549b58d588-dnld2\" (UID: \"264a8628-6305-4840-9925-88aeafe69834\") " pod="openshift-ingress/router-default-549b58d588-dnld2"
May 25 13:06:35.729138 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:06:35.729136    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/264a8628-6305-4840-9925-88aeafe69834-default-certificate\") pod \"router-default-549b58d588-dnld2\" (UID: \"264a8628-6305-4840-9925-88aeafe69834\") " pod="openshift-ingress/router-default-549b58d588-dnld2"
May 25 13:06:35.729330 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:06:35.729160    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f21589b0-1b80-46bb-92e8-1252201ed8f3-image-registry-private-configuration\") pod \"image-registry-769864757b-hs9k5\" (UID: \"f21589b0-1b80-46bb-92e8-1252201ed8f3\") " pod="openshift-image-registry/image-registry-769864757b-hs9k5"
May 25 13:06:35.729330 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:06:35.729179    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f21589b0-1b80-46bb-92e8-1252201ed8f3-registry-certificates\") pod \"image-registry-769864757b-hs9k5\" (UID: \"f21589b0-1b80-46bb-92e8-1252201ed8f3\") " pod="openshift-image-registry/image-registry-769864757b-hs9k5"
May 25 13:06:35.729330 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:06:35.729218    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/264a8628-6305-4840-9925-88aeafe69834-service-ca-bundle\") pod \"router-default-549b58d588-dnld2\" (UID: \"264a8628-6305-4840-9925-88aeafe69834\") " pod="openshift-ingress/router-default-549b58d588-dnld2"
May 25 13:06:35.729330 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:06:35.729239    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f21589b0-1b80-46bb-92e8-1252201ed8f3-ca-trust-extracted\") pod \"image-registry-769864757b-hs9k5\" (UID: \"f21589b0-1b80-46bb-92e8-1252201ed8f3\") " pod="openshift-image-registry/image-registry-769864757b-hs9k5"
May 25 13:06:35.729330 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:06:35.729257    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f21589b0-1b80-46bb-92e8-1252201ed8f3-trusted-ca\") pod \"image-registry-769864757b-hs9k5\" (UID: \"f21589b0-1b80-46bb-92e8-1252201ed8f3\") " pod="openshift-image-registry/image-registry-769864757b-hs9k5"
May 25 13:06:35.729330 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:06:35.729272    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f21589b0-1b80-46bb-92e8-1252201ed8f3-installation-pull-secrets\") pod \"image-registry-769864757b-hs9k5\" (UID: \"f21589b0-1b80-46bb-92e8-1252201ed8f3\") " pod="openshift-image-registry/image-registry-769864757b-hs9k5"
May 25 13:06:35.772938 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:06:35.772901    2580 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57b5b68b8-rh7nv"]
May 25 13:06:35.775119 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:06:35.775097    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-rh7nv"
May 25 13:06:35.776857 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:06:35.776831    2580 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-78bdfb888c-tnkbv"]
May 25 13:06:35.777673 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:06:35.777647    2580 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
May 25 13:06:35.777789 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:06:35.777743    2580 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
May 25 13:06:35.777967 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:06:35.777951    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-75bst\""
May 25 13:06:35.778087 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:06:35.778067    2580 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
May 25 13:06:35.778392 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:06:35.778378    2580 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7f44b6dd87-tmx95"]
May 25 13:06:35.778578 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:06:35.778451    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-78bdfb888c-tnkbv"
May 25 13:06:35.779928 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:06:35.779914    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7f44b6dd87-tmx95"
May 25 13:06:35.781433 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:06:35.781413    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
May 25 13:06:35.781737 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:06:35.781719    2580 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
May 25 13:06:35.781822 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:06:35.781741    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
May 25 13:06:35.781822 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:06:35.781794    2580 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
May 25 13:06:35.781935 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:06:35.781837    2580 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
May 25 13:06:35.782472 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:06:35.782448    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-crwtw\""
May 25 13:06:35.783040 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:06:35.783025    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-2rxr4\""
May 25 13:06:35.786712 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:06:35.786692    2580 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
May 25 13:06:35.792889 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:06:35.792868    2580 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57b5b68b8-rh7nv"]
May 25 13:06:35.793670 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:06:35.793651    2580 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7f44b6dd87-tmx95"]
May 25 13:06:35.794547 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:06:35.794528    2580 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-78bdfb888c-tnkbv"]
May 25 13:06:35.830119 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:06:35.830086    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cf32b96d-4371-4b92-89db-ff71111c977a-trusted-ca-bundle\") pod \"insights-operator-78bdfb888c-tnkbv\" (UID: \"cf32b96d-4371-4b92-89db-ff71111c977a\") " pod="openshift-insights/insights-operator-78bdfb888c-tnkbv"
May 25 13:06:35.830119 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:06:35.830125    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/518b0e48-f2ac-4401-9212-834cbad021f9-config\") pod \"service-ca-operator-57b5b68b8-rh7nv\" (UID: \"518b0e48-f2ac-4401-9212-834cbad021f9\") " pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-rh7nv"
May 25 13:06:35.830331 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:06:35.830187    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f21589b0-1b80-46bb-92e8-1252201ed8f3-installation-pull-secrets\") pod \"image-registry-769864757b-hs9k5\" (UID: \"f21589b0-1b80-46bb-92e8-1252201ed8f3\") " pod="openshift-image-registry/image-registry-769864757b-hs9k5"
May 25 13:06:35.830331 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:06:35.830219    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5hw5\" (UniqueName: \"kubernetes.io/projected/2334e399-4548-4098-b14a-0e4b8a82e640-kube-api-access-l5hw5\") pod \"network-check-source-7f44b6dd87-tmx95\" (UID: \"2334e399-4548-4098-b14a-0e4b8a82e640\") " pod="openshift-network-diagnostics/network-check-source-7f44b6dd87-tmx95"
May 25 13:06:35.830331 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:06:35.830316    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/cf32b96d-4371-4b92-89db-ff71111c977a-snapshots\") pod \"insights-operator-78bdfb888c-tnkbv\" (UID: \"cf32b96d-4371-4b92-89db-ff71111c977a\") " pod="openshift-insights/insights-operator-78bdfb888c-tnkbv"
May 25 13:06:35.830525 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:06:35.830402    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/264a8628-6305-4840-9925-88aeafe69834-stats-auth\") pod \"router-default-549b58d588-dnld2\" (UID: \"264a8628-6305-4840-9925-88aeafe69834\") " pod="openshift-ingress/router-default-549b58d588-dnld2"
May 25 13:06:35.830525 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:06:35.830429    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f21589b0-1b80-46bb-92e8-1252201ed8f3-registry-tls\") pod \"image-registry-769864757b-hs9k5\" (UID: \"f21589b0-1b80-46bb-92e8-1252201ed8f3\") " pod="openshift-image-registry/image-registry-769864757b-hs9k5"
May 25 13:06:35.830525 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:06:35.830455    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-vwjmn\" (UniqueName: \"kubernetes.io/projected/f21589b0-1b80-46bb-92e8-1252201ed8f3-kube-api-access-vwjmn\") pod \"image-registry-769864757b-hs9k5\" (UID: \"f21589b0-1b80-46bb-92e8-1252201ed8f3\") " pod="openshift-image-registry/image-registry-769864757b-hs9k5"
May 25 13:06:35.830525 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:06:35.830508    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tn5b2\" (UniqueName: \"kubernetes.io/projected/cf32b96d-4371-4b92-89db-ff71111c977a-kube-api-access-tn5b2\") pod \"insights-operator-78bdfb888c-tnkbv\" (UID: \"cf32b96d-4371-4b92-89db-ff71111c977a\") " pod="openshift-insights/insights-operator-78bdfb888c-tnkbv"
May 25 13:06:35.830707 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:06:35.830535    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/518b0e48-f2ac-4401-9212-834cbad021f9-serving-cert\") pod \"service-ca-operator-57b5b68b8-rh7nv\" (UID: \"518b0e48-f2ac-4401-9212-834cbad021f9\") " pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-rh7nv"
May 25 13:06:35.830707 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:06:35.830563    2580 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 25 13:06:35.830707 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:06:35.830587    2580 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-769864757b-hs9k5: secret "image-registry-tls" not found
May 25 13:06:35.830707 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:06:35.830630    2580 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
May 25 13:06:35.830707 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:06:35.830571    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/264a8628-6305-4840-9925-88aeafe69834-metrics-certs\") pod \"router-default-549b58d588-dnld2\" (UID: \"264a8628-6305-4840-9925-88aeafe69834\") " pod="openshift-ingress/router-default-549b58d588-dnld2"
May 25 13:06:35.830707 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:06:35.830657    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f21589b0-1b80-46bb-92e8-1252201ed8f3-registry-tls podName:f21589b0-1b80-46bb-92e8-1252201ed8f3 nodeName:}" failed. No retries permitted until 2026-05-25 13:06:36.330637792 +0000 UTC m=+96.522932143 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f21589b0-1b80-46bb-92e8-1252201ed8f3-registry-tls") pod "image-registry-769864757b-hs9k5" (UID: "f21589b0-1b80-46bb-92e8-1252201ed8f3") : secret "image-registry-tls" not found
May 25 13:06:35.830707 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:06:35.830691    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/264a8628-6305-4840-9925-88aeafe69834-metrics-certs podName:264a8628-6305-4840-9925-88aeafe69834 nodeName:}" failed. No retries permitted until 2026-05-25 13:06:36.330671931 +0000 UTC m=+96.522966269 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/264a8628-6305-4840-9925-88aeafe69834-metrics-certs") pod "router-default-549b58d588-dnld2" (UID: "264a8628-6305-4840-9925-88aeafe69834") : secret "router-metrics-certs-default" not found
May 25 13:06:35.830707 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:06:35.830711    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/264a8628-6305-4840-9925-88aeafe69834-default-certificate\") pod \"router-default-549b58d588-dnld2\" (UID: \"264a8628-6305-4840-9925-88aeafe69834\") " pod="openshift-ingress/router-default-549b58d588-dnld2"
May 25 13:06:35.831124 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:06:35.830734    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f21589b0-1b80-46bb-92e8-1252201ed8f3-registry-certificates\") pod \"image-registry-769864757b-hs9k5\" (UID: \"f21589b0-1b80-46bb-92e8-1252201ed8f3\") " pod="openshift-image-registry/image-registry-769864757b-hs9k5"
May 25 13:06:35.831124 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:06:35.830767    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f21589b0-1b80-46bb-92e8-1252201ed8f3-trusted-ca\") pod \"image-registry-769864757b-hs9k5\" (UID: \"f21589b0-1b80-46bb-92e8-1252201ed8f3\") " pod="openshift-image-registry/image-registry-769864757b-hs9k5"
May 25 13:06:35.831124 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:06:35.830784    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sb24q\" (UniqueName: \"kubernetes.io/projected/518b0e48-f2ac-4401-9212-834cbad021f9-kube-api-access-sb24q\") pod \"service-ca-operator-57b5b68b8-rh7nv\" (UID: \"518b0e48-f2ac-4401-9212-834cbad021f9\") " pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-rh7nv"
May 25 13:06:35.831124 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:06:35.830803    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/264a8628-6305-4840-9925-88aeafe69834-service-ca-bundle\") pod \"router-default-549b58d588-dnld2\" (UID: \"264a8628-6305-4840-9925-88aeafe69834\") " pod="openshift-ingress/router-default-549b58d588-dnld2"
May 25 13:06:35.831124 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:06:35.830833    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f21589b0-1b80-46bb-92e8-1252201ed8f3-ca-trust-extracted\") pod \"image-registry-769864757b-hs9k5\" (UID: \"f21589b0-1b80-46bb-92e8-1252201ed8f3\") " pod="openshift-image-registry/image-registry-769864757b-hs9k5"
May 25 13:06:35.831124 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:06:35.830873    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-glbcm\" (UniqueName: \"kubernetes.io/projected/264a8628-6305-4840-9925-88aeafe69834-kube-api-access-glbcm\") pod \"router-default-549b58d588-dnld2\" (UID: \"264a8628-6305-4840-9925-88aeafe69834\") " pod="openshift-ingress/router-default-549b58d588-dnld2"
May 25 13:06:35.831124 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:06:35.830977    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/264a8628-6305-4840-9925-88aeafe69834-service-ca-bundle podName:264a8628-6305-4840-9925-88aeafe69834 nodeName:}" failed. No retries permitted until 2026-05-25 13:06:36.330958203 +0000 UTC m=+96.523252542 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/264a8628-6305-4840-9925-88aeafe69834-service-ca-bundle") pod "router-default-549b58d588-dnld2" (UID: "264a8628-6305-4840-9925-88aeafe69834") : configmap references non-existent config key: service-ca.crt
May 25 13:06:35.831555 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:06:35.831250    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f21589b0-1b80-46bb-92e8-1252201ed8f3-image-registry-private-configuration\") pod \"image-registry-769864757b-hs9k5\" (UID: \"f21589b0-1b80-46bb-92e8-1252201ed8f3\") " pod="openshift-image-registry/image-registry-769864757b-hs9k5"
May 25 13:06:35.831555 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:06:35.831291    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f21589b0-1b80-46bb-92e8-1252201ed8f3-bound-sa-token\") pod \"image-registry-769864757b-hs9k5\" (UID: \"f21589b0-1b80-46bb-92e8-1252201ed8f3\") " pod="openshift-image-registry/image-registry-769864757b-hs9k5"
May 25 13:06:35.831555 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:06:35.831338    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cf32b96d-4371-4b92-89db-ff71111c977a-tmp\") pod \"insights-operator-78bdfb888c-tnkbv\" (UID: \"cf32b96d-4371-4b92-89db-ff71111c977a\") " pod="openshift-insights/insights-operator-78bdfb888c-tnkbv"
May 25 13:06:35.831555 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:06:35.831253    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f21589b0-1b80-46bb-92e8-1252201ed8f3-ca-trust-extracted\") pod \"image-registry-769864757b-hs9k5\" (UID: \"f21589b0-1b80-46bb-92e8-1252201ed8f3\") " pod="openshift-image-registry/image-registry-769864757b-hs9k5"
May 25 13:06:35.831555 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:06:35.831409    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cf32b96d-4371-4b92-89db-ff71111c977a-service-ca-bundle\") pod \"insights-operator-78bdfb888c-tnkbv\" (UID: \"cf32b96d-4371-4b92-89db-ff71111c977a\") " pod="openshift-insights/insights-operator-78bdfb888c-tnkbv"
May 25 13:06:35.831917 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:06:35.831656    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f21589b0-1b80-46bb-92e8-1252201ed8f3-registry-certificates\") pod \"image-registry-769864757b-hs9k5\" (UID: \"f21589b0-1b80-46bb-92e8-1252201ed8f3\") " pod="openshift-image-registry/image-registry-769864757b-hs9k5"
May 25 13:06:35.831917 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:06:35.831788    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/cf32b96d-4371-4b92-89db-ff71111c977a-serving-cert\") pod \"insights-operator-78bdfb888c-tnkbv\" (UID: \"cf32b96d-4371-4b92-89db-ff71111c977a\") " pod="openshift-insights/insights-operator-78bdfb888c-tnkbv"
May 25 13:06:35.832118 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:06:35.832098    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f21589b0-1b80-46bb-92e8-1252201ed8f3-trusted-ca\") pod \"image-registry-769864757b-hs9k5\" (UID: \"f21589b0-1b80-46bb-92e8-1252201ed8f3\") " pod="openshift-image-registry/image-registry-769864757b-hs9k5"
May 25 13:06:35.833137 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:06:35.833116    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/264a8628-6305-4840-9925-88aeafe69834-stats-auth\") pod \"router-default-549b58d588-dnld2\" (UID: \"264a8628-6305-4840-9925-88aeafe69834\") " pod="openshift-ingress/router-default-549b58d588-dnld2"
May 25 13:06:35.833446 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:06:35.833424    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f21589b0-1b80-46bb-92e8-1252201ed8f3-installation-pull-secrets\") pod \"image-registry-769864757b-hs9k5\" (UID: \"f21589b0-1b80-46bb-92e8-1252201ed8f3\") " pod="openshift-image-registry/image-registry-769864757b-hs9k5"
May 25 13:06:35.833528 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:06:35.833468    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/264a8628-6305-4840-9925-88aeafe69834-default-certificate\") pod \"router-default-549b58d588-dnld2\" (UID: \"264a8628-6305-4840-9925-88aeafe69834\") " pod="openshift-ingress/router-default-549b58d588-dnld2"
May 25 13:06:35.834104 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:06:35.834085    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f21589b0-1b80-46bb-92e8-1252201ed8f3-image-registry-private-configuration\") pod \"image-registry-769864757b-hs9k5\" (UID: \"f21589b0-1b80-46bb-92e8-1252201ed8f3\") " pod="openshift-image-registry/image-registry-769864757b-hs9k5"
May 25 13:06:35.839811 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:06:35.839757    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f21589b0-1b80-46bb-92e8-1252201ed8f3-bound-sa-token\") pod \"image-registry-769864757b-hs9k5\" (UID: \"f21589b0-1b80-46bb-92e8-1252201ed8f3\") " pod="openshift-image-registry/image-registry-769864757b-hs9k5"
May 25 13:06:35.840045 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:06:35.840026    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwjmn\" (UniqueName: \"kubernetes.io/projected/f21589b0-1b80-46bb-92e8-1252201ed8f3-kube-api-access-vwjmn\") pod \"image-registry-769864757b-hs9k5\" (UID: \"f21589b0-1b80-46bb-92e8-1252201ed8f3\") " pod="openshift-image-registry/image-registry-769864757b-hs9k5"
May 25 13:06:35.840162 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:06:35.840146    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-glbcm\" (UniqueName: \"kubernetes.io/projected/264a8628-6305-4840-9925-88aeafe69834-kube-api-access-glbcm\") pod \"router-default-549b58d588-dnld2\" (UID: \"264a8628-6305-4840-9925-88aeafe69834\") " pod="openshift-ingress/router-default-549b58d588-dnld2"
May 25 13:06:35.932446 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:06:35.932330    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-l5hw5\" (UniqueName: \"kubernetes.io/projected/2334e399-4548-4098-b14a-0e4b8a82e640-kube-api-access-l5hw5\") pod \"network-check-source-7f44b6dd87-tmx95\" (UID: \"2334e399-4548-4098-b14a-0e4b8a82e640\") " pod="openshift-network-diagnostics/network-check-source-7f44b6dd87-tmx95"
May 25 13:06:35.932446 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:06:35.932414    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/cf32b96d-4371-4b92-89db-ff71111c977a-snapshots\") pod \"insights-operator-78bdfb888c-tnkbv\" (UID: \"cf32b96d-4371-4b92-89db-ff71111c977a\") " pod="openshift-insights/insights-operator-78bdfb888c-tnkbv"
May 25 13:06:35.932671 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:06:35.932463    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-tn5b2\" (UniqueName: \"kubernetes.io/projected/cf32b96d-4371-4b92-89db-ff71111c977a-kube-api-access-tn5b2\") pod \"insights-operator-78bdfb888c-tnkbv\" (UID: \"cf32b96d-4371-4b92-89db-ff71111c977a\") " pod="openshift-insights/insights-operator-78bdfb888c-tnkbv"
May 25 13:06:35.932671 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:06:35.932480    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/518b0e48-f2ac-4401-9212-834cbad021f9-serving-cert\") pod \"service-ca-operator-57b5b68b8-rh7nv\" (UID: \"518b0e48-f2ac-4401-9212-834cbad021f9\") " pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-rh7nv"
May 25 13:06:35.932671 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:06:35.932531    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-sb24q\" (UniqueName: \"kubernetes.io/projected/518b0e48-f2ac-4401-9212-834cbad021f9-kube-api-access-sb24q\") pod \"service-ca-operator-57b5b68b8-rh7nv\" (UID: \"518b0e48-f2ac-4401-9212-834cbad021f9\") " pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-rh7nv"
May 25 13:06:35.932671 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:06:35.932602    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cf32b96d-4371-4b92-89db-ff71111c977a-tmp\") pod \"insights-operator-78bdfb888c-tnkbv\" (UID: \"cf32b96d-4371-4b92-89db-ff71111c977a\") " pod="openshift-insights/insights-operator-78bdfb888c-tnkbv"
May 25 13:06:35.932671 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:06:35.932627    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cf32b96d-4371-4b92-89db-ff71111c977a-service-ca-bundle\") pod \"insights-operator-78bdfb888c-tnkbv\" (UID: \"cf32b96d-4371-4b92-89db-ff71111c977a\") " pod="openshift-insights/insights-operator-78bdfb888c-tnkbv"
May 25 13:06:35.932671 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:06:35.932650    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/cf32b96d-4371-4b92-89db-ff71111c977a-serving-cert\") pod \"insights-operator-78bdfb888c-tnkbv\" (UID: \"cf32b96d-4371-4b92-89db-ff71111c977a\") " pod="openshift-insights/insights-operator-78bdfb888c-tnkbv"
May 25 13:06:35.932935 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:06:35.932673    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cf32b96d-4371-4b92-89db-ff71111c977a-trusted-ca-bundle\") pod \"insights-operator-78bdfb888c-tnkbv\" (UID: \"cf32b96d-4371-4b92-89db-ff71111c977a\") " pod="openshift-insights/insights-operator-78bdfb888c-tnkbv"
May 25 13:06:35.932935 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:06:35.932708    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/518b0e48-f2ac-4401-9212-834cbad021f9-config\") pod \"service-ca-operator-57b5b68b8-rh7nv\" (UID: \"518b0e48-f2ac-4401-9212-834cbad021f9\") " pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-rh7nv"
May 25 13:06:35.933105 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:06:35.933080    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cf32b96d-4371-4b92-89db-ff71111c977a-tmp\") pod \"insights-operator-78bdfb888c-tnkbv\" (UID: \"cf32b96d-4371-4b92-89db-ff71111c977a\") " pod="openshift-insights/insights-operator-78bdfb888c-tnkbv"
May 25 13:06:35.933169 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:06:35.933144    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/cf32b96d-4371-4b92-89db-ff71111c977a-snapshots\") pod \"insights-operator-78bdfb888c-tnkbv\" (UID: \"cf32b96d-4371-4b92-89db-ff71111c977a\") " pod="openshift-insights/insights-operator-78bdfb888c-tnkbv"
May 25 13:06:35.933472 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:06:35.933347    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/518b0e48-f2ac-4401-9212-834cbad021f9-config\") pod \"service-ca-operator-57b5b68b8-rh7nv\" (UID: \"518b0e48-f2ac-4401-9212-834cbad021f9\") " pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-rh7nv"
May 25 13:06:35.933515 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:06:35.933503    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cf32b96d-4371-4b92-89db-ff71111c977a-trusted-ca-bundle\") pod \"insights-operator-78bdfb888c-tnkbv\" (UID: \"cf32b96d-4371-4b92-89db-ff71111c977a\") " pod="openshift-insights/insights-operator-78bdfb888c-tnkbv"
May 25 13:06:35.933728 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:06:35.933708    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cf32b96d-4371-4b92-89db-ff71111c977a-service-ca-bundle\") pod \"insights-operator-78bdfb888c-tnkbv\" (UID: \"cf32b96d-4371-4b92-89db-ff71111c977a\") " pod="openshift-insights/insights-operator-78bdfb888c-tnkbv"
May 25 13:06:35.934937 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:06:35.934918    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/518b0e48-f2ac-4401-9212-834cbad021f9-serving-cert\") pod \"service-ca-operator-57b5b68b8-rh7nv\" (UID: \"518b0e48-f2ac-4401-9212-834cbad021f9\") " pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-rh7nv"
May 25 13:06:35.935116 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:06:35.935101    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/cf32b96d-4371-4b92-89db-ff71111c977a-serving-cert\") pod \"insights-operator-78bdfb888c-tnkbv\" (UID: \"cf32b96d-4371-4b92-89db-ff71111c977a\") " pod="openshift-insights/insights-operator-78bdfb888c-tnkbv"
May 25 13:06:35.940666 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:06:35.940621    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5hw5\" (UniqueName: \"kubernetes.io/projected/2334e399-4548-4098-b14a-0e4b8a82e640-kube-api-access-l5hw5\") pod \"network-check-source-7f44b6dd87-tmx95\" (UID: \"2334e399-4548-4098-b14a-0e4b8a82e640\") " pod="openshift-network-diagnostics/network-check-source-7f44b6dd87-tmx95"
May 25 13:06:35.940796 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:06:35.940774    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sb24q\" (UniqueName: \"kubernetes.io/projected/518b0e48-f2ac-4401-9212-834cbad021f9-kube-api-access-sb24q\") pod \"service-ca-operator-57b5b68b8-rh7nv\" (UID: \"518b0e48-f2ac-4401-9212-834cbad021f9\") " pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-rh7nv"
May 25 13:06:35.941098 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:06:35.941076    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tn5b2\" (UniqueName: \"kubernetes.io/projected/cf32b96d-4371-4b92-89db-ff71111c977a-kube-api-access-tn5b2\") pod \"insights-operator-78bdfb888c-tnkbv\" (UID: \"cf32b96d-4371-4b92-89db-ff71111c977a\") " pod="openshift-insights/insights-operator-78bdfb888c-tnkbv"
May 25 13:06:36.085770 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:06:36.085718    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-rh7nv"
May 25 13:06:36.091489 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:06:36.091465    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-78bdfb888c-tnkbv"
May 25 13:06:36.098050 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:06:36.098023    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7f44b6dd87-tmx95"
May 25 13:06:36.223998 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:06:36.223965    2580 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57b5b68b8-rh7nv"]
May 25 13:06:36.227806 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:06:36.227778    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod518b0e48_f2ac_4401_9212_834cbad021f9.slice/crio-37cadd39bd783bebef2c2c73a92216047317f40080e7fdfb77095d550c0f8291 WatchSource:0}: Error finding container 37cadd39bd783bebef2c2c73a92216047317f40080e7fdfb77095d550c0f8291: Status 404 returned error can't find the container with id 37cadd39bd783bebef2c2c73a92216047317f40080e7fdfb77095d550c0f8291
May 25 13:06:36.335641 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:06:36.335598    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f21589b0-1b80-46bb-92e8-1252201ed8f3-registry-tls\") pod \"image-registry-769864757b-hs9k5\" (UID: \"f21589b0-1b80-46bb-92e8-1252201ed8f3\") " pod="openshift-image-registry/image-registry-769864757b-hs9k5"
May 25 13:06:36.335641 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:06:36.335645    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/264a8628-6305-4840-9925-88aeafe69834-metrics-certs\") pod \"router-default-549b58d588-dnld2\" (UID: \"264a8628-6305-4840-9925-88aeafe69834\") " pod="openshift-ingress/router-default-549b58d588-dnld2"
May 25 13:06:36.335895 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:06:36.335689    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/264a8628-6305-4840-9925-88aeafe69834-service-ca-bundle\") pod \"router-default-549b58d588-dnld2\" (UID: \"264a8628-6305-4840-9925-88aeafe69834\") " pod="openshift-ingress/router-default-549b58d588-dnld2"
May 25 13:06:36.335895 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:06:36.335752    2580 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 25 13:06:36.335895 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:06:36.335775    2580 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-769864757b-hs9k5: secret "image-registry-tls" not found
May 25 13:06:36.335895 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:06:36.335812    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/264a8628-6305-4840-9925-88aeafe69834-service-ca-bundle podName:264a8628-6305-4840-9925-88aeafe69834 nodeName:}" failed. No retries permitted until 2026-05-25 13:06:37.335798331 +0000 UTC m=+97.528092670 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/264a8628-6305-4840-9925-88aeafe69834-service-ca-bundle") pod "router-default-549b58d588-dnld2" (UID: "264a8628-6305-4840-9925-88aeafe69834") : configmap references non-existent config key: service-ca.crt
May 25 13:06:36.335895 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:06:36.335828    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f21589b0-1b80-46bb-92e8-1252201ed8f3-registry-tls podName:f21589b0-1b80-46bb-92e8-1252201ed8f3 nodeName:}" failed. No retries permitted until 2026-05-25 13:06:37.335821534 +0000 UTC m=+97.528115872 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f21589b0-1b80-46bb-92e8-1252201ed8f3-registry-tls") pod "image-registry-769864757b-hs9k5" (UID: "f21589b0-1b80-46bb-92e8-1252201ed8f3") : secret "image-registry-tls" not found
May 25 13:06:36.335895 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:06:36.335853    2580 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
May 25 13:06:36.336096 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:06:36.335901    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/264a8628-6305-4840-9925-88aeafe69834-metrics-certs podName:264a8628-6305-4840-9925-88aeafe69834 nodeName:}" failed. No retries permitted until 2026-05-25 13:06:37.335883541 +0000 UTC m=+97.528177883 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/264a8628-6305-4840-9925-88aeafe69834-metrics-certs") pod "router-default-549b58d588-dnld2" (UID: "264a8628-6305-4840-9925-88aeafe69834") : secret "router-metrics-certs-default" not found
May 25 13:06:36.453201 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:06:36.451693    2580 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7f44b6dd87-tmx95"]
May 25 13:06:36.453201 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:06:36.451739    2580 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-78bdfb888c-tnkbv"]
May 25 13:06:36.453592 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:06:36.453566    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2334e399_4548_4098_b14a_0e4b8a82e640.slice/crio-29a3a6ceaccac8f9778a038c6a4da199188be744d36878744ba6af869d6fc4d5 WatchSource:0}: Error finding container 29a3a6ceaccac8f9778a038c6a4da199188be744d36878744ba6af869d6fc4d5: Status 404 returned error can't find the container with id 29a3a6ceaccac8f9778a038c6a4da199188be744d36878744ba6af869d6fc4d5
May 25 13:06:36.455170 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:06:36.455140    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf32b96d_4371_4b92_89db_ff71111c977a.slice/crio-c9b8a4033522dbcc678f55dd6986af519c1cab7d1585f3fbd0f40d8cfbe0ef4b WatchSource:0}: Error finding container c9b8a4033522dbcc678f55dd6986af519c1cab7d1585f3fbd0f40d8cfbe0ef4b: Status 404 returned error can't find the container with id c9b8a4033522dbcc678f55dd6986af519c1cab7d1585f3fbd0f40d8cfbe0ef4b
May 25 13:06:36.718964 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:06:36.718823    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7f44b6dd87-tmx95" event={"ID":"2334e399-4548-4098-b14a-0e4b8a82e640","Type":"ContainerStarted","Data":"8019fffe05d17cb6a3cc0dc7c9ac2f941935f9f253881c0e56f35e5abef40fd6"}
May 25 13:06:36.718964 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:06:36.718869    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7f44b6dd87-tmx95" event={"ID":"2334e399-4548-4098-b14a-0e4b8a82e640","Type":"ContainerStarted","Data":"29a3a6ceaccac8f9778a038c6a4da199188be744d36878744ba6af869d6fc4d5"}
May 25 13:06:36.720792 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:06:36.720752    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-78bdfb888c-tnkbv" event={"ID":"cf32b96d-4371-4b92-89db-ff71111c977a","Type":"ContainerStarted","Data":"c9b8a4033522dbcc678f55dd6986af519c1cab7d1585f3fbd0f40d8cfbe0ef4b"}
May 25 13:06:36.721894 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:06:36.721861    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-rh7nv" event={"ID":"518b0e48-f2ac-4401-9212-834cbad021f9","Type":"ContainerStarted","Data":"37cadd39bd783bebef2c2c73a92216047317f40080e7fdfb77095d550c0f8291"}
May 25 13:06:36.733768 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:06:36.733720    2580 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7f44b6dd87-tmx95" podStartSLOduration=1.733705038 podStartE2EDuration="1.733705038s" podCreationTimestamp="2026-05-25 13:06:35 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 13:06:36.733651334 +0000 UTC m=+96.925945694" watchObservedRunningTime="2026-05-25 13:06:36.733705038 +0000 UTC m=+96.925999395"
May 25 13:06:37.344244 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:06:37.344197    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f21589b0-1b80-46bb-92e8-1252201ed8f3-registry-tls\") pod \"image-registry-769864757b-hs9k5\" (UID: \"f21589b0-1b80-46bb-92e8-1252201ed8f3\") " pod="openshift-image-registry/image-registry-769864757b-hs9k5"
May 25 13:06:37.344452 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:06:37.344254    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/264a8628-6305-4840-9925-88aeafe69834-metrics-certs\") pod \"router-default-549b58d588-dnld2\" (UID: \"264a8628-6305-4840-9925-88aeafe69834\") " pod="openshift-ingress/router-default-549b58d588-dnld2"
May 25 13:06:37.344452 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:06:37.344289    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/264a8628-6305-4840-9925-88aeafe69834-service-ca-bundle\") pod \"router-default-549b58d588-dnld2\" (UID: \"264a8628-6305-4840-9925-88aeafe69834\") " pod="openshift-ingress/router-default-549b58d588-dnld2"
May 25 13:06:37.344452 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:06:37.344356    2580 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 25 13:06:37.344452 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:06:37.344396    2580 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-769864757b-hs9k5: secret "image-registry-tls" not found
May 25 13:06:37.344452 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:06:37.344431    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/264a8628-6305-4840-9925-88aeafe69834-service-ca-bundle podName:264a8628-6305-4840-9925-88aeafe69834 nodeName:}" failed. No retries permitted until 2026-05-25 13:06:39.344411373 +0000 UTC m=+99.536705713 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/264a8628-6305-4840-9925-88aeafe69834-service-ca-bundle") pod "router-default-549b58d588-dnld2" (UID: "264a8628-6305-4840-9925-88aeafe69834") : configmap references non-existent config key: service-ca.crt
May 25 13:06:37.344452 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:06:37.344451    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f21589b0-1b80-46bb-92e8-1252201ed8f3-registry-tls podName:f21589b0-1b80-46bb-92e8-1252201ed8f3 nodeName:}" failed. No retries permitted until 2026-05-25 13:06:39.344442423 +0000 UTC m=+99.536736760 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f21589b0-1b80-46bb-92e8-1252201ed8f3-registry-tls") pod "image-registry-769864757b-hs9k5" (UID: "f21589b0-1b80-46bb-92e8-1252201ed8f3") : secret "image-registry-tls" not found
May 25 13:06:37.344804 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:06:37.344492    2580 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
May 25 13:06:37.344804 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:06:37.344537    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/264a8628-6305-4840-9925-88aeafe69834-metrics-certs podName:264a8628-6305-4840-9925-88aeafe69834 nodeName:}" failed. No retries permitted until 2026-05-25 13:06:39.344519101 +0000 UTC m=+99.536813454 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/264a8628-6305-4840-9925-88aeafe69834-metrics-certs") pod "router-default-549b58d588-dnld2" (UID: "264a8628-6305-4840-9925-88aeafe69834") : secret "router-metrics-certs-default" not found
May 25 13:06:38.151693 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:06:38.151643    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2e9a6623-9ccb-484c-b669-38caefdf2976-cert\") pod \"ingress-canary-8gtmh\" (UID: \"2e9a6623-9ccb-484c-b669-38caefdf2976\") " pod="openshift-ingress-canary/ingress-canary-8gtmh"
May 25 13:06:38.152124 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:06:38.151714    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c04c0bd2-e6c2-42a0-949b-8f818987dd6b-metrics-tls\") pod \"dns-default-ltl6z\" (UID: \"c04c0bd2-e6c2-42a0-949b-8f818987dd6b\") " pod="openshift-dns/dns-default-ltl6z"
May 25 13:06:38.152124 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:06:38.151803    2580 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 25 13:06:38.152124 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:06:38.151879    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e9a6623-9ccb-484c-b669-38caefdf2976-cert podName:2e9a6623-9ccb-484c-b669-38caefdf2976 nodeName:}" failed. No retries permitted until 2026-05-25 13:07:42.151862209 +0000 UTC m=+162.344156554 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/2e9a6623-9ccb-484c-b669-38caefdf2976-cert") pod "ingress-canary-8gtmh" (UID: "2e9a6623-9ccb-484c-b669-38caefdf2976") : secret "canary-serving-cert" not found
May 25 13:06:38.152124 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:06:38.151885    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 25 13:06:38.152124 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:06:38.151945    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c04c0bd2-e6c2-42a0-949b-8f818987dd6b-metrics-tls podName:c04c0bd2-e6c2-42a0-949b-8f818987dd6b nodeName:}" failed. No retries permitted until 2026-05-25 13:07:42.151926776 +0000 UTC m=+162.344221120 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c04c0bd2-e6c2-42a0-949b-8f818987dd6b-metrics-tls") pod "dns-default-ltl6z" (UID: "c04c0bd2-e6c2-42a0-949b-8f818987dd6b") : secret "dns-default-metrics-tls" not found
May 25 13:06:38.732138 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:06:38.732084    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-78bdfb888c-tnkbv" event={"ID":"cf32b96d-4371-4b92-89db-ff71111c977a","Type":"ContainerStarted","Data":"291643ef9573c23e5a69b92bf62c4ba4e8a36d7b36c1e5d6d9183c4e804df02a"}
May 25 13:06:38.734264 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:06:38.733799    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-rh7nv" event={"ID":"518b0e48-f2ac-4401-9212-834cbad021f9","Type":"ContainerStarted","Data":"2dc9c3ce06a1288a1f477fd417610634cfea5f0623e5b50edc785fb992e97a1a"}
May 25 13:06:38.747208 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:06:38.747077    2580 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-insights/insights-operator-78bdfb888c-tnkbv" podStartSLOduration=1.588114507 podStartE2EDuration="3.747059941s" podCreationTimestamp="2026-05-25 13:06:35 +0000 UTC" firstStartedPulling="2026-05-25 13:06:36.456968448 +0000 UTC m=+96.649262786" lastFinishedPulling="2026-05-25 13:06:38.615913876 +0000 UTC m=+98.808208220" observedRunningTime="2026-05-25 13:06:38.746616337 +0000 UTC m=+98.938910699" watchObservedRunningTime="2026-05-25 13:06:38.747059941 +0000 UTC m=+98.939354304"
May 25 13:06:38.760978 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:06:38.760931    2580 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-rh7nv" podStartSLOduration=1.374940689 podStartE2EDuration="3.760915941s" podCreationTimestamp="2026-05-25 13:06:35 +0000 UTC" firstStartedPulling="2026-05-25 13:06:36.229595919 +0000 UTC m=+96.421890258" lastFinishedPulling="2026-05-25 13:06:38.615571172 +0000 UTC m=+98.807865510" observedRunningTime="2026-05-25 13:06:38.760040216 +0000 UTC m=+98.952334576" watchObservedRunningTime="2026-05-25 13:06:38.760915941 +0000 UTC m=+98.953210300"
May 25 13:06:39.362384 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:06:39.362319    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f21589b0-1b80-46bb-92e8-1252201ed8f3-registry-tls\") pod \"image-registry-769864757b-hs9k5\" (UID: \"f21589b0-1b80-46bb-92e8-1252201ed8f3\") " pod="openshift-image-registry/image-registry-769864757b-hs9k5"
May 25 13:06:39.362846 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:06:39.362397    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/264a8628-6305-4840-9925-88aeafe69834-metrics-certs\") pod \"router-default-549b58d588-dnld2\" (UID: \"264a8628-6305-4840-9925-88aeafe69834\") " pod="openshift-ingress/router-default-549b58d588-dnld2"
May 25 13:06:39.362846 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:06:39.362482    2580 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 25 13:06:39.362846 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:06:39.362505    2580 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-769864757b-hs9k5: secret "image-registry-tls" not found
May 25 13:06:39.362846 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:06:39.362505    2580 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
May 25 13:06:39.362846 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:06:39.362558    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/264a8628-6305-4840-9925-88aeafe69834-service-ca-bundle\") pod \"router-default-549b58d588-dnld2\" (UID: \"264a8628-6305-4840-9925-88aeafe69834\") " pod="openshift-ingress/router-default-549b58d588-dnld2"
May 25 13:06:39.362846 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:06:39.362568    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/264a8628-6305-4840-9925-88aeafe69834-metrics-certs podName:264a8628-6305-4840-9925-88aeafe69834 nodeName:}" failed. No retries permitted until 2026-05-25 13:06:43.362549243 +0000 UTC m=+103.554843584 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/264a8628-6305-4840-9925-88aeafe69834-metrics-certs") pod "router-default-549b58d588-dnld2" (UID: "264a8628-6305-4840-9925-88aeafe69834") : secret "router-metrics-certs-default" not found
May 25 13:06:39.362846 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:06:39.362644    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f21589b0-1b80-46bb-92e8-1252201ed8f3-registry-tls podName:f21589b0-1b80-46bb-92e8-1252201ed8f3 nodeName:}" failed. No retries permitted until 2026-05-25 13:06:43.362624145 +0000 UTC m=+103.554918486 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f21589b0-1b80-46bb-92e8-1252201ed8f3-registry-tls") pod "image-registry-769864757b-hs9k5" (UID: "f21589b0-1b80-46bb-92e8-1252201ed8f3") : secret "image-registry-tls" not found
May 25 13:06:39.362846 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:06:39.362667    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/264a8628-6305-4840-9925-88aeafe69834-service-ca-bundle podName:264a8628-6305-4840-9925-88aeafe69834 nodeName:}" failed. No retries permitted until 2026-05-25 13:06:43.36266036 +0000 UTC m=+103.554954698 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/264a8628-6305-4840-9925-88aeafe69834-service-ca-bundle") pod "router-default-549b58d588-dnld2" (UID: "264a8628-6305-4840-9925-88aeafe69834") : configmap references non-existent config key: service-ca.crt
May 25 13:06:40.670804 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:06:40.670770    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-25spc"
May 25 13:06:42.610297 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:06:42.610270    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-vlk4j_37b3e162-412d-4e21-8f2f-bb8398f30435/dns-node-resolver/0.log"
May 25 13:06:43.395246 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:06:43.395212    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f21589b0-1b80-46bb-92e8-1252201ed8f3-registry-tls\") pod \"image-registry-769864757b-hs9k5\" (UID: \"f21589b0-1b80-46bb-92e8-1252201ed8f3\") " pod="openshift-image-registry/image-registry-769864757b-hs9k5"
May 25 13:06:43.395480 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:06:43.395260    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/264a8628-6305-4840-9925-88aeafe69834-metrics-certs\") pod \"router-default-549b58d588-dnld2\" (UID: \"264a8628-6305-4840-9925-88aeafe69834\") " pod="openshift-ingress/router-default-549b58d588-dnld2"
May 25 13:06:43.395480 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:06:43.395294    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/264a8628-6305-4840-9925-88aeafe69834-service-ca-bundle\") pod \"router-default-549b58d588-dnld2\" (UID: \"264a8628-6305-4840-9925-88aeafe69834\") " pod="openshift-ingress/router-default-549b58d588-dnld2"
May 25 13:06:43.395480 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:06:43.395388    2580 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 25 13:06:43.395480 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:06:43.395408    2580 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
May 25 13:06:43.395480 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:06:43.395436    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/264a8628-6305-4840-9925-88aeafe69834-service-ca-bundle podName:264a8628-6305-4840-9925-88aeafe69834 nodeName:}" failed. No retries permitted until 2026-05-25 13:06:51.395422799 +0000 UTC m=+111.587717137 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/264a8628-6305-4840-9925-88aeafe69834-service-ca-bundle") pod "router-default-549b58d588-dnld2" (UID: "264a8628-6305-4840-9925-88aeafe69834") : configmap references non-existent config key: service-ca.crt
May 25 13:06:43.395480 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:06:43.395472    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/264a8628-6305-4840-9925-88aeafe69834-metrics-certs podName:264a8628-6305-4840-9925-88aeafe69834 nodeName:}" failed. No retries permitted until 2026-05-25 13:06:51.395454624 +0000 UTC m=+111.587748982 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/264a8628-6305-4840-9925-88aeafe69834-metrics-certs") pod "router-default-549b58d588-dnld2" (UID: "264a8628-6305-4840-9925-88aeafe69834") : secret "router-metrics-certs-default" not found
May 25 13:06:43.395748 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:06:43.395411    2580 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-769864757b-hs9k5: secret "image-registry-tls" not found
May 25 13:06:43.395748 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:06:43.395546    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f21589b0-1b80-46bb-92e8-1252201ed8f3-registry-tls podName:f21589b0-1b80-46bb-92e8-1252201ed8f3 nodeName:}" failed. No retries permitted until 2026-05-25 13:06:51.395527149 +0000 UTC m=+111.587821487 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f21589b0-1b80-46bb-92e8-1252201ed8f3-registry-tls") pod "image-registry-769864757b-hs9k5" (UID: "f21589b0-1b80-46bb-92e8-1252201ed8f3") : secret "image-registry-tls" not found
May 25 13:06:43.608875 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:06:43.608845    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-wlq4q_6ef8bb8d-98b5-4fe9-94a7-8e06d34ab971/node-ca/0.log"
May 25 13:06:51.457652 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:06:51.457617    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f21589b0-1b80-46bb-92e8-1252201ed8f3-registry-tls\") pod \"image-registry-769864757b-hs9k5\" (UID: \"f21589b0-1b80-46bb-92e8-1252201ed8f3\") " pod="openshift-image-registry/image-registry-769864757b-hs9k5"
May 25 13:06:51.458052 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:06:51.457666    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/264a8628-6305-4840-9925-88aeafe69834-metrics-certs\") pod \"router-default-549b58d588-dnld2\" (UID: \"264a8628-6305-4840-9925-88aeafe69834\") " pod="openshift-ingress/router-default-549b58d588-dnld2"
May 25 13:06:51.458052 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:06:51.457704    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/264a8628-6305-4840-9925-88aeafe69834-service-ca-bundle\") pod \"router-default-549b58d588-dnld2\" (UID: \"264a8628-6305-4840-9925-88aeafe69834\") " pod="openshift-ingress/router-default-549b58d588-dnld2"
May 25 13:06:51.458052 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:06:51.457852    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/264a8628-6305-4840-9925-88aeafe69834-service-ca-bundle podName:264a8628-6305-4840-9925-88aeafe69834 nodeName:}" failed. No retries permitted until 2026-05-25 13:07:07.457833505 +0000 UTC m=+127.650127857 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/264a8628-6305-4840-9925-88aeafe69834-service-ca-bundle") pod "router-default-549b58d588-dnld2" (UID: "264a8628-6305-4840-9925-88aeafe69834") : configmap references non-existent config key: service-ca.crt
May 25 13:06:51.460272 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:06:51.460242    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/264a8628-6305-4840-9925-88aeafe69834-metrics-certs\") pod \"router-default-549b58d588-dnld2\" (UID: \"264a8628-6305-4840-9925-88aeafe69834\") " pod="openshift-ingress/router-default-549b58d588-dnld2"
May 25 13:06:51.460362 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:06:51.460242    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f21589b0-1b80-46bb-92e8-1252201ed8f3-registry-tls\") pod \"image-registry-769864757b-hs9k5\" (UID: \"f21589b0-1b80-46bb-92e8-1252201ed8f3\") " pod="openshift-image-registry/image-registry-769864757b-hs9k5"
May 25 13:06:51.583660 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:06:51.583619    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-769864757b-hs9k5"
May 25 13:06:51.715633 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:06:51.715561    2580 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-769864757b-hs9k5"]
May 25 13:06:51.719333 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:06:51.719302    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf21589b0_1b80_46bb_92e8_1252201ed8f3.slice/crio-cae00a9e52499e55d11dcdcc34dc25c372a7d81946179536e3a85fe902e9e37d WatchSource:0}: Error finding container cae00a9e52499e55d11dcdcc34dc25c372a7d81946179536e3a85fe902e9e37d: Status 404 returned error can't find the container with id cae00a9e52499e55d11dcdcc34dc25c372a7d81946179536e3a85fe902e9e37d
May 25 13:06:51.770004 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:06:51.769974    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-769864757b-hs9k5" event={"ID":"f21589b0-1b80-46bb-92e8-1252201ed8f3","Type":"ContainerStarted","Data":"cae00a9e52499e55d11dcdcc34dc25c372a7d81946179536e3a85fe902e9e37d"}
May 25 13:06:52.773478 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:06:52.773439    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-769864757b-hs9k5" event={"ID":"f21589b0-1b80-46bb-92e8-1252201ed8f3","Type":"ContainerStarted","Data":"739aaf59fcd334cb67940a2e8ce48c66d6cdc823e495ea846aa5843e210cb265"}
May 25 13:06:52.773943 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:06:52.773557    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-769864757b-hs9k5"
May 25 13:06:52.795965 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:06:52.795918    2580 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-image-registry/image-registry-769864757b-hs9k5" podStartSLOduration=17.795903716 podStartE2EDuration="17.795903716s" podCreationTimestamp="2026-05-25 13:06:35 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 13:06:52.794507046 +0000 UTC m=+112.986801417" watchObservedRunningTime="2026-05-25 13:06:52.795903716 +0000 UTC m=+112.988198123"
May 25 13:07:02.873186 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:02.873144    2580 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-ql787"]
May 25 13:07:02.876411 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:02.876391    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-ql787"
May 25 13:07:02.881149 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:02.881130    2580 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
May 25 13:07:02.881558 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:02.881541    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-bzxd9\""
May 25 13:07:02.881669 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:02.881582    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
May 25 13:07:02.886357 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:02.886332    2580 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-d4d8f94f9-dlbn5"]
May 25 13:07:02.888194 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:02.888175    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-d4d8f94f9-dlbn5"
May 25 13:07:02.894236 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:02.894215    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
May 25 13:07:02.894392 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:02.894354    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-ld52b\""
May 25 13:07:02.897413 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:02.897397    2580 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
May 25 13:07:02.915013 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:02.914980    2580 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-d4d8f94f9-dlbn5"]
May 25 13:07:02.928565 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:02.928534    2580 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-769864757b-hs9k5"]
May 25 13:07:02.945568 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:02.945535    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/dc041e36-d4ec-4dac-b8e7-d0092d61edf5-networking-console-plugin-cert\") pod \"networking-console-plugin-d4d8f94f9-dlbn5\" (UID: \"dc041e36-d4ec-4dac-b8e7-d0092d61edf5\") " pod="openshift-network-console/networking-console-plugin-d4d8f94f9-dlbn5"
May 25 13:07:02.945725 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:02.945580    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1f1fc207-9a8d-4a4c-beb7-bb4405007c8e-crio-socket\") pod \"insights-runtime-extractor-ql787\" (UID: \"1f1fc207-9a8d-4a4c-beb7-bb4405007c8e\") " pod="openshift-insights/insights-runtime-extractor-ql787"
May 25 13:07:02.945725 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:02.945602    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/dc041e36-d4ec-4dac-b8e7-d0092d61edf5-nginx-conf\") pod \"networking-console-plugin-d4d8f94f9-dlbn5\" (UID: \"dc041e36-d4ec-4dac-b8e7-d0092d61edf5\") " pod="openshift-network-console/networking-console-plugin-d4d8f94f9-dlbn5"
May 25 13:07:02.945725 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:02.945619    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68wdk\" (UniqueName: \"kubernetes.io/projected/1f1fc207-9a8d-4a4c-beb7-bb4405007c8e-kube-api-access-68wdk\") pod \"insights-runtime-extractor-ql787\" (UID: \"1f1fc207-9a8d-4a4c-beb7-bb4405007c8e\") " pod="openshift-insights/insights-runtime-extractor-ql787"
May 25 13:07:02.945725 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:02.945714    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1f1fc207-9a8d-4a4c-beb7-bb4405007c8e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ql787\" (UID: \"1f1fc207-9a8d-4a4c-beb7-bb4405007c8e\") " pod="openshift-insights/insights-runtime-extractor-ql787"
May 25 13:07:02.945878 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:02.945766    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1f1fc207-9a8d-4a4c-beb7-bb4405007c8e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ql787\" (UID: \"1f1fc207-9a8d-4a4c-beb7-bb4405007c8e\") " pod="openshift-insights/insights-runtime-extractor-ql787"
May 25 13:07:02.945878 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:02.945809    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1f1fc207-9a8d-4a4c-beb7-bb4405007c8e-data-volume\") pod \"insights-runtime-extractor-ql787\" (UID: \"1f1fc207-9a8d-4a4c-beb7-bb4405007c8e\") " pod="openshift-insights/insights-runtime-extractor-ql787"
May 25 13:07:02.955042 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:02.955012    2580 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-ql787"]
May 25 13:07:03.046770 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:03.046736    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/dc041e36-d4ec-4dac-b8e7-d0092d61edf5-networking-console-plugin-cert\") pod \"networking-console-plugin-d4d8f94f9-dlbn5\" (UID: \"dc041e36-d4ec-4dac-b8e7-d0092d61edf5\") " pod="openshift-network-console/networking-console-plugin-d4d8f94f9-dlbn5"
May 25 13:07:03.046770 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:03.046775    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1f1fc207-9a8d-4a4c-beb7-bb4405007c8e-crio-socket\") pod \"insights-runtime-extractor-ql787\" (UID: \"1f1fc207-9a8d-4a4c-beb7-bb4405007c8e\") " pod="openshift-insights/insights-runtime-extractor-ql787"
May 25 13:07:03.046971 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:03.046796    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/dc041e36-d4ec-4dac-b8e7-d0092d61edf5-nginx-conf\") pod \"networking-console-plugin-d4d8f94f9-dlbn5\" (UID: \"dc041e36-d4ec-4dac-b8e7-d0092d61edf5\") " pod="openshift-network-console/networking-console-plugin-d4d8f94f9-dlbn5"
May 25 13:07:03.046971 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:03.046856    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1f1fc207-9a8d-4a4c-beb7-bb4405007c8e-crio-socket\") pod \"insights-runtime-extractor-ql787\" (UID: \"1f1fc207-9a8d-4a4c-beb7-bb4405007c8e\") " pod="openshift-insights/insights-runtime-extractor-ql787"
May 25 13:07:03.046971 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:03.046925    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-68wdk\" (UniqueName: \"kubernetes.io/projected/1f1fc207-9a8d-4a4c-beb7-bb4405007c8e-kube-api-access-68wdk\") pod \"insights-runtime-extractor-ql787\" (UID: \"1f1fc207-9a8d-4a4c-beb7-bb4405007c8e\") " pod="openshift-insights/insights-runtime-extractor-ql787"
May 25 13:07:03.047073 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:03.046994    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1f1fc207-9a8d-4a4c-beb7-bb4405007c8e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ql787\" (UID: \"1f1fc207-9a8d-4a4c-beb7-bb4405007c8e\") " pod="openshift-insights/insights-runtime-extractor-ql787"
May 25 13:07:03.047073 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:03.047050    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1f1fc207-9a8d-4a4c-beb7-bb4405007c8e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ql787\" (UID: \"1f1fc207-9a8d-4a4c-beb7-bb4405007c8e\") " pod="openshift-insights/insights-runtime-extractor-ql787"
May 25 13:07:03.047145 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:03.047076    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1f1fc207-9a8d-4a4c-beb7-bb4405007c8e-data-volume\") pod \"insights-runtime-extractor-ql787\" (UID: \"1f1fc207-9a8d-4a4c-beb7-bb4405007c8e\") " pod="openshift-insights/insights-runtime-extractor-ql787"
May 25 13:07:03.047447 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:03.047425    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1f1fc207-9a8d-4a4c-beb7-bb4405007c8e-data-volume\") pod \"insights-runtime-extractor-ql787\" (UID: \"1f1fc207-9a8d-4a4c-beb7-bb4405007c8e\") " pod="openshift-insights/insights-runtime-extractor-ql787"
May 25 13:07:03.047622 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:03.047605    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/dc041e36-d4ec-4dac-b8e7-d0092d61edf5-nginx-conf\") pod \"networking-console-plugin-d4d8f94f9-dlbn5\" (UID: \"dc041e36-d4ec-4dac-b8e7-d0092d61edf5\") " pod="openshift-network-console/networking-console-plugin-d4d8f94f9-dlbn5"
May 25 13:07:03.047808 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:03.047761    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1f1fc207-9a8d-4a4c-beb7-bb4405007c8e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ql787\" (UID: \"1f1fc207-9a8d-4a4c-beb7-bb4405007c8e\") " pod="openshift-insights/insights-runtime-extractor-ql787"
May 25 13:07:03.049418 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:03.049397    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/dc041e36-d4ec-4dac-b8e7-d0092d61edf5-networking-console-plugin-cert\") pod \"networking-console-plugin-d4d8f94f9-dlbn5\" (UID: \"dc041e36-d4ec-4dac-b8e7-d0092d61edf5\") " pod="openshift-network-console/networking-console-plugin-d4d8f94f9-dlbn5"
May 25 13:07:03.049509 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:03.049495    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1f1fc207-9a8d-4a4c-beb7-bb4405007c8e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ql787\" (UID: \"1f1fc207-9a8d-4a4c-beb7-bb4405007c8e\") " pod="openshift-insights/insights-runtime-extractor-ql787"
May 25 13:07:03.058669 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:03.058646    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-68wdk\" (UniqueName: \"kubernetes.io/projected/1f1fc207-9a8d-4a4c-beb7-bb4405007c8e-kube-api-access-68wdk\") pod \"insights-runtime-extractor-ql787\" (UID: \"1f1fc207-9a8d-4a4c-beb7-bb4405007c8e\") " pod="openshift-insights/insights-runtime-extractor-ql787"
May 25 13:07:03.185661 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:03.185571    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-ql787"
May 25 13:07:03.196500 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:03.196469    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-d4d8f94f9-dlbn5"
May 25 13:07:03.325298 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:03.325266    2580 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-ql787"]
May 25 13:07:03.328751 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:07:03.328724    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f1fc207_9a8d_4a4c_beb7_bb4405007c8e.slice/crio-3f3f945473148d6cdc19e191d257c9f9f288e0a398adca25874c1999dc53a4c0 WatchSource:0}: Error finding container 3f3f945473148d6cdc19e191d257c9f9f288e0a398adca25874c1999dc53a4c0: Status 404 returned error can't find the container with id 3f3f945473148d6cdc19e191d257c9f9f288e0a398adca25874c1999dc53a4c0
May 25 13:07:03.339009 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:03.338970    2580 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-d4d8f94f9-dlbn5"]
May 25 13:07:03.342390 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:07:03.342342    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc041e36_d4ec_4dac_b8e7_d0092d61edf5.slice/crio-e99c96dcbcf53c6cd8820a90da73597084b8c60e0cbec74349dd939fa637adf0 WatchSource:0}: Error finding container e99c96dcbcf53c6cd8820a90da73597084b8c60e0cbec74349dd939fa637adf0: Status 404 returned error can't find the container with id e99c96dcbcf53c6cd8820a90da73597084b8c60e0cbec74349dd939fa637adf0
May 25 13:07:03.802289 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:03.802253    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ql787" event={"ID":"1f1fc207-9a8d-4a4c-beb7-bb4405007c8e","Type":"ContainerStarted","Data":"2e79c4a7dc182c5544c1a46e447358a87eadc9917b5b3e4350ef68dba0cfe422"}
May 25 13:07:03.802289 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:03.802292    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ql787" event={"ID":"1f1fc207-9a8d-4a4c-beb7-bb4405007c8e","Type":"ContainerStarted","Data":"3f3f945473148d6cdc19e191d257c9f9f288e0a398adca25874c1999dc53a4c0"}
May 25 13:07:03.803479 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:03.803451    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-d4d8f94f9-dlbn5" event={"ID":"dc041e36-d4ec-4dac-b8e7-d0092d61edf5","Type":"ContainerStarted","Data":"e99c96dcbcf53c6cd8820a90da73597084b8c60e0cbec74349dd939fa637adf0"}
May 25 13:07:04.807867 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:04.807825    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ql787" event={"ID":"1f1fc207-9a8d-4a4c-beb7-bb4405007c8e","Type":"ContainerStarted","Data":"b507d99352f3f54c0b1d6a6e720d5878cb0be90a8e4ceb44dc70bf1f16c630b5"}
May 25 13:07:04.809279 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:04.809248    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-d4d8f94f9-dlbn5" event={"ID":"dc041e36-d4ec-4dac-b8e7-d0092d61edf5","Type":"ContainerStarted","Data":"11f272d8f9a1c531578d4f9705c5802e63dfbca0290531499f95d99f2e0a66e2"}
May 25 13:07:04.823995 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:04.823945    2580 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-d4d8f94f9-dlbn5" podStartSLOduration=1.860255922 podStartE2EDuration="2.823930842s" podCreationTimestamp="2026-05-25 13:07:02 +0000 UTC" firstStartedPulling="2026-05-25 13:07:03.344755415 +0000 UTC m=+123.537049753" lastFinishedPulling="2026-05-25 13:07:04.308430331 +0000 UTC m=+124.500724673" observedRunningTime="2026-05-25 13:07:04.823095523 +0000 UTC m=+125.015389888" watchObservedRunningTime="2026-05-25 13:07:04.823930842 +0000 UTC m=+125.016225202"
May 25 13:07:05.816982 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:05.816937    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ql787" event={"ID":"1f1fc207-9a8d-4a4c-beb7-bb4405007c8e","Type":"ContainerStarted","Data":"7827613e08aadb178c1e4e5ed265e22cad2b5307fa08fb0880053c06ef00fbf5"}
May 25 13:07:05.836019 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:05.835932    2580 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-ql787" podStartSLOduration=2.033594367 podStartE2EDuration="3.835916707s" podCreationTimestamp="2026-05-25 13:07:02 +0000 UTC" firstStartedPulling="2026-05-25 13:07:03.388020493 +0000 UTC m=+123.580314831" lastFinishedPulling="2026-05-25 13:07:05.190342832 +0000 UTC m=+125.382637171" observedRunningTime="2026-05-25 13:07:05.835141448 +0000 UTC m=+126.027435808" watchObservedRunningTime="2026-05-25 13:07:05.835916707 +0000 UTC m=+126.028211069"
May 25 13:07:07.487722 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:07.487683    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/264a8628-6305-4840-9925-88aeafe69834-service-ca-bundle\") pod \"router-default-549b58d588-dnld2\" (UID: \"264a8628-6305-4840-9925-88aeafe69834\") " pod="openshift-ingress/router-default-549b58d588-dnld2"
May 25 13:07:07.488236 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:07.488218    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/264a8628-6305-4840-9925-88aeafe69834-service-ca-bundle\") pod \"router-default-549b58d588-dnld2\" (UID: \"264a8628-6305-4840-9925-88aeafe69834\") " pod="openshift-ingress/router-default-549b58d588-dnld2"
May 25 13:07:07.780956 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:07.780878    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-8b5l2\""
May 25 13:07:07.789722 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:07.789697    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-549b58d588-dnld2"
May 25 13:07:07.914268 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:07.914236    2580 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-549b58d588-dnld2"]
May 25 13:07:07.917476 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:07:07.917451    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod264a8628_6305_4840_9925_88aeafe69834.slice/crio-4bbb95a71978121368f8420f32f081bf3b572c53d0b25f216130520a32a5cb5b WatchSource:0}: Error finding container 4bbb95a71978121368f8420f32f081bf3b572c53d0b25f216130520a32a5cb5b: Status 404 returned error can't find the container with id 4bbb95a71978121368f8420f32f081bf3b572c53d0b25f216130520a32a5cb5b
May 25 13:07:08.825987 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:08.825951    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-549b58d588-dnld2" event={"ID":"264a8628-6305-4840-9925-88aeafe69834","Type":"ContainerStarted","Data":"1d4dd6b4863f61757f4792a1da142326f2ab1b5f9e60c59167a9cb48298b5119"}
May 25 13:07:08.825987 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:08.825986    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-549b58d588-dnld2" event={"ID":"264a8628-6305-4840-9925-88aeafe69834","Type":"ContainerStarted","Data":"4bbb95a71978121368f8420f32f081bf3b572c53d0b25f216130520a32a5cb5b"}
May 25 13:07:08.851754 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:08.851700    2580 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ingress/router-default-549b58d588-dnld2" podStartSLOduration=33.851683769 podStartE2EDuration="33.851683769s" podCreationTimestamp="2026-05-25 13:06:35 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 13:07:08.850798429 +0000 UTC m=+129.043092789" watchObservedRunningTime="2026-05-25 13:07:08.851683769 +0000 UTC m=+129.043978131"
May 25 13:07:09.790343 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:09.790299    2580 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-549b58d588-dnld2"
May 25 13:07:09.793324 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:09.793298    2580 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-549b58d588-dnld2"
May 25 13:07:09.828428 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:09.828392    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-549b58d588-dnld2"
May 25 13:07:09.829678 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:09.829657    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-549b58d588-dnld2"
May 25 13:07:10.108074 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:10.108030    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e445efe1-4ac4-4df5-881f-748bcf8c5e0f-metrics-certs\") pod \"network-metrics-daemon-svlzs\" (UID: \"e445efe1-4ac4-4df5-881f-748bcf8c5e0f\") " pod="openshift-multus/network-metrics-daemon-svlzs"
May 25 13:07:10.110498 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:10.110469    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e445efe1-4ac4-4df5-881f-748bcf8c5e0f-metrics-certs\") pod \"network-metrics-daemon-svlzs\" (UID: \"e445efe1-4ac4-4df5-881f-748bcf8c5e0f\") " pod="openshift-multus/network-metrics-daemon-svlzs"
May 25 13:07:10.386773 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:10.386695    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-rx2c5\""
May 25 13:07:10.394619 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:10.394577    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-svlzs"
May 25 13:07:10.517527 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:10.517492    2580 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-svlzs"]
May 25 13:07:10.521511 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:07:10.521483    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode445efe1_4ac4_4df5_881f_748bcf8c5e0f.slice/crio-894e2b9a68972f85d22f3289cc5cf4b1e0c1ba415f78baaf6c6d01240af97df6 WatchSource:0}: Error finding container 894e2b9a68972f85d22f3289cc5cf4b1e0c1ba415f78baaf6c6d01240af97df6: Status 404 returned error can't find the container with id 894e2b9a68972f85d22f3289cc5cf4b1e0c1ba415f78baaf6c6d01240af97df6
May 25 13:07:10.832069 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:10.831980    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-svlzs" event={"ID":"e445efe1-4ac4-4df5-881f-748bcf8c5e0f","Type":"ContainerStarted","Data":"894e2b9a68972f85d22f3289cc5cf4b1e0c1ba415f78baaf6c6d01240af97df6"}
May 25 13:07:11.838278 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:11.838243    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-svlzs" event={"ID":"e445efe1-4ac4-4df5-881f-748bcf8c5e0f","Type":"ContainerStarted","Data":"2959201cb5ccd8ce7008cafdb1bc102b5bb322640e491e5fb6bec177d0a1ed57"}
May 25 13:07:11.838278 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:11.838281    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-svlzs" event={"ID":"e445efe1-4ac4-4df5-881f-748bcf8c5e0f","Type":"ContainerStarted","Data":"dac0ebc7332c869f5b21d4c5a0b3c0d67b9cb733d756897550ee8ead16ed3365"}
May 25 13:07:11.859237 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:11.859193    2580 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-svlzs" podStartSLOduration=130.854376191 podStartE2EDuration="2m11.859179537s" podCreationTimestamp="2026-05-25 13:05:00 +0000 UTC" firstStartedPulling="2026-05-25 13:07:10.523455268 +0000 UTC m=+130.715749613" lastFinishedPulling="2026-05-25 13:07:11.528258622 +0000 UTC m=+131.720552959" observedRunningTime="2026-05-25 13:07:11.858112822 +0000 UTC m=+132.050407193" watchObservedRunningTime="2026-05-25 13:07:11.859179537 +0000 UTC m=+132.051473941"
May 25 13:07:12.316498 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:12.316412    2580 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-7779bbdd78-x4h7v"]
May 25 13:07:12.319234 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:12.319217    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-7779bbdd78-x4h7v"
May 25 13:07:12.323583 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:12.323557    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
May 25 13:07:12.323721 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:12.323627    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
May 25 13:07:12.323781 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:12.323766    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-bc8t5\""
May 25 13:07:12.323901 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:12.323885    2580 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
May 25 13:07:12.324766 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:12.324742    2580 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
May 25 13:07:12.324887 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:12.324745    2580 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
May 25 13:07:12.331939 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:12.331914    2580 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-7779bbdd78-x4h7v"]
May 25 13:07:12.427976 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:12.427944    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2eceea46-f0be-4ff7-9b90-45761f6431e0-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7779bbdd78-x4h7v\" (UID: \"2eceea46-f0be-4ff7-9b90-45761f6431e0\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-x4h7v"
May 25 13:07:12.428136 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:12.427987    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8cws\" (UniqueName: \"kubernetes.io/projected/2eceea46-f0be-4ff7-9b90-45761f6431e0-kube-api-access-h8cws\") pod \"prometheus-operator-7779bbdd78-x4h7v\" (UID: \"2eceea46-f0be-4ff7-9b90-45761f6431e0\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-x4h7v"
May 25 13:07:12.428136 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:12.428055    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/2eceea46-f0be-4ff7-9b90-45761f6431e0-prometheus-operator-tls\") pod \"prometheus-operator-7779bbdd78-x4h7v\" (UID: \"2eceea46-f0be-4ff7-9b90-45761f6431e0\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-x4h7v"
May 25 13:07:12.428136 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:12.428124    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2eceea46-f0be-4ff7-9b90-45761f6431e0-metrics-client-ca\") pod \"prometheus-operator-7779bbdd78-x4h7v\" (UID: \"2eceea46-f0be-4ff7-9b90-45761f6431e0\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-x4h7v"
May 25 13:07:12.529223 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:12.529183    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2eceea46-f0be-4ff7-9b90-45761f6431e0-metrics-client-ca\") pod \"prometheus-operator-7779bbdd78-x4h7v\" (UID: \"2eceea46-f0be-4ff7-9b90-45761f6431e0\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-x4h7v"
May 25 13:07:12.529223 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:12.529229    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2eceea46-f0be-4ff7-9b90-45761f6431e0-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7779bbdd78-x4h7v\" (UID: \"2eceea46-f0be-4ff7-9b90-45761f6431e0\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-x4h7v"
May 25 13:07:12.529479 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:12.529253    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-h8cws\" (UniqueName: \"kubernetes.io/projected/2eceea46-f0be-4ff7-9b90-45761f6431e0-kube-api-access-h8cws\") pod \"prometheus-operator-7779bbdd78-x4h7v\" (UID: \"2eceea46-f0be-4ff7-9b90-45761f6431e0\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-x4h7v"
May 25 13:07:12.529479 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:12.529289    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/2eceea46-f0be-4ff7-9b90-45761f6431e0-prometheus-operator-tls\") pod \"prometheus-operator-7779bbdd78-x4h7v\" (UID: \"2eceea46-f0be-4ff7-9b90-45761f6431e0\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-x4h7v"
May 25 13:07:12.530030 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:12.529969    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2eceea46-f0be-4ff7-9b90-45761f6431e0-metrics-client-ca\") pod \"prometheus-operator-7779bbdd78-x4h7v\" (UID: \"2eceea46-f0be-4ff7-9b90-45761f6431e0\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-x4h7v"
May 25 13:07:12.531946 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:12.531920    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/2eceea46-f0be-4ff7-9b90-45761f6431e0-prometheus-operator-tls\") pod \"prometheus-operator-7779bbdd78-x4h7v\" (UID: \"2eceea46-f0be-4ff7-9b90-45761f6431e0\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-x4h7v"
May 25 13:07:12.532045 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:12.532009    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2eceea46-f0be-4ff7-9b90-45761f6431e0-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7779bbdd78-x4h7v\" (UID: \"2eceea46-f0be-4ff7-9b90-45761f6431e0\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-x4h7v"
May 25 13:07:12.537865 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:12.537833    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8cws\" (UniqueName: \"kubernetes.io/projected/2eceea46-f0be-4ff7-9b90-45761f6431e0-kube-api-access-h8cws\") pod \"prometheus-operator-7779bbdd78-x4h7v\" (UID: \"2eceea46-f0be-4ff7-9b90-45761f6431e0\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-x4h7v"
May 25 13:07:12.628175 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:12.628137    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-7779bbdd78-x4h7v"
May 25 13:07:12.781853 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:12.781733    2580 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-7779bbdd78-x4h7v"]
May 25 13:07:12.784351 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:07:12.784321    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2eceea46_f0be_4ff7_9b90_45761f6431e0.slice/crio-79417c6a2e9f0e5e344f139043705bb76c18d94c3b7edb89228a7f842aec2e8d WatchSource:0}: Error finding container 79417c6a2e9f0e5e344f139043705bb76c18d94c3b7edb89228a7f842aec2e8d: Status 404 returned error can't find the container with id 79417c6a2e9f0e5e344f139043705bb76c18d94c3b7edb89228a7f842aec2e8d
May 25 13:07:12.842091 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:12.842054    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7779bbdd78-x4h7v" event={"ID":"2eceea46-f0be-4ff7-9b90-45761f6431e0","Type":"ContainerStarted","Data":"79417c6a2e9f0e5e344f139043705bb76c18d94c3b7edb89228a7f842aec2e8d"}
May 25 13:07:12.935399 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:12.935288    2580 patch_prober.go:28] interesting pod/image-registry-769864757b-hs9k5 container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
May 25 13:07:12.935399 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:12.935360    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-769864757b-hs9k5" podUID="f21589b0-1b80-46bb-92e8-1252201ed8f3" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 13:07:14.848014 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:14.847970    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7779bbdd78-x4h7v" event={"ID":"2eceea46-f0be-4ff7-9b90-45761f6431e0","Type":"ContainerStarted","Data":"ffbe661998d317e4a777a6eee945a2e3955b453eb59ced2134733037b2b9bc46"}
May 25 13:07:14.848014 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:14.848015    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7779bbdd78-x4h7v" event={"ID":"2eceea46-f0be-4ff7-9b90-45761f6431e0","Type":"ContainerStarted","Data":"c25bc35fbde27fcf3217ce3a3512112fb5622b7c4b3eb5c9dddb52535c1f77e5"}
May 25 13:07:14.863654 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:14.863593    2580 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-7779bbdd78-x4h7v" podStartSLOduration=1.683116769 podStartE2EDuration="2.863578569s" podCreationTimestamp="2026-05-25 13:07:12 +0000 UTC" firstStartedPulling="2026-05-25 13:07:12.786230796 +0000 UTC m=+132.978525138" lastFinishedPulling="2026-05-25 13:07:13.9666926 +0000 UTC m=+134.158986938" observedRunningTime="2026-05-25 13:07:14.863476789 +0000 UTC m=+135.055771170" watchObservedRunningTime="2026-05-25 13:07:14.863578569 +0000 UTC m=+135.055872923"
May 25 13:07:14.970270 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:14.970237    2580 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console/console-5895d9d8f5-rwfmg"]
May 25 13:07:14.973194 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:14.973177    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5895d9d8f5-rwfmg"
May 25 13:07:14.975859 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:14.975837    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
May 25 13:07:14.975982 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:14.975936    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
May 25 13:07:14.975982 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:14.975954    2580 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
May 25 13:07:14.976087 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:14.975996    2580 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
May 25 13:07:14.976349 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:14.976332    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-8jjtw\""
May 25 13:07:14.976466 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:14.976399    2580 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
May 25 13:07:14.976655 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:14.976639    2580 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
May 25 13:07:14.976929 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:14.976911    2580 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
May 25 13:07:14.982170 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:14.982154    2580 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
May 25 13:07:14.987798 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:14.987774    2580 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5895d9d8f5-rwfmg"]
May 25 13:07:15.153526 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:15.153426    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2f2c2a43-8334-476b-91ad-49475b5fb7b0-service-ca\") pod \"console-5895d9d8f5-rwfmg\" (UID: \"2f2c2a43-8334-476b-91ad-49475b5fb7b0\") " pod="openshift-console/console-5895d9d8f5-rwfmg"
May 25 13:07:15.153526 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:15.153481    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2f2c2a43-8334-476b-91ad-49475b5fb7b0-console-config\") pod \"console-5895d9d8f5-rwfmg\" (UID: \"2f2c2a43-8334-476b-91ad-49475b5fb7b0\") " pod="openshift-console/console-5895d9d8f5-rwfmg"
May 25 13:07:15.153526 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:15.153506    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2f2c2a43-8334-476b-91ad-49475b5fb7b0-oauth-serving-cert\") pod \"console-5895d9d8f5-rwfmg\" (UID: \"2f2c2a43-8334-476b-91ad-49475b5fb7b0\") " pod="openshift-console/console-5895d9d8f5-rwfmg"
May 25 13:07:15.153742 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:15.153600    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2f2c2a43-8334-476b-91ad-49475b5fb7b0-console-serving-cert\") pod \"console-5895d9d8f5-rwfmg\" (UID: \"2f2c2a43-8334-476b-91ad-49475b5fb7b0\") " pod="openshift-console/console-5895d9d8f5-rwfmg"
May 25 13:07:15.153742 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:15.153642    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2f2c2a43-8334-476b-91ad-49475b5fb7b0-console-oauth-config\") pod \"console-5895d9d8f5-rwfmg\" (UID: \"2f2c2a43-8334-476b-91ad-49475b5fb7b0\") " pod="openshift-console/console-5895d9d8f5-rwfmg"
May 25 13:07:15.153742 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:15.153702    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qk9k\" (UniqueName: \"kubernetes.io/projected/2f2c2a43-8334-476b-91ad-49475b5fb7b0-kube-api-access-5qk9k\") pod \"console-5895d9d8f5-rwfmg\" (UID: \"2f2c2a43-8334-476b-91ad-49475b5fb7b0\") " pod="openshift-console/console-5895d9d8f5-rwfmg"
May 25 13:07:15.153742 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:15.153729    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2f2c2a43-8334-476b-91ad-49475b5fb7b0-trusted-ca-bundle\") pod \"console-5895d9d8f5-rwfmg\" (UID: \"2f2c2a43-8334-476b-91ad-49475b5fb7b0\") " pod="openshift-console/console-5895d9d8f5-rwfmg"
May 25 13:07:15.254990 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:15.254941    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-5qk9k\" (UniqueName: \"kubernetes.io/projected/2f2c2a43-8334-476b-91ad-49475b5fb7b0-kube-api-access-5qk9k\") pod \"console-5895d9d8f5-rwfmg\" (UID: \"2f2c2a43-8334-476b-91ad-49475b5fb7b0\") " pod="openshift-console/console-5895d9d8f5-rwfmg"
May 25 13:07:15.254990 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:15.254998    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2f2c2a43-8334-476b-91ad-49475b5fb7b0-trusted-ca-bundle\") pod \"console-5895d9d8f5-rwfmg\" (UID: \"2f2c2a43-8334-476b-91ad-49475b5fb7b0\") " pod="openshift-console/console-5895d9d8f5-rwfmg"
May 25 13:07:15.255163 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:15.255035    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2f2c2a43-8334-476b-91ad-49475b5fb7b0-service-ca\") pod \"console-5895d9d8f5-rwfmg\" (UID: \"2f2c2a43-8334-476b-91ad-49475b5fb7b0\") " pod="openshift-console/console-5895d9d8f5-rwfmg"
May 25 13:07:15.255163 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:15.255056    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2f2c2a43-8334-476b-91ad-49475b5fb7b0-console-config\") pod \"console-5895d9d8f5-rwfmg\" (UID: \"2f2c2a43-8334-476b-91ad-49475b5fb7b0\") " pod="openshift-console/console-5895d9d8f5-rwfmg"
May 25 13:07:15.255163 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:15.255078    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2f2c2a43-8334-476b-91ad-49475b5fb7b0-oauth-serving-cert\") pod \"console-5895d9d8f5-rwfmg\" (UID: \"2f2c2a43-8334-476b-91ad-49475b5fb7b0\") " pod="openshift-console/console-5895d9d8f5-rwfmg"
May 25 13:07:15.255163 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:15.255108    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2f2c2a43-8334-476b-91ad-49475b5fb7b0-console-serving-cert\") pod \"console-5895d9d8f5-rwfmg\" (UID: \"2f2c2a43-8334-476b-91ad-49475b5fb7b0\") " pod="openshift-console/console-5895d9d8f5-rwfmg"
May 25 13:07:15.255316 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:15.255243    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2f2c2a43-8334-476b-91ad-49475b5fb7b0-console-oauth-config\") pod \"console-5895d9d8f5-rwfmg\" (UID: \"2f2c2a43-8334-476b-91ad-49475b5fb7b0\") " pod="openshift-console/console-5895d9d8f5-rwfmg"
May 25 13:07:15.255947 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:15.255923    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2f2c2a43-8334-476b-91ad-49475b5fb7b0-console-config\") pod \"console-5895d9d8f5-rwfmg\" (UID: \"2f2c2a43-8334-476b-91ad-49475b5fb7b0\") " pod="openshift-console/console-5895d9d8f5-rwfmg"
May 25 13:07:15.256050 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:15.255954    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2f2c2a43-8334-476b-91ad-49475b5fb7b0-service-ca\") pod \"console-5895d9d8f5-rwfmg\" (UID: \"2f2c2a43-8334-476b-91ad-49475b5fb7b0\") " pod="openshift-console/console-5895d9d8f5-rwfmg"
May 25 13:07:15.256128 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:15.255913    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2f2c2a43-8334-476b-91ad-49475b5fb7b0-oauth-serving-cert\") pod \"console-5895d9d8f5-rwfmg\" (UID: \"2f2c2a43-8334-476b-91ad-49475b5fb7b0\") " pod="openshift-console/console-5895d9d8f5-rwfmg"
May 25 13:07:15.256164 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:15.256110    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2f2c2a43-8334-476b-91ad-49475b5fb7b0-trusted-ca-bundle\") pod \"console-5895d9d8f5-rwfmg\" (UID: \"2f2c2a43-8334-476b-91ad-49475b5fb7b0\") " pod="openshift-console/console-5895d9d8f5-rwfmg"
May 25 13:07:15.257813 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:15.257790    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2f2c2a43-8334-476b-91ad-49475b5fb7b0-console-oauth-config\") pod \"console-5895d9d8f5-rwfmg\" (UID: \"2f2c2a43-8334-476b-91ad-49475b5fb7b0\") " pod="openshift-console/console-5895d9d8f5-rwfmg"
May 25 13:07:15.257926 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:15.257913    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2f2c2a43-8334-476b-91ad-49475b5fb7b0-console-serving-cert\") pod \"console-5895d9d8f5-rwfmg\" (UID: \"2f2c2a43-8334-476b-91ad-49475b5fb7b0\") " pod="openshift-console/console-5895d9d8f5-rwfmg"
May 25 13:07:15.264618 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:15.264594    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qk9k\" (UniqueName: \"kubernetes.io/projected/2f2c2a43-8334-476b-91ad-49475b5fb7b0-kube-api-access-5qk9k\") pod \"console-5895d9d8f5-rwfmg\" (UID: \"2f2c2a43-8334-476b-91ad-49475b5fb7b0\") " pod="openshift-console/console-5895d9d8f5-rwfmg"
May 25 13:07:15.282625 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:15.282588    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5895d9d8f5-rwfmg"
May 25 13:07:15.411899 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:15.411817    2580 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5895d9d8f5-rwfmg"]
May 25 13:07:15.414968 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:07:15.414938    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f2c2a43_8334_476b_91ad_49475b5fb7b0.slice/crio-33cd5e1db7c755c284ba469e9536937618b878bd90ce9c518d51427eb50de28a WatchSource:0}: Error finding container 33cd5e1db7c755c284ba469e9536937618b878bd90ce9c518d51427eb50de28a: Status 404 returned error can't find the container with id 33cd5e1db7c755c284ba469e9536937618b878bd90ce9c518d51427eb50de28a
May 25 13:07:15.852123 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:15.852091    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5895d9d8f5-rwfmg" event={"ID":"2f2c2a43-8334-476b-91ad-49475b5fb7b0","Type":"ContainerStarted","Data":"33cd5e1db7c755c284ba469e9536937618b878bd90ce9c518d51427eb50de28a"}
May 25 13:07:16.722462 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:16.722421    2580 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-j2jrt"]
May 25 13:07:16.726165 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:16.726141    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-j2jrt"
May 25 13:07:16.730032 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:16.730002    2580 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
May 25 13:07:16.730152 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:16.730043    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
May 25 13:07:16.730152 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:16.730067    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-fvln4\""
May 25 13:07:16.730152 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:16.730067    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
May 25 13:07:16.868629 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:16.868597    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/7355e93e-5dee-448a-a74a-2f54c9d5e943-node-exporter-accelerators-collector-config\") pod \"node-exporter-j2jrt\" (UID: \"7355e93e-5dee-448a-a74a-2f54c9d5e943\") " pod="openshift-monitoring/node-exporter-j2jrt"
May 25 13:07:16.869077 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:16.868655    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/7355e93e-5dee-448a-a74a-2f54c9d5e943-root\") pod \"node-exporter-j2jrt\" (UID: \"7355e93e-5dee-448a-a74a-2f54c9d5e943\") " pod="openshift-monitoring/node-exporter-j2jrt"
May 25 13:07:16.869077 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:16.868682    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7355e93e-5dee-448a-a74a-2f54c9d5e943-metrics-client-ca\") pod \"node-exporter-j2jrt\" (UID: \"7355e93e-5dee-448a-a74a-2f54c9d5e943\") " pod="openshift-monitoring/node-exporter-j2jrt"
May 25 13:07:16.869077 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:16.868771    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7355e93e-5dee-448a-a74a-2f54c9d5e943-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-j2jrt\" (UID: \"7355e93e-5dee-448a-a74a-2f54c9d5e943\") " pod="openshift-monitoring/node-exporter-j2jrt"
May 25 13:07:16.869077 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:16.868808    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7355e93e-5dee-448a-a74a-2f54c9d5e943-node-exporter-tls\") pod \"node-exporter-j2jrt\" (UID: \"7355e93e-5dee-448a-a74a-2f54c9d5e943\") " pod="openshift-monitoring/node-exporter-j2jrt"
May 25 13:07:16.869077 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:16.868847    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rq6x6\" (UniqueName: \"kubernetes.io/projected/7355e93e-5dee-448a-a74a-2f54c9d5e943-kube-api-access-rq6x6\") pod \"node-exporter-j2jrt\" (UID: \"7355e93e-5dee-448a-a74a-2f54c9d5e943\") " pod="openshift-monitoring/node-exporter-j2jrt"
May 25 13:07:16.869077 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:16.868925    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/7355e93e-5dee-448a-a74a-2f54c9d5e943-node-exporter-wtmp\") pod \"node-exporter-j2jrt\" (UID: \"7355e93e-5dee-448a-a74a-2f54c9d5e943\") " pod="openshift-monitoring/node-exporter-j2jrt"
May 25 13:07:16.869077 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:16.868958    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/7355e93e-5dee-448a-a74a-2f54c9d5e943-node-exporter-textfile\") pod \"node-exporter-j2jrt\" (UID: \"7355e93e-5dee-448a-a74a-2f54c9d5e943\") " pod="openshift-monitoring/node-exporter-j2jrt"
May 25 13:07:16.869077 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:16.868984    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7355e93e-5dee-448a-a74a-2f54c9d5e943-sys\") pod \"node-exporter-j2jrt\" (UID: \"7355e93e-5dee-448a-a74a-2f54c9d5e943\") " pod="openshift-monitoring/node-exporter-j2jrt"
May 25 13:07:16.969969 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:16.969936    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/7355e93e-5dee-448a-a74a-2f54c9d5e943-node-exporter-wtmp\") pod \"node-exporter-j2jrt\" (UID: \"7355e93e-5dee-448a-a74a-2f54c9d5e943\") " pod="openshift-monitoring/node-exporter-j2jrt"
May 25 13:07:16.970148 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:16.969978    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/7355e93e-5dee-448a-a74a-2f54c9d5e943-node-exporter-textfile\") pod \"node-exporter-j2jrt\" (UID: \"7355e93e-5dee-448a-a74a-2f54c9d5e943\") " pod="openshift-monitoring/node-exporter-j2jrt"
May 25 13:07:16.970148 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:16.970009    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7355e93e-5dee-448a-a74a-2f54c9d5e943-sys\") pod \"node-exporter-j2jrt\" (UID: \"7355e93e-5dee-448a-a74a-2f54c9d5e943\") " pod="openshift-monitoring/node-exporter-j2jrt"
May 25 13:07:16.970148 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:16.970042    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/7355e93e-5dee-448a-a74a-2f54c9d5e943-node-exporter-accelerators-collector-config\") pod \"node-exporter-j2jrt\" (UID: \"7355e93e-5dee-448a-a74a-2f54c9d5e943\") " pod="openshift-monitoring/node-exporter-j2jrt"
May 25 13:07:16.970148 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:16.970091    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/7355e93e-5dee-448a-a74a-2f54c9d5e943-root\") pod \"node-exporter-j2jrt\" (UID: \"7355e93e-5dee-448a-a74a-2f54c9d5e943\") " pod="openshift-monitoring/node-exporter-j2jrt"
May 25 13:07:16.970148 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:16.970122    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/7355e93e-5dee-448a-a74a-2f54c9d5e943-node-exporter-wtmp\") pod \"node-exporter-j2jrt\" (UID: \"7355e93e-5dee-448a-a74a-2f54c9d5e943\") " pod="openshift-monitoring/node-exporter-j2jrt"
May 25 13:07:16.970424 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:16.970160    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7355e93e-5dee-448a-a74a-2f54c9d5e943-sys\") pod \"node-exporter-j2jrt\" (UID: \"7355e93e-5dee-448a-a74a-2f54c9d5e943\") " pod="openshift-monitoring/node-exporter-j2jrt"
May 25 13:07:16.970424 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:16.970167    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7355e93e-5dee-448a-a74a-2f54c9d5e943-metrics-client-ca\") pod \"node-exporter-j2jrt\" (UID: \"7355e93e-5dee-448a-a74a-2f54c9d5e943\") " pod="openshift-monitoring/node-exporter-j2jrt"
May 25 13:07:16.970424 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:16.970273    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7355e93e-5dee-448a-a74a-2f54c9d5e943-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-j2jrt\" (UID: \"7355e93e-5dee-448a-a74a-2f54c9d5e943\") " pod="openshift-monitoring/node-exporter-j2jrt"
May 25 13:07:16.970424 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:16.970316    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7355e93e-5dee-448a-a74a-2f54c9d5e943-node-exporter-tls\") pod \"node-exporter-j2jrt\" (UID: \"7355e93e-5dee-448a-a74a-2f54c9d5e943\") " pod="openshift-monitoring/node-exporter-j2jrt"
May 25 13:07:16.970424 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:16.970393    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/7355e93e-5dee-448a-a74a-2f54c9d5e943-node-exporter-textfile\") pod \"node-exporter-j2jrt\" (UID: \"7355e93e-5dee-448a-a74a-2f54c9d5e943\") " pod="openshift-monitoring/node-exporter-j2jrt"
May 25 13:07:16.970598 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:16.970361    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-rq6x6\" (UniqueName: \"kubernetes.io/projected/7355e93e-5dee-448a-a74a-2f54c9d5e943-kube-api-access-rq6x6\") pod \"node-exporter-j2jrt\" (UID: \"7355e93e-5dee-448a-a74a-2f54c9d5e943\") " pod="openshift-monitoring/node-exporter-j2jrt"
May 25 13:07:16.970696 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:16.970675    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7355e93e-5dee-448a-a74a-2f54c9d5e943-metrics-client-ca\") pod \"node-exporter-j2jrt\" (UID: \"7355e93e-5dee-448a-a74a-2f54c9d5e943\") " pod="openshift-monitoring/node-exporter-j2jrt"
May 25 13:07:16.970759 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:16.970675    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/7355e93e-5dee-448a-a74a-2f54c9d5e943-node-exporter-accelerators-collector-config\") pod \"node-exporter-j2jrt\" (UID: \"7355e93e-5dee-448a-a74a-2f54c9d5e943\") " pod="openshift-monitoring/node-exporter-j2jrt"
May 25 13:07:16.970759 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:16.970713    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/7355e93e-5dee-448a-a74a-2f54c9d5e943-root\") pod \"node-exporter-j2jrt\" (UID: \"7355e93e-5dee-448a-a74a-2f54c9d5e943\") " pod="openshift-monitoring/node-exporter-j2jrt"
May 25 13:07:16.970862 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:07:16.970771    2580 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
May 25 13:07:16.970862 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:07:16.970841    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7355e93e-5dee-448a-a74a-2f54c9d5e943-node-exporter-tls podName:7355e93e-5dee-448a-a74a-2f54c9d5e943 nodeName:}" failed. No retries permitted until 2026-05-25 13:07:17.47082276 +0000 UTC m=+137.663117129 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/7355e93e-5dee-448a-a74a-2f54c9d5e943-node-exporter-tls") pod "node-exporter-j2jrt" (UID: "7355e93e-5dee-448a-a74a-2f54c9d5e943") : secret "node-exporter-tls" not found
May 25 13:07:16.972939 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:16.972883    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7355e93e-5dee-448a-a74a-2f54c9d5e943-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-j2jrt\" (UID: \"7355e93e-5dee-448a-a74a-2f54c9d5e943\") " pod="openshift-monitoring/node-exporter-j2jrt"
May 25 13:07:16.982587 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:16.982562    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rq6x6\" (UniqueName: \"kubernetes.io/projected/7355e93e-5dee-448a-a74a-2f54c9d5e943-kube-api-access-rq6x6\") pod \"node-exporter-j2jrt\" (UID: \"7355e93e-5dee-448a-a74a-2f54c9d5e943\") " pod="openshift-monitoring/node-exporter-j2jrt"
May 25 13:07:17.475657 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:17.475614    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7355e93e-5dee-448a-a74a-2f54c9d5e943-node-exporter-tls\") pod \"node-exporter-j2jrt\" (UID: \"7355e93e-5dee-448a-a74a-2f54c9d5e943\") " pod="openshift-monitoring/node-exporter-j2jrt"
May 25 13:07:17.478414 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:17.478361    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7355e93e-5dee-448a-a74a-2f54c9d5e943-node-exporter-tls\") pod \"node-exporter-j2jrt\" (UID: \"7355e93e-5dee-448a-a74a-2f54c9d5e943\") " pod="openshift-monitoring/node-exporter-j2jrt"
May 25 13:07:17.637566 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:17.637520    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-j2jrt"
May 25 13:07:17.814261 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:17.813802    2580 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
May 25 13:07:17.823728 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:17.823697    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
May 25 13:07:17.829451 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:17.827036    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-q4l9n\""
May 25 13:07:17.829451 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:17.827104    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
May 25 13:07:17.829451 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:17.827265    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
May 25 13:07:17.829451 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:17.827348    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
May 25 13:07:17.829451 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:17.827616    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
May 25 13:07:17.829451 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:17.828137    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
May 25 13:07:17.829451 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:17.828343    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
May 25 13:07:17.829451 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:17.828559    2580 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
May 25 13:07:17.829451 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:17.828737    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
May 25 13:07:17.829451 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:17.828906    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
May 25 13:07:17.829451 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:17.829417    2580 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
May 25 13:07:17.981028 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:17.980980    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/693562c8-a5e4-4462-92b9-8aca51cb1d27-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"693562c8-a5e4-4462-92b9-8aca51cb1d27\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 13:07:17.981487 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:17.981049    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/693562c8-a5e4-4462-92b9-8aca51cb1d27-config-volume\") pod \"alertmanager-main-0\" (UID: \"693562c8-a5e4-4462-92b9-8aca51cb1d27\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 13:07:17.981487 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:17.981090    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/693562c8-a5e4-4462-92b9-8aca51cb1d27-web-config\") pod \"alertmanager-main-0\" (UID: \"693562c8-a5e4-4462-92b9-8aca51cb1d27\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 13:07:17.981487 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:17.981236    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/693562c8-a5e4-4462-92b9-8aca51cb1d27-tls-assets\") pod \"alertmanager-main-0\" (UID: \"693562c8-a5e4-4462-92b9-8aca51cb1d27\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 13:07:17.981487 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:17.981282    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/693562c8-a5e4-4462-92b9-8aca51cb1d27-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"693562c8-a5e4-4462-92b9-8aca51cb1d27\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 13:07:17.981487 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:17.981324    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/693562c8-a5e4-4462-92b9-8aca51cb1d27-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"693562c8-a5e4-4462-92b9-8aca51cb1d27\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 13:07:17.981487 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:17.981385    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/693562c8-a5e4-4462-92b9-8aca51cb1d27-config-out\") pod \"alertmanager-main-0\" (UID: \"693562c8-a5e4-4462-92b9-8aca51cb1d27\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 13:07:17.981487 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:17.981409    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhhht\" (UniqueName: \"kubernetes.io/projected/693562c8-a5e4-4462-92b9-8aca51cb1d27-kube-api-access-qhhht\") pod \"alertmanager-main-0\" (UID: \"693562c8-a5e4-4462-92b9-8aca51cb1d27\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 13:07:17.981487 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:17.981456    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/693562c8-a5e4-4462-92b9-8aca51cb1d27-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"693562c8-a5e4-4462-92b9-8aca51cb1d27\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 13:07:17.981827 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:17.981525    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/693562c8-a5e4-4462-92b9-8aca51cb1d27-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"693562c8-a5e4-4462-92b9-8aca51cb1d27\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 13:07:17.981827 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:17.981563    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/693562c8-a5e4-4462-92b9-8aca51cb1d27-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"693562c8-a5e4-4462-92b9-8aca51cb1d27\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 13:07:17.981827 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:17.981605    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/693562c8-a5e4-4462-92b9-8aca51cb1d27-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"693562c8-a5e4-4462-92b9-8aca51cb1d27\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 13:07:17.981827 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:17.981630    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/693562c8-a5e4-4462-92b9-8aca51cb1d27-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"693562c8-a5e4-4462-92b9-8aca51cb1d27\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 13:07:18.082509 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:18.082472    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/693562c8-a5e4-4462-92b9-8aca51cb1d27-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"693562c8-a5e4-4462-92b9-8aca51cb1d27\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 13:07:18.082692 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:18.082525    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/693562c8-a5e4-4462-92b9-8aca51cb1d27-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"693562c8-a5e4-4462-92b9-8aca51cb1d27\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 13:07:18.082692 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:18.082568    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/693562c8-a5e4-4462-92b9-8aca51cb1d27-config-volume\") pod \"alertmanager-main-0\" (UID: \"693562c8-a5e4-4462-92b9-8aca51cb1d27\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 13:07:18.082692 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:18.082590    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/693562c8-a5e4-4462-92b9-8aca51cb1d27-web-config\") pod \"alertmanager-main-0\" (UID: \"693562c8-a5e4-4462-92b9-8aca51cb1d27\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 13:07:18.082836 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:18.082740    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/693562c8-a5e4-4462-92b9-8aca51cb1d27-tls-assets\") pod \"alertmanager-main-0\" (UID: \"693562c8-a5e4-4462-92b9-8aca51cb1d27\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 13:07:18.082836 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:18.082778    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/693562c8-a5e4-4462-92b9-8aca51cb1d27-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"693562c8-a5e4-4462-92b9-8aca51cb1d27\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 13:07:18.082836 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:18.082814    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/693562c8-a5e4-4462-92b9-8aca51cb1d27-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"693562c8-a5e4-4462-92b9-8aca51cb1d27\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 13:07:18.082984 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:18.082859    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/693562c8-a5e4-4462-92b9-8aca51cb1d27-config-out\") pod \"alertmanager-main-0\" (UID: \"693562c8-a5e4-4462-92b9-8aca51cb1d27\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 13:07:18.082984 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:18.082886    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-qhhht\" (UniqueName: \"kubernetes.io/projected/693562c8-a5e4-4462-92b9-8aca51cb1d27-kube-api-access-qhhht\") pod \"alertmanager-main-0\" (UID: \"693562c8-a5e4-4462-92b9-8aca51cb1d27\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 13:07:18.082984 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:18.082915    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/693562c8-a5e4-4462-92b9-8aca51cb1d27-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"693562c8-a5e4-4462-92b9-8aca51cb1d27\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 13:07:18.083142 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:18.082986    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/693562c8-a5e4-4462-92b9-8aca51cb1d27-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"693562c8-a5e4-4462-92b9-8aca51cb1d27\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 13:07:18.083142 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:18.083017    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/693562c8-a5e4-4462-92b9-8aca51cb1d27-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"693562c8-a5e4-4462-92b9-8aca51cb1d27\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 13:07:18.083142 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:18.083079    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/693562c8-a5e4-4462-92b9-8aca51cb1d27-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"693562c8-a5e4-4462-92b9-8aca51cb1d27\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 13:07:18.083414 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:18.083331    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/693562c8-a5e4-4462-92b9-8aca51cb1d27-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"693562c8-a5e4-4462-92b9-8aca51cb1d27\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 13:07:18.084417 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:18.084138    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/693562c8-a5e4-4462-92b9-8aca51cb1d27-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"693562c8-a5e4-4462-92b9-8aca51cb1d27\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 13:07:18.084917 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:18.084643    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/693562c8-a5e4-4462-92b9-8aca51cb1d27-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"693562c8-a5e4-4462-92b9-8aca51cb1d27\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 13:07:18.086524 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:18.086066    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/693562c8-a5e4-4462-92b9-8aca51cb1d27-web-config\") pod \"alertmanager-main-0\" (UID: \"693562c8-a5e4-4462-92b9-8aca51cb1d27\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 13:07:18.086524 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:18.086183    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/693562c8-a5e4-4462-92b9-8aca51cb1d27-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"693562c8-a5e4-4462-92b9-8aca51cb1d27\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 13:07:18.086524 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:18.086308    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/693562c8-a5e4-4462-92b9-8aca51cb1d27-tls-assets\") pod \"alertmanager-main-0\" (UID: \"693562c8-a5e4-4462-92b9-8aca51cb1d27\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 13:07:18.086524 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:18.086483    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/693562c8-a5e4-4462-92b9-8aca51cb1d27-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"693562c8-a5e4-4462-92b9-8aca51cb1d27\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 13:07:18.086882 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:18.086856    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/693562c8-a5e4-4462-92b9-8aca51cb1d27-config-volume\") pod \"alertmanager-main-0\" (UID: \"693562c8-a5e4-4462-92b9-8aca51cb1d27\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 13:07:18.087049 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:18.087016    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/693562c8-a5e4-4462-92b9-8aca51cb1d27-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"693562c8-a5e4-4462-92b9-8aca51cb1d27\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 13:07:18.087336 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:18.087316    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/693562c8-a5e4-4462-92b9-8aca51cb1d27-config-out\") pod \"alertmanager-main-0\" (UID: \"693562c8-a5e4-4462-92b9-8aca51cb1d27\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 13:07:18.088260 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:18.088232    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/693562c8-a5e4-4462-92b9-8aca51cb1d27-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"693562c8-a5e4-4462-92b9-8aca51cb1d27\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 13:07:18.088333 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:18.088234    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/693562c8-a5e4-4462-92b9-8aca51cb1d27-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"693562c8-a5e4-4462-92b9-8aca51cb1d27\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 13:07:18.095193 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:18.095169    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhhht\" (UniqueName: \"kubernetes.io/projected/693562c8-a5e4-4462-92b9-8aca51cb1d27-kube-api-access-qhhht\") pod \"alertmanager-main-0\" (UID: \"693562c8-a5e4-4462-92b9-8aca51cb1d27\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 13:07:18.109174 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:07:18.109140    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7355e93e_5dee_448a_a74a_2f54c9d5e943.slice/crio-0e7bdde0ae035cee3a91b4b6318ef37c4d0284dfe0887f72911cc7e128e36bd8 WatchSource:0}: Error finding container 0e7bdde0ae035cee3a91b4b6318ef37c4d0284dfe0887f72911cc7e128e36bd8: Status 404 returned error can't find the container with id 0e7bdde0ae035cee3a91b4b6318ef37c4d0284dfe0887f72911cc7e128e36bd8
May 25 13:07:18.140771 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:18.140732    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
May 25 13:07:18.288773 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:18.288743    2580 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
May 25 13:07:18.291908 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:07:18.291881    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod693562c8_a5e4_4462_92b9_8aca51cb1d27.slice/crio-b443199fde871746fb44b562ac9b53216f2abc4c8c11de4285e0ec21d2b6e007 WatchSource:0}: Error finding container b443199fde871746fb44b562ac9b53216f2abc4c8c11de4285e0ec21d2b6e007: Status 404 returned error can't find the container with id b443199fde871746fb44b562ac9b53216f2abc4c8c11de4285e0ec21d2b6e007
May 25 13:07:18.863127 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:18.863082    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"693562c8-a5e4-4462-92b9-8aca51cb1d27","Type":"ContainerStarted","Data":"b443199fde871746fb44b562ac9b53216f2abc4c8c11de4285e0ec21d2b6e007"}
May 25 13:07:18.864294 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:18.864251    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-j2jrt" event={"ID":"7355e93e-5dee-448a-a74a-2f54c9d5e943","Type":"ContainerStarted","Data":"0e7bdde0ae035cee3a91b4b6318ef37c4d0284dfe0887f72911cc7e128e36bd8"}
May 25 13:07:18.865632 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:18.865607    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5895d9d8f5-rwfmg" event={"ID":"2f2c2a43-8334-476b-91ad-49475b5fb7b0","Type":"ContainerStarted","Data":"aced0f880262508ef83c488494b70f2d79e3f342f20a29c3c91cf807c89169a6"}
May 25 13:07:18.884486 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:18.884444    2580 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console/console-5895d9d8f5-rwfmg" podStartSLOduration=2.14703002 podStartE2EDuration="4.884429387s" podCreationTimestamp="2026-05-25 13:07:14 +0000 UTC" firstStartedPulling="2026-05-25 13:07:15.416853153 +0000 UTC m=+135.609147490" lastFinishedPulling="2026-05-25 13:07:18.154252512 +0000 UTC m=+138.346546857" observedRunningTime="2026-05-25 13:07:18.884418179 +0000 UTC m=+139.076712541" watchObservedRunningTime="2026-05-25 13:07:18.884429387 +0000 UTC m=+139.076723765"
May 25 13:07:19.870017 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:19.869978    2580 generic.go:358] "Generic (PLEG): container finished" podID="693562c8-a5e4-4462-92b9-8aca51cb1d27" containerID="b6a3bb75b483cbc46e5e8d39a6cc36acf62033a1aaebb75eda31ddafac8b0ff1" exitCode=0
May 25 13:07:19.870463 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:19.870070    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"693562c8-a5e4-4462-92b9-8aca51cb1d27","Type":"ContainerDied","Data":"b6a3bb75b483cbc46e5e8d39a6cc36acf62033a1aaebb75eda31ddafac8b0ff1"}
May 25 13:07:19.871633 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:19.871615    2580 generic.go:358] "Generic (PLEG): container finished" podID="7355e93e-5dee-448a-a74a-2f54c9d5e943" containerID="2350e96d63b4ce7d7e02cd72cb86ebc212fd68bfee7d1c047f333251f1060dd8" exitCode=0
May 25 13:07:19.871719 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:19.871698    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-j2jrt" event={"ID":"7355e93e-5dee-448a-a74a-2f54c9d5e943","Type":"ContainerDied","Data":"2350e96d63b4ce7d7e02cd72cb86ebc212fd68bfee7d1c047f333251f1060dd8"}
May 25 13:07:20.877039 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:20.877005    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-j2jrt" event={"ID":"7355e93e-5dee-448a-a74a-2f54c9d5e943","Type":"ContainerStarted","Data":"b23bd9521de38779fa0191b0a3d20f6c32fdb2e874b4f1f5e64cae9d60077d27"}
May 25 13:07:20.877039 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:20.877040    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-j2jrt" event={"ID":"7355e93e-5dee-448a-a74a-2f54c9d5e943","Type":"ContainerStarted","Data":"e392e3fd2ab39d70d3cd3ddb118b1c65b73228e3d4d9c6df7343b1d5f840d23d"}
May 25 13:07:20.899383 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:20.899312    2580 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-j2jrt" podStartSLOduration=3.74090691 podStartE2EDuration="4.899295324s" podCreationTimestamp="2026-05-25 13:07:16 +0000 UTC" firstStartedPulling="2026-05-25 13:07:18.111057582 +0000 UTC m=+138.303351920" lastFinishedPulling="2026-05-25 13:07:19.269445991 +0000 UTC m=+139.461740334" observedRunningTime="2026-05-25 13:07:20.896898145 +0000 UTC m=+141.089192507" watchObservedRunningTime="2026-05-25 13:07:20.899295324 +0000 UTC m=+141.091589686"
May 25 13:07:21.492823 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:21.492789    2580 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-665c9c8584-jlbqb"]
May 25 13:07:21.497104 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:21.497070    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-665c9c8584-jlbqb"
May 25 13:07:21.499205 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:21.499186    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
May 25 13:07:21.499303 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:21.499200    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"default-dockercfg-lw5zf\""
May 25 13:07:21.509107 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:21.509079    2580 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-665c9c8584-jlbqb"]
May 25 13:07:21.513684 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:21.513658    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a094f284-76a0-46e2-97ad-f6ee7d1910d5-monitoring-plugin-cert\") pod \"monitoring-plugin-665c9c8584-jlbqb\" (UID: \"a094f284-76a0-46e2-97ad-f6ee7d1910d5\") " pod="openshift-monitoring/monitoring-plugin-665c9c8584-jlbqb"
May 25 13:07:21.614726 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:21.614686    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a094f284-76a0-46e2-97ad-f6ee7d1910d5-monitoring-plugin-cert\") pod \"monitoring-plugin-665c9c8584-jlbqb\" (UID: \"a094f284-76a0-46e2-97ad-f6ee7d1910d5\") " pod="openshift-monitoring/monitoring-plugin-665c9c8584-jlbqb"
May 25 13:07:21.617240 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:21.617214    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a094f284-76a0-46e2-97ad-f6ee7d1910d5-monitoring-plugin-cert\") pod \"monitoring-plugin-665c9c8584-jlbqb\" (UID: \"a094f284-76a0-46e2-97ad-f6ee7d1910d5\") " pod="openshift-monitoring/monitoring-plugin-665c9c8584-jlbqb"
May 25 13:07:21.805990 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:21.805902    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-665c9c8584-jlbqb"
May 25 13:07:21.884189 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:21.884147    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"693562c8-a5e4-4462-92b9-8aca51cb1d27","Type":"ContainerStarted","Data":"4a42f35ba9db495bb10048b5e202bf492b75b0712c9ce815cf56d2e973399e50"}
May 25 13:07:21.884189 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:21.884190    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"693562c8-a5e4-4462-92b9-8aca51cb1d27","Type":"ContainerStarted","Data":"6cea575213b2373e861c1915eaa3fa9311f40b8f6200701aee91b2582a5abd6a"}
May 25 13:07:21.884627 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:21.884202    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"693562c8-a5e4-4462-92b9-8aca51cb1d27","Type":"ContainerStarted","Data":"bd25337aa78177007fec752b4a73210745cc2bb02976bce5af003b7ae18dafe4"}
May 25 13:07:21.884627 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:21.884213    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"693562c8-a5e4-4462-92b9-8aca51cb1d27","Type":"ContainerStarted","Data":"3574df43d0dea5fcfc6fdae97f9089de7c74f43da849bb42c81e8bb124a4c877"}
May 25 13:07:21.884627 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:21.884227    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"693562c8-a5e4-4462-92b9-8aca51cb1d27","Type":"ContainerStarted","Data":"ae6632d4f817fcc2f8fbdab4746cb3a148dc6898fef3d8b73016077f550fec90"}
May 25 13:07:21.946501 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:21.945902    2580 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-665c9c8584-jlbqb"]
May 25 13:07:21.950200 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:07:21.950171    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda094f284_76a0_46e2_97ad_f6ee7d1910d5.slice/crio-d62c00dfc6f6ab71836072c5989d78ce9cb192849c0c784b722e565d18feaef8 WatchSource:0}: Error finding container d62c00dfc6f6ab71836072c5989d78ce9cb192849c0c784b722e565d18feaef8: Status 404 returned error can't find the container with id d62c00dfc6f6ab71836072c5989d78ce9cb192849c0c784b722e565d18feaef8
May 25 13:07:22.892150 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:22.892111    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"693562c8-a5e4-4462-92b9-8aca51cb1d27","Type":"ContainerStarted","Data":"4639338fc8224f556ff1574102f6e62e00f088057a491ba37db319e5fa6a1b19"}
May 25 13:07:22.895469 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:22.895442    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-665c9c8584-jlbqb" event={"ID":"a094f284-76a0-46e2-97ad-f6ee7d1910d5","Type":"ContainerStarted","Data":"d62c00dfc6f6ab71836072c5989d78ce9cb192849c0c784b722e565d18feaef8"}
May 25 13:07:22.934529 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:22.934472    2580 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.091684228 podStartE2EDuration="5.934455762s" podCreationTimestamp="2026-05-25 13:07:17 +0000 UTC" firstStartedPulling="2026-05-25 13:07:18.2937161 +0000 UTC m=+138.486010438" lastFinishedPulling="2026-05-25 13:07:22.136487629 +0000 UTC m=+142.328781972" observedRunningTime="2026-05-25 13:07:22.928619005 +0000 UTC m=+143.120913382" watchObservedRunningTime="2026-05-25 13:07:22.934455762 +0000 UTC m=+143.126750122"
May 25 13:07:22.937408 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:22.937346    2580 patch_prober.go:28] interesting pod/image-registry-769864757b-hs9k5 container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
May 25 13:07:22.937560 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:22.937434    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-769864757b-hs9k5" podUID="f21589b0-1b80-46bb-92e8-1252201ed8f3" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 13:07:23.047896 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:23.047865    2580 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console/console-74f45df5f9-q5cfk"]
May 25 13:07:23.052392 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:23.052344    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74f45df5f9-q5cfk"
May 25 13:07:23.060017 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:23.059593    2580 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-74f45df5f9-q5cfk"]
May 25 13:07:23.131056 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:23.131008    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3f62ecc5-82b8-4f02-ab8e-6ddb496456a0-oauth-serving-cert\") pod \"console-74f45df5f9-q5cfk\" (UID: \"3f62ecc5-82b8-4f02-ab8e-6ddb496456a0\") " pod="openshift-console/console-74f45df5f9-q5cfk"
May 25 13:07:23.131254 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:23.131077    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3f62ecc5-82b8-4f02-ab8e-6ddb496456a0-service-ca\") pod \"console-74f45df5f9-q5cfk\" (UID: \"3f62ecc5-82b8-4f02-ab8e-6ddb496456a0\") " pod="openshift-console/console-74f45df5f9-q5cfk"
May 25 13:07:23.131254 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:23.131111    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3f62ecc5-82b8-4f02-ab8e-6ddb496456a0-trusted-ca-bundle\") pod \"console-74f45df5f9-q5cfk\" (UID: \"3f62ecc5-82b8-4f02-ab8e-6ddb496456a0\") " pod="openshift-console/console-74f45df5f9-q5cfk"
May 25 13:07:23.131254 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:23.131168    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3f62ecc5-82b8-4f02-ab8e-6ddb496456a0-console-config\") pod \"console-74f45df5f9-q5cfk\" (UID: \"3f62ecc5-82b8-4f02-ab8e-6ddb496456a0\") " pod="openshift-console/console-74f45df5f9-q5cfk"
May 25 13:07:23.131467 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:23.131288    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldmss\" (UniqueName: \"kubernetes.io/projected/3f62ecc5-82b8-4f02-ab8e-6ddb496456a0-kube-api-access-ldmss\") pod \"console-74f45df5f9-q5cfk\" (UID: \"3f62ecc5-82b8-4f02-ab8e-6ddb496456a0\") " pod="openshift-console/console-74f45df5f9-q5cfk"
May 25 13:07:23.131467 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:23.131332    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3f62ecc5-82b8-4f02-ab8e-6ddb496456a0-console-oauth-config\") pod \"console-74f45df5f9-q5cfk\" (UID: \"3f62ecc5-82b8-4f02-ab8e-6ddb496456a0\") " pod="openshift-console/console-74f45df5f9-q5cfk"
May 25 13:07:23.131467 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:23.131405    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3f62ecc5-82b8-4f02-ab8e-6ddb496456a0-console-serving-cert\") pod \"console-74f45df5f9-q5cfk\" (UID: \"3f62ecc5-82b8-4f02-ab8e-6ddb496456a0\") " pod="openshift-console/console-74f45df5f9-q5cfk"
May 25 13:07:23.234235 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:23.232749    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3f62ecc5-82b8-4f02-ab8e-6ddb496456a0-console-config\") pod \"console-74f45df5f9-q5cfk\" (UID: \"3f62ecc5-82b8-4f02-ab8e-6ddb496456a0\") " pod="openshift-console/console-74f45df5f9-q5cfk"
May 25 13:07:23.234235 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:23.232846    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-ldmss\" (UniqueName: \"kubernetes.io/projected/3f62ecc5-82b8-4f02-ab8e-6ddb496456a0-kube-api-access-ldmss\") pod \"console-74f45df5f9-q5cfk\" (UID: \"3f62ecc5-82b8-4f02-ab8e-6ddb496456a0\") " pod="openshift-console/console-74f45df5f9-q5cfk"
May 25 13:07:23.234235 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:23.232876    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3f62ecc5-82b8-4f02-ab8e-6ddb496456a0-console-oauth-config\") pod \"console-74f45df5f9-q5cfk\" (UID: \"3f62ecc5-82b8-4f02-ab8e-6ddb496456a0\") " pod="openshift-console/console-74f45df5f9-q5cfk"
May 25 13:07:23.234235 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:23.232927    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3f62ecc5-82b8-4f02-ab8e-6ddb496456a0-console-serving-cert\") pod \"console-74f45df5f9-q5cfk\" (UID: \"3f62ecc5-82b8-4f02-ab8e-6ddb496456a0\") " pod="openshift-console/console-74f45df5f9-q5cfk"
May 25 13:07:23.234235 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:23.232989    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3f62ecc5-82b8-4f02-ab8e-6ddb496456a0-oauth-serving-cert\") pod \"console-74f45df5f9-q5cfk\" (UID: \"3f62ecc5-82b8-4f02-ab8e-6ddb496456a0\") " pod="openshift-console/console-74f45df5f9-q5cfk"
May 25 13:07:23.234235 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:23.233042    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3f62ecc5-82b8-4f02-ab8e-6ddb496456a0-service-ca\") pod \"console-74f45df5f9-q5cfk\" (UID: \"3f62ecc5-82b8-4f02-ab8e-6ddb496456a0\") " pod="openshift-console/console-74f45df5f9-q5cfk"
May 25 13:07:23.234235 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:23.233072    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3f62ecc5-82b8-4f02-ab8e-6ddb496456a0-trusted-ca-bundle\") pod \"console-74f45df5f9-q5cfk\" (UID: \"3f62ecc5-82b8-4f02-ab8e-6ddb496456a0\") " pod="openshift-console/console-74f45df5f9-q5cfk"
May 25 13:07:23.234235 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:23.233498    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3f62ecc5-82b8-4f02-ab8e-6ddb496456a0-console-config\") pod \"console-74f45df5f9-q5cfk\" (UID: \"3f62ecc5-82b8-4f02-ab8e-6ddb496456a0\") " pod="openshift-console/console-74f45df5f9-q5cfk"
May 25 13:07:23.234235 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:23.233880    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3f62ecc5-82b8-4f02-ab8e-6ddb496456a0-trusted-ca-bundle\") pod \"console-74f45df5f9-q5cfk\" (UID: \"3f62ecc5-82b8-4f02-ab8e-6ddb496456a0\") " pod="openshift-console/console-74f45df5f9-q5cfk"
May 25 13:07:23.234817 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:23.234457    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3f62ecc5-82b8-4f02-ab8e-6ddb496456a0-oauth-serving-cert\") pod \"console-74f45df5f9-q5cfk\" (UID: \"3f62ecc5-82b8-4f02-ab8e-6ddb496456a0\") " pod="openshift-console/console-74f45df5f9-q5cfk"
May 25 13:07:23.234817 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:23.234478    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3f62ecc5-82b8-4f02-ab8e-6ddb496456a0-service-ca\") pod \"console-74f45df5f9-q5cfk\" (UID: \"3f62ecc5-82b8-4f02-ab8e-6ddb496456a0\") " pod="openshift-console/console-74f45df5f9-q5cfk"
May 25 13:07:23.236590 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:23.236559    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3f62ecc5-82b8-4f02-ab8e-6ddb496456a0-console-serving-cert\") pod \"console-74f45df5f9-q5cfk\" (UID: \"3f62ecc5-82b8-4f02-ab8e-6ddb496456a0\") " pod="openshift-console/console-74f45df5f9-q5cfk"
May 25 13:07:23.236854 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:23.236834    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3f62ecc5-82b8-4f02-ab8e-6ddb496456a0-console-oauth-config\") pod \"console-74f45df5f9-q5cfk\" (UID: \"3f62ecc5-82b8-4f02-ab8e-6ddb496456a0\") " pod="openshift-console/console-74f45df5f9-q5cfk"
May 25 13:07:23.245336 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:23.245309    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldmss\" (UniqueName: \"kubernetes.io/projected/3f62ecc5-82b8-4f02-ab8e-6ddb496456a0-kube-api-access-ldmss\") pod \"console-74f45df5f9-q5cfk\" (UID: \"3f62ecc5-82b8-4f02-ab8e-6ddb496456a0\") " pod="openshift-console/console-74f45df5f9-q5cfk"
May 25 13:07:23.365152 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:23.365109    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74f45df5f9-q5cfk"
May 25 13:07:23.498529 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:23.498454    2580 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-74f45df5f9-q5cfk"]
May 25 13:07:23.501504 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:07:23.501477    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f62ecc5_82b8_4f02_ab8e_6ddb496456a0.slice/crio-ad8976dc638b09c02e0c690dcb7ed722aa7332b3fe42cf9d1ce5180d3fdc3c42 WatchSource:0}: Error finding container ad8976dc638b09c02e0c690dcb7ed722aa7332b3fe42cf9d1ce5180d3fdc3c42: Status 404 returned error can't find the container with id ad8976dc638b09c02e0c690dcb7ed722aa7332b3fe42cf9d1ce5180d3fdc3c42
May 25 13:07:23.899786 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:23.899750    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-665c9c8584-jlbqb" event={"ID":"a094f284-76a0-46e2-97ad-f6ee7d1910d5","Type":"ContainerStarted","Data":"6e92ae7542796c3c5c42fec74cbd136c727e864fd61d506a5d19bcf51aae4e66"}
May 25 13:07:23.900242 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:23.899902    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-665c9c8584-jlbqb"
May 25 13:07:23.901236 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:23.901208    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74f45df5f9-q5cfk" event={"ID":"3f62ecc5-82b8-4f02-ab8e-6ddb496456a0","Type":"ContainerStarted","Data":"38ba31e251ea0ecd5463c64830500f8da50015dc47d9a3ef20c239a7e369218f"}
May 25 13:07:23.901339 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:23.901243    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74f45df5f9-q5cfk" event={"ID":"3f62ecc5-82b8-4f02-ab8e-6ddb496456a0","Type":"ContainerStarted","Data":"ad8976dc638b09c02e0c690dcb7ed722aa7332b3fe42cf9d1ce5180d3fdc3c42"}
May 25 13:07:23.905019 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:23.905001    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-665c9c8584-jlbqb"
May 25 13:07:23.914067 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:23.914019    2580 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-665c9c8584-jlbqb" podStartSLOduration=1.6045082229999998 podStartE2EDuration="2.914005334s" podCreationTimestamp="2026-05-25 13:07:21 +0000 UTC" firstStartedPulling="2026-05-25 13:07:21.953115909 +0000 UTC m=+142.145410252" lastFinishedPulling="2026-05-25 13:07:23.262613021 +0000 UTC m=+143.454907363" observedRunningTime="2026-05-25 13:07:23.912916656 +0000 UTC m=+144.105211018" watchObservedRunningTime="2026-05-25 13:07:23.914005334 +0000 UTC m=+144.106299695"
May 25 13:07:23.946197 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:23.946140    2580 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console/console-74f45df5f9-q5cfk" podStartSLOduration=0.946120445 podStartE2EDuration="946.120445ms" podCreationTimestamp="2026-05-25 13:07:23 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 13:07:23.944822395 +0000 UTC m=+144.137116758" watchObservedRunningTime="2026-05-25 13:07:23.946120445 +0000 UTC m=+144.138414805"
May 25 13:07:25.283641 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:25.283602    2580 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5895d9d8f5-rwfmg"
May 25 13:07:25.284032 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:25.283649    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5895d9d8f5-rwfmg"
May 25 13:07:25.285008 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:25.284985    2580 patch_prober.go:28] interesting pod/console-5895d9d8f5-rwfmg container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.134.0.16:8443/health\": dial tcp 10.134.0.16:8443: connect: connection refused" start-of-body=
May 25 13:07:25.285103 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:25.285028    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-console/console-5895d9d8f5-rwfmg" podUID="2f2c2a43-8334-476b-91ad-49475b5fb7b0" containerName="console" probeResult="failure" output="Get \"https://10.134.0.16:8443/health\": dial tcp 10.134.0.16:8443: connect: connection refused"
May 25 13:07:27.947793 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:27.947720    2580 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-image-registry/image-registry-769864757b-hs9k5" podUID="f21589b0-1b80-46bb-92e8-1252201ed8f3" containerName="registry" containerID="cri-o://739aaf59fcd334cb67940a2e8ce48c66d6cdc823e495ea846aa5843e210cb265" gracePeriod=30
May 25 13:07:28.196483 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:28.196461    2580 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-769864757b-hs9k5"
May 25 13:07:28.274507 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:28.274427    2580 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f21589b0-1b80-46bb-92e8-1252201ed8f3-registry-tls\") pod \"f21589b0-1b80-46bb-92e8-1252201ed8f3\" (UID: \"f21589b0-1b80-46bb-92e8-1252201ed8f3\") "
May 25 13:07:28.274507 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:28.274467    2580 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f21589b0-1b80-46bb-92e8-1252201ed8f3-trusted-ca\") pod \"f21589b0-1b80-46bb-92e8-1252201ed8f3\" (UID: \"f21589b0-1b80-46bb-92e8-1252201ed8f3\") "
May 25 13:07:28.274507 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:28.274499    2580 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f21589b0-1b80-46bb-92e8-1252201ed8f3-image-registry-private-configuration\") pod \"f21589b0-1b80-46bb-92e8-1252201ed8f3\" (UID: \"f21589b0-1b80-46bb-92e8-1252201ed8f3\") "
May 25 13:07:28.274752 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:28.274528    2580 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f21589b0-1b80-46bb-92e8-1252201ed8f3-installation-pull-secrets\") pod \"f21589b0-1b80-46bb-92e8-1252201ed8f3\" (UID: \"f21589b0-1b80-46bb-92e8-1252201ed8f3\") "
May 25 13:07:28.274752 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:28.274554    2580 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f21589b0-1b80-46bb-92e8-1252201ed8f3-bound-sa-token\") pod \"f21589b0-1b80-46bb-92e8-1252201ed8f3\" (UID: \"f21589b0-1b80-46bb-92e8-1252201ed8f3\") "
May 25 13:07:28.274752 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:28.274592    2580 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vwjmn\" (UniqueName: \"kubernetes.io/projected/f21589b0-1b80-46bb-92e8-1252201ed8f3-kube-api-access-vwjmn\") pod \"f21589b0-1b80-46bb-92e8-1252201ed8f3\" (UID: \"f21589b0-1b80-46bb-92e8-1252201ed8f3\") "
May 25 13:07:28.274752 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:28.274640    2580 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f21589b0-1b80-46bb-92e8-1252201ed8f3-ca-trust-extracted\") pod \"f21589b0-1b80-46bb-92e8-1252201ed8f3\" (UID: \"f21589b0-1b80-46bb-92e8-1252201ed8f3\") "
May 25 13:07:28.274752 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:28.274671    2580 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f21589b0-1b80-46bb-92e8-1252201ed8f3-registry-certificates\") pod \"f21589b0-1b80-46bb-92e8-1252201ed8f3\" (UID: \"f21589b0-1b80-46bb-92e8-1252201ed8f3\") "
May 25 13:07:28.274998 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:28.274898    2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f21589b0-1b80-46bb-92e8-1252201ed8f3-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "f21589b0-1b80-46bb-92e8-1252201ed8f3" (UID: "f21589b0-1b80-46bb-92e8-1252201ed8f3"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 13:07:28.275496 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:28.275455    2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f21589b0-1b80-46bb-92e8-1252201ed8f3-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "f21589b0-1b80-46bb-92e8-1252201ed8f3" (UID: "f21589b0-1b80-46bb-92e8-1252201ed8f3"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 13:07:28.276995 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:28.276969    2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f21589b0-1b80-46bb-92e8-1252201ed8f3-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "f21589b0-1b80-46bb-92e8-1252201ed8f3" (UID: "f21589b0-1b80-46bb-92e8-1252201ed8f3"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 13:07:28.277085 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:28.277029    2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f21589b0-1b80-46bb-92e8-1252201ed8f3-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "f21589b0-1b80-46bb-92e8-1252201ed8f3" (UID: "f21589b0-1b80-46bb-92e8-1252201ed8f3"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 13:07:28.277306 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:28.277278    2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f21589b0-1b80-46bb-92e8-1252201ed8f3-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "f21589b0-1b80-46bb-92e8-1252201ed8f3" (UID: "f21589b0-1b80-46bb-92e8-1252201ed8f3"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 13:07:28.277390 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:28.277283    2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f21589b0-1b80-46bb-92e8-1252201ed8f3-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "f21589b0-1b80-46bb-92e8-1252201ed8f3" (UID: "f21589b0-1b80-46bb-92e8-1252201ed8f3"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 13:07:28.277643 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:28.277623    2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f21589b0-1b80-46bb-92e8-1252201ed8f3-kube-api-access-vwjmn" (OuterVolumeSpecName: "kube-api-access-vwjmn") pod "f21589b0-1b80-46bb-92e8-1252201ed8f3" (UID: "f21589b0-1b80-46bb-92e8-1252201ed8f3"). InnerVolumeSpecName "kube-api-access-vwjmn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 13:07:28.283546 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:28.283524    2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f21589b0-1b80-46bb-92e8-1252201ed8f3-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "f21589b0-1b80-46bb-92e8-1252201ed8f3" (UID: "f21589b0-1b80-46bb-92e8-1252201ed8f3"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 13:07:28.375525 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:28.375456    2580 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f21589b0-1b80-46bb-92e8-1252201ed8f3-installation-pull-secrets\") on node \"ip-10-0-135-103.ec2.internal\" DevicePath \"\""
May 25 13:07:28.375525 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:28.375489    2580 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f21589b0-1b80-46bb-92e8-1252201ed8f3-bound-sa-token\") on node \"ip-10-0-135-103.ec2.internal\" DevicePath \"\""
May 25 13:07:28.375525 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:28.375505    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vwjmn\" (UniqueName: \"kubernetes.io/projected/f21589b0-1b80-46bb-92e8-1252201ed8f3-kube-api-access-vwjmn\") on node \"ip-10-0-135-103.ec2.internal\" DevicePath \"\""
May 25 13:07:28.375525 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:28.375520    2580 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f21589b0-1b80-46bb-92e8-1252201ed8f3-ca-trust-extracted\") on node \"ip-10-0-135-103.ec2.internal\" DevicePath \"\""
May 25 13:07:28.375806 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:28.375537    2580 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f21589b0-1b80-46bb-92e8-1252201ed8f3-registry-certificates\") on node \"ip-10-0-135-103.ec2.internal\" DevicePath \"\""
May 25 13:07:28.375806 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:28.375551    2580 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f21589b0-1b80-46bb-92e8-1252201ed8f3-registry-tls\") on node \"ip-10-0-135-103.ec2.internal\" DevicePath \"\""
May 25 13:07:28.375806 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:28.375564    2580 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f21589b0-1b80-46bb-92e8-1252201ed8f3-trusted-ca\") on node \"ip-10-0-135-103.ec2.internal\" DevicePath \"\""
May 25 13:07:28.375806 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:28.375576    2580 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f21589b0-1b80-46bb-92e8-1252201ed8f3-image-registry-private-configuration\") on node \"ip-10-0-135-103.ec2.internal\" DevicePath \"\""
May 25 13:07:28.915675 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:28.915635    2580 generic.go:358] "Generic (PLEG): container finished" podID="f21589b0-1b80-46bb-92e8-1252201ed8f3" containerID="739aaf59fcd334cb67940a2e8ce48c66d6cdc823e495ea846aa5843e210cb265" exitCode=0
May 25 13:07:28.915840 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:28.915693    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-769864757b-hs9k5" event={"ID":"f21589b0-1b80-46bb-92e8-1252201ed8f3","Type":"ContainerDied","Data":"739aaf59fcd334cb67940a2e8ce48c66d6cdc823e495ea846aa5843e210cb265"}
May 25 13:07:28.915840 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:28.915722    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-769864757b-hs9k5" event={"ID":"f21589b0-1b80-46bb-92e8-1252201ed8f3","Type":"ContainerDied","Data":"cae00a9e52499e55d11dcdcc34dc25c372a7d81946179536e3a85fe902e9e37d"}
May 25 13:07:28.915840 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:28.915739    2580 scope.go:117] "RemoveContainer" containerID="739aaf59fcd334cb67940a2e8ce48c66d6cdc823e495ea846aa5843e210cb265"
May 25 13:07:28.915840 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:28.915698    2580 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-769864757b-hs9k5"
May 25 13:07:28.925163 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:28.925146    2580 scope.go:117] "RemoveContainer" containerID="739aaf59fcd334cb67940a2e8ce48c66d6cdc823e495ea846aa5843e210cb265"
May 25 13:07:28.925436 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:07:28.925411    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"739aaf59fcd334cb67940a2e8ce48c66d6cdc823e495ea846aa5843e210cb265\": container with ID starting with 739aaf59fcd334cb67940a2e8ce48c66d6cdc823e495ea846aa5843e210cb265 not found: ID does not exist" containerID="739aaf59fcd334cb67940a2e8ce48c66d6cdc823e495ea846aa5843e210cb265"
May 25 13:07:28.925477 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:28.925446    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"739aaf59fcd334cb67940a2e8ce48c66d6cdc823e495ea846aa5843e210cb265"} err="failed to get container status \"739aaf59fcd334cb67940a2e8ce48c66d6cdc823e495ea846aa5843e210cb265\": rpc error: code = NotFound desc = could not find container \"739aaf59fcd334cb67940a2e8ce48c66d6cdc823e495ea846aa5843e210cb265\": container with ID starting with 739aaf59fcd334cb67940a2e8ce48c66d6cdc823e495ea846aa5843e210cb265 not found: ID does not exist"
May 25 13:07:28.932647 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:28.932617    2580 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-769864757b-hs9k5"]
May 25 13:07:28.936400 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:28.936359    2580 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-769864757b-hs9k5"]
May 25 13:07:30.377574 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:30.377533    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f21589b0-1b80-46bb-92e8-1252201ed8f3" path="/var/lib/kubelet/pods/f21589b0-1b80-46bb-92e8-1252201ed8f3/volumes"
May 25 13:07:33.365684 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:33.365650    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-74f45df5f9-q5cfk"
May 25 13:07:33.366120 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:33.365703    2580 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-74f45df5f9-q5cfk"
May 25 13:07:33.367066 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:33.367042    2580 patch_prober.go:28] interesting pod/console-74f45df5f9-q5cfk container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.134.0.19:8443/health\": dial tcp 10.134.0.19:8443: connect: connection refused" start-of-body=
May 25 13:07:33.367161 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:33.367086    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-console/console-74f45df5f9-q5cfk" podUID="3f62ecc5-82b8-4f02-ab8e-6ddb496456a0" containerName="console" probeResult="failure" output="Get \"https://10.134.0.19:8443/health\": dial tcp 10.134.0.19:8443: connect: connection refused"
May 25 13:07:35.283440 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:35.283407    2580 patch_prober.go:28] interesting pod/console-5895d9d8f5-rwfmg container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.134.0.16:8443/health\": dial tcp 10.134.0.16:8443: connect: connection refused" start-of-body=
May 25 13:07:35.283814 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:35.283460    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-console/console-5895d9d8f5-rwfmg" podUID="2f2c2a43-8334-476b-91ad-49475b5fb7b0" containerName="console" probeResult="failure" output="Get \"https://10.134.0.16:8443/health\": dial tcp 10.134.0.16:8443: connect: connection refused"
May 25 13:07:37.269442 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:07:37.269360    2580 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-ltl6z" podUID="c04c0bd2-e6c2-42a0-949b-8f818987dd6b"
May 25 13:07:37.283524 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:07:37.283493    2580 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-ingress-canary/ingress-canary-8gtmh" podUID="2e9a6623-9ccb-484c-b669-38caefdf2976"
May 25 13:07:37.940466 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:37.940438    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-8gtmh"
May 25 13:07:37.940651 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:37.940441    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-ltl6z"
May 25 13:07:41.951549 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:41.951511    2580 generic.go:358] "Generic (PLEG): container finished" podID="1a680b18-bede-464c-a3ba-c5eb503d8166" containerID="e727e99294174dd1f1688b61ae6133d3ef09762850e5ea9dee333b3e51029337" exitCode=255
May 25 13:07:41.951908 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:41.951575    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6f7545f5cb-h8cph" event={"ID":"1a680b18-bede-464c-a3ba-c5eb503d8166","Type":"ContainerDied","Data":"e727e99294174dd1f1688b61ae6133d3ef09762850e5ea9dee333b3e51029337"}
May 25 13:07:41.957572 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:41.957552    2580 scope.go:117] "RemoveContainer" containerID="e727e99294174dd1f1688b61ae6133d3ef09762850e5ea9dee333b3e51029337"
May 25 13:07:42.188298 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:42.188258    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2e9a6623-9ccb-484c-b669-38caefdf2976-cert\") pod \"ingress-canary-8gtmh\" (UID: \"2e9a6623-9ccb-484c-b669-38caefdf2976\") " pod="openshift-ingress-canary/ingress-canary-8gtmh"
May 25 13:07:42.188482 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:42.188314    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c04c0bd2-e6c2-42a0-949b-8f818987dd6b-metrics-tls\") pod \"dns-default-ltl6z\" (UID: \"c04c0bd2-e6c2-42a0-949b-8f818987dd6b\") " pod="openshift-dns/dns-default-ltl6z"
May 25 13:07:42.190830 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:42.190796    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c04c0bd2-e6c2-42a0-949b-8f818987dd6b-metrics-tls\") pod \"dns-default-ltl6z\" (UID: \"c04c0bd2-e6c2-42a0-949b-8f818987dd6b\") " pod="openshift-dns/dns-default-ltl6z"
May 25 13:07:42.190830 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:42.190815    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2e9a6623-9ccb-484c-b669-38caefdf2976-cert\") pod \"ingress-canary-8gtmh\" (UID: \"2e9a6623-9ccb-484c-b669-38caefdf2976\") " pod="openshift-ingress-canary/ingress-canary-8gtmh"
May 25 13:07:42.443093 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:42.443063    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-zhtj8\""
May 25 13:07:42.443727 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:42.443711    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-9wl44\""
May 25 13:07:42.451492 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:42.451471    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-8gtmh"
May 25 13:07:42.451584 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:42.451567    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-ltl6z"
May 25 13:07:42.587979 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:42.587948    2580 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-ltl6z"]
May 25 13:07:42.590759 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:07:42.590725    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc04c0bd2_e6c2_42a0_949b_8f818987dd6b.slice/crio-5b79fe6f43ef8cc4b7fd58b46fc70b4221a56448fac225deae6f84e185108c8c WatchSource:0}: Error finding container 5b79fe6f43ef8cc4b7fd58b46fc70b4221a56448fac225deae6f84e185108c8c: Status 404 returned error can't find the container with id 5b79fe6f43ef8cc4b7fd58b46fc70b4221a56448fac225deae6f84e185108c8c
May 25 13:07:42.608684 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:42.608653    2580 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-8gtmh"]
May 25 13:07:42.611585 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:07:42.611561    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e9a6623_9ccb_484c_b669_38caefdf2976.slice/crio-9212632175be077b55b086523330beb9402db652188f587b753991555e444f8d WatchSource:0}: Error finding container 9212632175be077b55b086523330beb9402db652188f587b753991555e444f8d: Status 404 returned error can't find the container with id 9212632175be077b55b086523330beb9402db652188f587b753991555e444f8d
May 25 13:07:42.956308 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:42.956272    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-8gtmh" event={"ID":"2e9a6623-9ccb-484c-b669-38caefdf2976","Type":"ContainerStarted","Data":"9212632175be077b55b086523330beb9402db652188f587b753991555e444f8d"}
May 25 13:07:42.957183 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:42.957160    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-ltl6z" event={"ID":"c04c0bd2-e6c2-42a0-949b-8f818987dd6b","Type":"ContainerStarted","Data":"5b79fe6f43ef8cc4b7fd58b46fc70b4221a56448fac225deae6f84e185108c8c"}
May 25 13:07:42.958478 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:42.958458    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6f7545f5cb-h8cph" event={"ID":"1a680b18-bede-464c-a3ba-c5eb503d8166","Type":"ContainerStarted","Data":"3247acc30b04dc38b355358fe6ee157e99a9dcb000b57daebd6d0eabf390d77a"}
May 25 13:07:43.366474 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:43.366438    2580 patch_prober.go:28] interesting pod/console-74f45df5f9-q5cfk container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.134.0.19:8443/health\": dial tcp 10.134.0.19:8443: connect: connection refused" start-of-body=
May 25 13:07:43.366691 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:43.366496    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-console/console-74f45df5f9-q5cfk" podUID="3f62ecc5-82b8-4f02-ab8e-6ddb496456a0" containerName="console" probeResult="failure" output="Get \"https://10.134.0.19:8443/health\": dial tcp 10.134.0.19:8443: connect: connection refused"
May 25 13:07:44.965468 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:44.965436    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-8gtmh" event={"ID":"2e9a6623-9ccb-484c-b669-38caefdf2976","Type":"ContainerStarted","Data":"72f60c46ca883bd92dd1737032c7915e69a4760b901b1b85fe24888f8a9fe179"}
May 25 13:07:44.967017 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:44.966992    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-ltl6z" event={"ID":"c04c0bd2-e6c2-42a0-949b-8f818987dd6b","Type":"ContainerStarted","Data":"872fc3feb342aa91e47f8fb58ac0efcd46061719822984276bb817213075df01"}
May 25 13:07:44.967120 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:44.967028    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-ltl6z" event={"ID":"c04c0bd2-e6c2-42a0-949b-8f818987dd6b","Type":"ContainerStarted","Data":"cc013bd4d459e58534d528bb96dd113f619a813ba3dec1143f80392d4352a27f"}
May 25 13:07:44.967120 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:44.967095    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-ltl6z"
May 25 13:07:44.981653 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:44.981616    2580 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-8gtmh" podStartSLOduration=129.089619247 podStartE2EDuration="2m10.981604342s" podCreationTimestamp="2026-05-25 13:05:34 +0000 UTC" firstStartedPulling="2026-05-25 13:07:42.613421911 +0000 UTC m=+162.805716248" lastFinishedPulling="2026-05-25 13:07:44.505407002 +0000 UTC m=+164.697701343" observedRunningTime="2026-05-25 13:07:44.979698854 +0000 UTC m=+165.171993210" watchObservedRunningTime="2026-05-25 13:07:44.981604342 +0000 UTC m=+165.173898701"
May 25 13:07:44.997238 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:44.997019    2580 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-dns/dns-default-ltl6z" podStartSLOduration=129.088245663 podStartE2EDuration="2m10.997003386s" podCreationTimestamp="2026-05-25 13:05:34 +0000 UTC" firstStartedPulling="2026-05-25 13:07:42.593332629 +0000 UTC m=+162.785626967" lastFinishedPulling="2026-05-25 13:07:44.502090346 +0000 UTC m=+164.694384690" observedRunningTime="2026-05-25 13:07:44.996401506 +0000 UTC m=+165.188695868" watchObservedRunningTime="2026-05-25 13:07:44.997003386 +0000 UTC m=+165.189297746"
May 25 13:07:45.287497 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:45.287420    2580 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5895d9d8f5-rwfmg"
May 25 13:07:45.291492 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:45.291461    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5895d9d8f5-rwfmg"
May 25 13:07:53.370022 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:53.369989    2580 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-74f45df5f9-q5cfk"
May 25 13:07:53.374386 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:53.374337    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-74f45df5f9-q5cfk"
May 25 13:07:53.437618 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:53.437575    2580 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5895d9d8f5-rwfmg"]
May 25 13:07:54.971757 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:54.971730    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-ltl6z"
May 25 13:07:54.996871 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:54.996834    2580 generic.go:358] "Generic (PLEG): container finished" podID="cf32b96d-4371-4b92-89db-ff71111c977a" containerID="291643ef9573c23e5a69b92bf62c4ba4e8a36d7b36c1e5d6d9183c4e804df02a" exitCode=0
May 25 13:07:54.997059 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:54.996893    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-78bdfb888c-tnkbv" event={"ID":"cf32b96d-4371-4b92-89db-ff71111c977a","Type":"ContainerDied","Data":"291643ef9573c23e5a69b92bf62c4ba4e8a36d7b36c1e5d6d9183c4e804df02a"}
May 25 13:07:54.997232 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:54.997217    2580 scope.go:117] "RemoveContainer" containerID="291643ef9573c23e5a69b92bf62c4ba4e8a36d7b36c1e5d6d9183c4e804df02a"
May 25 13:07:56.001534 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:07:56.001494    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-78bdfb888c-tnkbv" event={"ID":"cf32b96d-4371-4b92-89db-ff71111c977a","Type":"ContainerStarted","Data":"241d63b04b9462a25e51195fda909f59e6bd3a47797aebccec120c2bb78b85ff"}
May 25 13:08:10.045283 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:10.045245    2580 generic.go:358] "Generic (PLEG): container finished" podID="518b0e48-f2ac-4401-9212-834cbad021f9" containerID="2dc9c3ce06a1288a1f477fd417610634cfea5f0623e5b50edc785fb992e97a1a" exitCode=0
May 25 13:08:10.045735 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:10.045294    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-rh7nv" event={"ID":"518b0e48-f2ac-4401-9212-834cbad021f9","Type":"ContainerDied","Data":"2dc9c3ce06a1288a1f477fd417610634cfea5f0623e5b50edc785fb992e97a1a"}
May 25 13:08:10.045735 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:10.045624    2580 scope.go:117] "RemoveContainer" containerID="2dc9c3ce06a1288a1f477fd417610634cfea5f0623e5b50edc785fb992e97a1a"
May 25 13:08:11.049892 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:11.049860    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-rh7nv" event={"ID":"518b0e48-f2ac-4401-9212-834cbad021f9","Type":"ContainerStarted","Data":"ede4a937347ff2d44ad32bc025a64d333199f8e4672e2b71257bd065abd6e257"}
May 25 13:08:18.459908 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:18.459857    2580 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-console/console-5895d9d8f5-rwfmg" podUID="2f2c2a43-8334-476b-91ad-49475b5fb7b0" containerName="console" containerID="cri-o://aced0f880262508ef83c488494b70f2d79e3f342f20a29c3c91cf807c89169a6" gracePeriod=15
May 25 13:08:18.701720 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:18.701699    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-5895d9d8f5-rwfmg_2f2c2a43-8334-476b-91ad-49475b5fb7b0/console/0.log"
May 25 13:08:18.701838 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:18.701759    2580 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5895d9d8f5-rwfmg"
May 25 13:08:18.795297 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:18.795211    2580 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2f2c2a43-8334-476b-91ad-49475b5fb7b0-oauth-serving-cert\") pod \"2f2c2a43-8334-476b-91ad-49475b5fb7b0\" (UID: \"2f2c2a43-8334-476b-91ad-49475b5fb7b0\") "
May 25 13:08:18.795297 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:18.795248    2580 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2f2c2a43-8334-476b-91ad-49475b5fb7b0-trusted-ca-bundle\") pod \"2f2c2a43-8334-476b-91ad-49475b5fb7b0\" (UID: \"2f2c2a43-8334-476b-91ad-49475b5fb7b0\") "
May 25 13:08:18.795297 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:18.795273    2580 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5qk9k\" (UniqueName: \"kubernetes.io/projected/2f2c2a43-8334-476b-91ad-49475b5fb7b0-kube-api-access-5qk9k\") pod \"2f2c2a43-8334-476b-91ad-49475b5fb7b0\" (UID: \"2f2c2a43-8334-476b-91ad-49475b5fb7b0\") "
May 25 13:08:18.795591 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:18.795298    2580 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2f2c2a43-8334-476b-91ad-49475b5fb7b0-console-serving-cert\") pod \"2f2c2a43-8334-476b-91ad-49475b5fb7b0\" (UID: \"2f2c2a43-8334-476b-91ad-49475b5fb7b0\") "
May 25 13:08:18.795591 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:18.795326    2580 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2f2c2a43-8334-476b-91ad-49475b5fb7b0-console-oauth-config\") pod \"2f2c2a43-8334-476b-91ad-49475b5fb7b0\" (UID: \"2f2c2a43-8334-476b-91ad-49475b5fb7b0\") "
May 25 13:08:18.795591 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:18.795362    2580 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2f2c2a43-8334-476b-91ad-49475b5fb7b0-console-config\") pod \"2f2c2a43-8334-476b-91ad-49475b5fb7b0\" (UID: \"2f2c2a43-8334-476b-91ad-49475b5fb7b0\") "
May 25 13:08:18.795591 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:18.795488    2580 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2f2c2a43-8334-476b-91ad-49475b5fb7b0-service-ca\") pod \"2f2c2a43-8334-476b-91ad-49475b5fb7b0\" (UID: \"2f2c2a43-8334-476b-91ad-49475b5fb7b0\") "
May 25 13:08:18.795796 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:18.795671    2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f2c2a43-8334-476b-91ad-49475b5fb7b0-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "2f2c2a43-8334-476b-91ad-49475b5fb7b0" (UID: "2f2c2a43-8334-476b-91ad-49475b5fb7b0"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 13:08:18.795849 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:18.795810    2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f2c2a43-8334-476b-91ad-49475b5fb7b0-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "2f2c2a43-8334-476b-91ad-49475b5fb7b0" (UID: "2f2c2a43-8334-476b-91ad-49475b5fb7b0"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 13:08:18.795997 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:18.795893    2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f2c2a43-8334-476b-91ad-49475b5fb7b0-service-ca" (OuterVolumeSpecName: "service-ca") pod "2f2c2a43-8334-476b-91ad-49475b5fb7b0" (UID: "2f2c2a43-8334-476b-91ad-49475b5fb7b0"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 13:08:18.796100 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:18.796033    2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f2c2a43-8334-476b-91ad-49475b5fb7b0-console-config" (OuterVolumeSpecName: "console-config") pod "2f2c2a43-8334-476b-91ad-49475b5fb7b0" (UID: "2f2c2a43-8334-476b-91ad-49475b5fb7b0"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 13:08:18.797678 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:18.797655    2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f2c2a43-8334-476b-91ad-49475b5fb7b0-kube-api-access-5qk9k" (OuterVolumeSpecName: "kube-api-access-5qk9k") pod "2f2c2a43-8334-476b-91ad-49475b5fb7b0" (UID: "2f2c2a43-8334-476b-91ad-49475b5fb7b0"). InnerVolumeSpecName "kube-api-access-5qk9k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 13:08:18.797762 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:18.797729    2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f2c2a43-8334-476b-91ad-49475b5fb7b0-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "2f2c2a43-8334-476b-91ad-49475b5fb7b0" (UID: "2f2c2a43-8334-476b-91ad-49475b5fb7b0"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 13:08:18.797803 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:18.797748    2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f2c2a43-8334-476b-91ad-49475b5fb7b0-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "2f2c2a43-8334-476b-91ad-49475b5fb7b0" (UID: "2f2c2a43-8334-476b-91ad-49475b5fb7b0"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 13:08:18.896255 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:18.896217    2580 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2f2c2a43-8334-476b-91ad-49475b5fb7b0-oauth-serving-cert\") on node \"ip-10-0-135-103.ec2.internal\" DevicePath \"\""
May 25 13:08:18.896255 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:18.896249    2580 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2f2c2a43-8334-476b-91ad-49475b5fb7b0-trusted-ca-bundle\") on node \"ip-10-0-135-103.ec2.internal\" DevicePath \"\""
May 25 13:08:18.896255 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:18.896259    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5qk9k\" (UniqueName: \"kubernetes.io/projected/2f2c2a43-8334-476b-91ad-49475b5fb7b0-kube-api-access-5qk9k\") on node \"ip-10-0-135-103.ec2.internal\" DevicePath \"\""
May 25 13:08:18.896255 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:18.896267    2580 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2f2c2a43-8334-476b-91ad-49475b5fb7b0-console-serving-cert\") on node \"ip-10-0-135-103.ec2.internal\" DevicePath \"\""
May 25 13:08:18.896531 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:18.896277    2580 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2f2c2a43-8334-476b-91ad-49475b5fb7b0-console-oauth-config\") on node \"ip-10-0-135-103.ec2.internal\" DevicePath \"\""
May 25 13:08:18.896531 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:18.896287    2580 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2f2c2a43-8334-476b-91ad-49475b5fb7b0-console-config\") on node \"ip-10-0-135-103.ec2.internal\" DevicePath \"\""
May 25 13:08:18.896531 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:18.896296    2580 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2f2c2a43-8334-476b-91ad-49475b5fb7b0-service-ca\") on node \"ip-10-0-135-103.ec2.internal\" DevicePath \"\""
May 25 13:08:19.073219 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:19.073138    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-5895d9d8f5-rwfmg_2f2c2a43-8334-476b-91ad-49475b5fb7b0/console/0.log"
May 25 13:08:19.073219 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:19.073182    2580 generic.go:358] "Generic (PLEG): container finished" podID="2f2c2a43-8334-476b-91ad-49475b5fb7b0" containerID="aced0f880262508ef83c488494b70f2d79e3f342f20a29c3c91cf807c89169a6" exitCode=2
May 25 13:08:19.073423 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:19.073248    2580 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5895d9d8f5-rwfmg"
May 25 13:08:19.073423 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:19.073267    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5895d9d8f5-rwfmg" event={"ID":"2f2c2a43-8334-476b-91ad-49475b5fb7b0","Type":"ContainerDied","Data":"aced0f880262508ef83c488494b70f2d79e3f342f20a29c3c91cf807c89169a6"}
May 25 13:08:19.073423 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:19.073309    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5895d9d8f5-rwfmg" event={"ID":"2f2c2a43-8334-476b-91ad-49475b5fb7b0","Type":"ContainerDied","Data":"33cd5e1db7c755c284ba469e9536937618b878bd90ce9c518d51427eb50de28a"}
May 25 13:08:19.073423 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:19.073326    2580 scope.go:117] "RemoveContainer" containerID="aced0f880262508ef83c488494b70f2d79e3f342f20a29c3c91cf807c89169a6"
May 25 13:08:19.086474 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:19.086453    2580 scope.go:117] "RemoveContainer" containerID="aced0f880262508ef83c488494b70f2d79e3f342f20a29c3c91cf807c89169a6"
May 25 13:08:19.086738 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:08:19.086713    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aced0f880262508ef83c488494b70f2d79e3f342f20a29c3c91cf807c89169a6\": container with ID starting with aced0f880262508ef83c488494b70f2d79e3f342f20a29c3c91cf807c89169a6 not found: ID does not exist" containerID="aced0f880262508ef83c488494b70f2d79e3f342f20a29c3c91cf807c89169a6"
May 25 13:08:19.086874 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:19.086743    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aced0f880262508ef83c488494b70f2d79e3f342f20a29c3c91cf807c89169a6"} err="failed to get container status \"aced0f880262508ef83c488494b70f2d79e3f342f20a29c3c91cf807c89169a6\": rpc error: code = NotFound desc = could not find container \"aced0f880262508ef83c488494b70f2d79e3f342f20a29c3c91cf807c89169a6\": container with ID starting with aced0f880262508ef83c488494b70f2d79e3f342f20a29c3c91cf807c89169a6 not found: ID does not exist"
May 25 13:08:19.095786 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:19.095765    2580 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5895d9d8f5-rwfmg"]
May 25 13:08:19.099870 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:19.099847    2580 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5895d9d8f5-rwfmg"]
May 25 13:08:20.377567 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:20.377534    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2f2c2a43-8334-476b-91ad-49475b5fb7b0" path="/var/lib/kubelet/pods/2f2c2a43-8334-476b-91ad-49475b5fb7b0/volumes"
May 25 13:08:36.950729 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:36.950697    2580 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
May 25 13:08:36.951427 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:36.951225    2580 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="693562c8-a5e4-4462-92b9-8aca51cb1d27" containerName="alertmanager" containerID="cri-o://ae6632d4f817fcc2f8fbdab4746cb3a148dc6898fef3d8b73016077f550fec90" gracePeriod=120
May 25 13:08:36.951427 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:36.951262    2580 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="693562c8-a5e4-4462-92b9-8aca51cb1d27" containerName="kube-rbac-proxy-metric" containerID="cri-o://4a42f35ba9db495bb10048b5e202bf492b75b0712c9ce815cf56d2e973399e50" gracePeriod=120
May 25 13:08:36.951427 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:36.951305    2580 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="693562c8-a5e4-4462-92b9-8aca51cb1d27" containerName="kube-rbac-proxy-web" containerID="cri-o://bd25337aa78177007fec752b4a73210745cc2bb02976bce5af003b7ae18dafe4" gracePeriod=120
May 25 13:08:36.951427 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:36.951326    2580 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="693562c8-a5e4-4462-92b9-8aca51cb1d27" containerName="kube-rbac-proxy" containerID="cri-o://6cea575213b2373e861c1915eaa3fa9311f40b8f6200701aee91b2582a5abd6a" gracePeriod=120
May 25 13:08:36.951427 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:36.951341    2580 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="693562c8-a5e4-4462-92b9-8aca51cb1d27" containerName="config-reloader" containerID="cri-o://3574df43d0dea5fcfc6fdae97f9089de7c74f43da849bb42c81e8bb124a4c877" gracePeriod=120
May 25 13:08:36.951427 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:36.951283    2580 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="693562c8-a5e4-4462-92b9-8aca51cb1d27" containerName="prom-label-proxy" containerID="cri-o://4639338fc8224f556ff1574102f6e62e00f088057a491ba37db319e5fa6a1b19" gracePeriod=120
May 25 13:08:37.132049 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:37.132017    2580 generic.go:358] "Generic (PLEG): container finished" podID="693562c8-a5e4-4462-92b9-8aca51cb1d27" containerID="4639338fc8224f556ff1574102f6e62e00f088057a491ba37db319e5fa6a1b19" exitCode=0
May 25 13:08:37.132049 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:37.132041    2580 generic.go:358] "Generic (PLEG): container finished" podID="693562c8-a5e4-4462-92b9-8aca51cb1d27" containerID="6cea575213b2373e861c1915eaa3fa9311f40b8f6200701aee91b2582a5abd6a" exitCode=0
May 25 13:08:37.132049 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:37.132047    2580 generic.go:358] "Generic (PLEG): container finished" podID="693562c8-a5e4-4462-92b9-8aca51cb1d27" containerID="3574df43d0dea5fcfc6fdae97f9089de7c74f43da849bb42c81e8bb124a4c877" exitCode=0
May 25 13:08:37.132049 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:37.132053    2580 generic.go:358] "Generic (PLEG): container finished" podID="693562c8-a5e4-4462-92b9-8aca51cb1d27" containerID="ae6632d4f817fcc2f8fbdab4746cb3a148dc6898fef3d8b73016077f550fec90" exitCode=0
May 25 13:08:37.132272 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:37.132087    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"693562c8-a5e4-4462-92b9-8aca51cb1d27","Type":"ContainerDied","Data":"4639338fc8224f556ff1574102f6e62e00f088057a491ba37db319e5fa6a1b19"}
May 25 13:08:37.132272 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:37.132130    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"693562c8-a5e4-4462-92b9-8aca51cb1d27","Type":"ContainerDied","Data":"6cea575213b2373e861c1915eaa3fa9311f40b8f6200701aee91b2582a5abd6a"}
May 25 13:08:37.132272 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:37.132148    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"693562c8-a5e4-4462-92b9-8aca51cb1d27","Type":"ContainerDied","Data":"3574df43d0dea5fcfc6fdae97f9089de7c74f43da849bb42c81e8bb124a4c877"}
May 25 13:08:37.132272 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:37.132162    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"693562c8-a5e4-4462-92b9-8aca51cb1d27","Type":"ContainerDied","Data":"ae6632d4f817fcc2f8fbdab4746cb3a148dc6898fef3d8b73016077f550fec90"}
May 25 13:08:38.138686 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:38.138647    2580 generic.go:358] "Generic (PLEG): container finished" podID="693562c8-a5e4-4462-92b9-8aca51cb1d27" containerID="4a42f35ba9db495bb10048b5e202bf492b75b0712c9ce815cf56d2e973399e50" exitCode=0
May 25 13:08:38.138686 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:38.138681    2580 generic.go:358] "Generic (PLEG): container finished" podID="693562c8-a5e4-4462-92b9-8aca51cb1d27" containerID="bd25337aa78177007fec752b4a73210745cc2bb02976bce5af003b7ae18dafe4" exitCode=0
May 25 13:08:38.139068 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:38.138719    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"693562c8-a5e4-4462-92b9-8aca51cb1d27","Type":"ContainerDied","Data":"4a42f35ba9db495bb10048b5e202bf492b75b0712c9ce815cf56d2e973399e50"}
May 25 13:08:38.139068 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:38.138760    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"693562c8-a5e4-4462-92b9-8aca51cb1d27","Type":"ContainerDied","Data":"bd25337aa78177007fec752b4a73210745cc2bb02976bce5af003b7ae18dafe4"}
May 25 13:08:38.191915 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:38.191891    2580 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
May 25 13:08:38.243456 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:38.243420    2580 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/693562c8-a5e4-4462-92b9-8aca51cb1d27-config-volume\") pod \"693562c8-a5e4-4462-92b9-8aca51cb1d27\" (UID: \"693562c8-a5e4-4462-92b9-8aca51cb1d27\") "
May 25 13:08:38.243456 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:38.243455    2580 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/693562c8-a5e4-4462-92b9-8aca51cb1d27-tls-assets\") pod \"693562c8-a5e4-4462-92b9-8aca51cb1d27\" (UID: \"693562c8-a5e4-4462-92b9-8aca51cb1d27\") "
May 25 13:08:38.243681 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:38.243484    2580 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/693562c8-a5e4-4462-92b9-8aca51cb1d27-alertmanager-trusted-ca-bundle\") pod \"693562c8-a5e4-4462-92b9-8aca51cb1d27\" (UID: \"693562c8-a5e4-4462-92b9-8aca51cb1d27\") "
May 25 13:08:38.243681 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:38.243522    2580 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/693562c8-a5e4-4462-92b9-8aca51cb1d27-web-config\") pod \"693562c8-a5e4-4462-92b9-8aca51cb1d27\" (UID: \"693562c8-a5e4-4462-92b9-8aca51cb1d27\") "
May 25 13:08:38.243681 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:38.243550    2580 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/693562c8-a5e4-4462-92b9-8aca51cb1d27-secret-alertmanager-kube-rbac-proxy-metric\") pod \"693562c8-a5e4-4462-92b9-8aca51cb1d27\" (UID: \"693562c8-a5e4-4462-92b9-8aca51cb1d27\") "
May 25 13:08:38.243830 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:38.243813    2580 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/693562c8-a5e4-4462-92b9-8aca51cb1d27-metrics-client-ca\") pod \"693562c8-a5e4-4462-92b9-8aca51cb1d27\" (UID: \"693562c8-a5e4-4462-92b9-8aca51cb1d27\") "
May 25 13:08:38.243884 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:38.243862    2580 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/693562c8-a5e4-4462-92b9-8aca51cb1d27-alertmanager-main-db\") pod \"693562c8-a5e4-4462-92b9-8aca51cb1d27\" (UID: \"693562c8-a5e4-4462-92b9-8aca51cb1d27\") "
May 25 13:08:38.243937 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:38.243892    2580 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/693562c8-a5e4-4462-92b9-8aca51cb1d27-secret-alertmanager-kube-rbac-proxy\") pod \"693562c8-a5e4-4462-92b9-8aca51cb1d27\" (UID: \"693562c8-a5e4-4462-92b9-8aca51cb1d27\") "
May 25 13:08:38.243992 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:38.243932    2580 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/693562c8-a5e4-4462-92b9-8aca51cb1d27-config-out\") pod \"693562c8-a5e4-4462-92b9-8aca51cb1d27\" (UID: \"693562c8-a5e4-4462-92b9-8aca51cb1d27\") "
May 25 13:08:38.243992 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:38.243938    2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/693562c8-a5e4-4462-92b9-8aca51cb1d27-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "693562c8-a5e4-4462-92b9-8aca51cb1d27" (UID: "693562c8-a5e4-4462-92b9-8aca51cb1d27"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 13:08:38.243992 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:38.243974    2580 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/693562c8-a5e4-4462-92b9-8aca51cb1d27-secret-alertmanager-kube-rbac-proxy-web\") pod \"693562c8-a5e4-4462-92b9-8aca51cb1d27\" (UID: \"693562c8-a5e4-4462-92b9-8aca51cb1d27\") "
May 25 13:08:38.244136 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:38.243998    2580 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/693562c8-a5e4-4462-92b9-8aca51cb1d27-cluster-tls-config\") pod \"693562c8-a5e4-4462-92b9-8aca51cb1d27\" (UID: \"693562c8-a5e4-4462-92b9-8aca51cb1d27\") "
May 25 13:08:38.244136 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:38.244038    2580 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/693562c8-a5e4-4462-92b9-8aca51cb1d27-secret-alertmanager-main-tls\") pod \"693562c8-a5e4-4462-92b9-8aca51cb1d27\" (UID: \"693562c8-a5e4-4462-92b9-8aca51cb1d27\") "
May 25 13:08:38.244136 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:38.244080    2580 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qhhht\" (UniqueName: \"kubernetes.io/projected/693562c8-a5e4-4462-92b9-8aca51cb1d27-kube-api-access-qhhht\") pod \"693562c8-a5e4-4462-92b9-8aca51cb1d27\" (UID: \"693562c8-a5e4-4462-92b9-8aca51cb1d27\") "
May 25 13:08:38.244320 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:38.244302    2580 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/693562c8-a5e4-4462-92b9-8aca51cb1d27-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-135-103.ec2.internal\" DevicePath \"\""
May 25 13:08:38.244587 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:38.244554    2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/693562c8-a5e4-4462-92b9-8aca51cb1d27-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "693562c8-a5e4-4462-92b9-8aca51cb1d27" (UID: "693562c8-a5e4-4462-92b9-8aca51cb1d27"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 13:08:38.244587 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:38.244570    2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/693562c8-a5e4-4462-92b9-8aca51cb1d27-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "693562c8-a5e4-4462-92b9-8aca51cb1d27" (UID: "693562c8-a5e4-4462-92b9-8aca51cb1d27"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 13:08:38.246772 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:38.246743    2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/693562c8-a5e4-4462-92b9-8aca51cb1d27-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "693562c8-a5e4-4462-92b9-8aca51cb1d27" (UID: "693562c8-a5e4-4462-92b9-8aca51cb1d27"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 13:08:38.246884 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:38.246787    2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/693562c8-a5e4-4462-92b9-8aca51cb1d27-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "693562c8-a5e4-4462-92b9-8aca51cb1d27" (UID: "693562c8-a5e4-4462-92b9-8aca51cb1d27"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 13:08:38.246884 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:38.246824    2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/693562c8-a5e4-4462-92b9-8aca51cb1d27-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "693562c8-a5e4-4462-92b9-8aca51cb1d27" (UID: "693562c8-a5e4-4462-92b9-8aca51cb1d27"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 13:08:38.248383 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:38.248331    2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/693562c8-a5e4-4462-92b9-8aca51cb1d27-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "693562c8-a5e4-4462-92b9-8aca51cb1d27" (UID: "693562c8-a5e4-4462-92b9-8aca51cb1d27"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 13:08:38.248487 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:38.248345    2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/693562c8-a5e4-4462-92b9-8aca51cb1d27-kube-api-access-qhhht" (OuterVolumeSpecName: "kube-api-access-qhhht") pod "693562c8-a5e4-4462-92b9-8aca51cb1d27" (UID: "693562c8-a5e4-4462-92b9-8aca51cb1d27"). InnerVolumeSpecName "kube-api-access-qhhht". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 13:08:38.248731 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:38.248701    2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/693562c8-a5e4-4462-92b9-8aca51cb1d27-config-volume" (OuterVolumeSpecName: "config-volume") pod "693562c8-a5e4-4462-92b9-8aca51cb1d27" (UID: "693562c8-a5e4-4462-92b9-8aca51cb1d27"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 13:08:38.249167 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:38.249139    2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/693562c8-a5e4-4462-92b9-8aca51cb1d27-config-out" (OuterVolumeSpecName: "config-out") pod "693562c8-a5e4-4462-92b9-8aca51cb1d27" (UID: "693562c8-a5e4-4462-92b9-8aca51cb1d27"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 13:08:38.249737 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:38.249708    2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/693562c8-a5e4-4462-92b9-8aca51cb1d27-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "693562c8-a5e4-4462-92b9-8aca51cb1d27" (UID: "693562c8-a5e4-4462-92b9-8aca51cb1d27"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 13:08:38.253143 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:38.253122    2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/693562c8-a5e4-4462-92b9-8aca51cb1d27-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "693562c8-a5e4-4462-92b9-8aca51cb1d27" (UID: "693562c8-a5e4-4462-92b9-8aca51cb1d27"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 13:08:38.259806 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:38.259781    2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/693562c8-a5e4-4462-92b9-8aca51cb1d27-web-config" (OuterVolumeSpecName: "web-config") pod "693562c8-a5e4-4462-92b9-8aca51cb1d27" (UID: "693562c8-a5e4-4462-92b9-8aca51cb1d27"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 13:08:38.345140 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:38.345107    2580 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/693562c8-a5e4-4462-92b9-8aca51cb1d27-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-135-103.ec2.internal\" DevicePath \"\""
May 25 13:08:38.345140 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:38.345141    2580 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/693562c8-a5e4-4462-92b9-8aca51cb1d27-cluster-tls-config\") on node \"ip-10-0-135-103.ec2.internal\" DevicePath \"\""
May 25 13:08:38.345326 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:38.345153    2580 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/693562c8-a5e4-4462-92b9-8aca51cb1d27-secret-alertmanager-main-tls\") on node \"ip-10-0-135-103.ec2.internal\" DevicePath \"\""
May 25 13:08:38.345326 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:38.345162    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qhhht\" (UniqueName: \"kubernetes.io/projected/693562c8-a5e4-4462-92b9-8aca51cb1d27-kube-api-access-qhhht\") on node \"ip-10-0-135-103.ec2.internal\" DevicePath \"\""
May 25 13:08:38.345326 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:38.345171    2580 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/693562c8-a5e4-4462-92b9-8aca51cb1d27-config-volume\") on node \"ip-10-0-135-103.ec2.internal\" DevicePath \"\""
May 25 13:08:38.345326 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:38.345179    2580 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/693562c8-a5e4-4462-92b9-8aca51cb1d27-tls-assets\") on node \"ip-10-0-135-103.ec2.internal\" DevicePath \"\""
May 25 13:08:38.345326 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:38.345187    2580 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/693562c8-a5e4-4462-92b9-8aca51cb1d27-web-config\") on node \"ip-10-0-135-103.ec2.internal\" DevicePath \"\""
May 25 13:08:38.345326 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:38.345195    2580 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/693562c8-a5e4-4462-92b9-8aca51cb1d27-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-135-103.ec2.internal\" DevicePath \"\""
May 25 13:08:38.345326 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:38.345204    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/693562c8-a5e4-4462-92b9-8aca51cb1d27-metrics-client-ca\") on node \"ip-10-0-135-103.ec2.internal\" DevicePath \"\""
May 25 13:08:38.345326 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:38.345212    2580 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/693562c8-a5e4-4462-92b9-8aca51cb1d27-alertmanager-main-db\") on node \"ip-10-0-135-103.ec2.internal\" DevicePath \"\""
May 25 13:08:38.345326 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:38.345221    2580 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/693562c8-a5e4-4462-92b9-8aca51cb1d27-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-135-103.ec2.internal\" DevicePath \"\""
May 25 13:08:38.345326 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:38.345229    2580 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/693562c8-a5e4-4462-92b9-8aca51cb1d27-config-out\") on node \"ip-10-0-135-103.ec2.internal\" DevicePath \"\""
May 25 13:08:39.144756 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:39.144718    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"693562c8-a5e4-4462-92b9-8aca51cb1d27","Type":"ContainerDied","Data":"b443199fde871746fb44b562ac9b53216f2abc4c8c11de4285e0ec21d2b6e007"}
May 25 13:08:39.145146 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:39.144772    2580 scope.go:117] "RemoveContainer" containerID="4639338fc8224f556ff1574102f6e62e00f088057a491ba37db319e5fa6a1b19"
May 25 13:08:39.145146 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:39.144801    2580 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
May 25 13:08:39.152145 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:39.152127    2580 scope.go:117] "RemoveContainer" containerID="4a42f35ba9db495bb10048b5e202bf492b75b0712c9ce815cf56d2e973399e50"
May 25 13:08:39.159523 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:39.159505    2580 scope.go:117] "RemoveContainer" containerID="6cea575213b2373e861c1915eaa3fa9311f40b8f6200701aee91b2582a5abd6a"
May 25 13:08:39.165992 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:39.165971    2580 scope.go:117] "RemoveContainer" containerID="bd25337aa78177007fec752b4a73210745cc2bb02976bce5af003b7ae18dafe4"
May 25 13:08:39.166179 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:39.166157    2580 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
May 25 13:08:39.172681 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:39.172653    2580 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
May 25 13:08:39.173390 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:39.173351    2580 scope.go:117] "RemoveContainer" containerID="3574df43d0dea5fcfc6fdae97f9089de7c74f43da849bb42c81e8bb124a4c877"
May 25 13:08:39.180010 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:39.179990    2580 scope.go:117] "RemoveContainer" containerID="ae6632d4f817fcc2f8fbdab4746cb3a148dc6898fef3d8b73016077f550fec90"
May 25 13:08:39.186269 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:39.186251    2580 scope.go:117] "RemoveContainer" containerID="b6a3bb75b483cbc46e5e8d39a6cc36acf62033a1aaebb75eda31ddafac8b0ff1"
May 25 13:08:39.200673 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:39.200648    2580 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
May 25 13:08:39.200989 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:39.200975    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f21589b0-1b80-46bb-92e8-1252201ed8f3" containerName="registry"
May 25 13:08:39.201032 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:39.200994    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f21589b0-1b80-46bb-92e8-1252201ed8f3" containerName="registry"
May 25 13:08:39.201032 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:39.201006    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="693562c8-a5e4-4462-92b9-8aca51cb1d27" containerName="kube-rbac-proxy"
May 25 13:08:39.201032 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:39.201014    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="693562c8-a5e4-4462-92b9-8aca51cb1d27" containerName="kube-rbac-proxy"
May 25 13:08:39.201032 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:39.201029    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="693562c8-a5e4-4462-92b9-8aca51cb1d27" containerName="kube-rbac-proxy-web"
May 25 13:08:39.201179 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:39.201038    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="693562c8-a5e4-4462-92b9-8aca51cb1d27" containerName="kube-rbac-proxy-web"
May 25 13:08:39.201179 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:39.201048    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="693562c8-a5e4-4462-92b9-8aca51cb1d27" containerName="kube-rbac-proxy-metric"
May 25 13:08:39.201179 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:39.201056    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="693562c8-a5e4-4462-92b9-8aca51cb1d27" containerName="kube-rbac-proxy-metric"
May 25 13:08:39.201179 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:39.201068    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="693562c8-a5e4-4462-92b9-8aca51cb1d27" containerName="config-reloader"
May 25 13:08:39.201179 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:39.201075    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="693562c8-a5e4-4462-92b9-8aca51cb1d27" containerName="config-reloader"
May 25 13:08:39.201179 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:39.201084    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="693562c8-a5e4-4462-92b9-8aca51cb1d27" containerName="prom-label-proxy"
May 25 13:08:39.201179 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:39.201091    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="693562c8-a5e4-4462-92b9-8aca51cb1d27" containerName="prom-label-proxy"
May 25 13:08:39.201179 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:39.201108    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f2c2a43-8334-476b-91ad-49475b5fb7b0" containerName="console"
May 25 13:08:39.201179 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:39.201115    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f2c2a43-8334-476b-91ad-49475b5fb7b0" containerName="console"
May 25 13:08:39.201179 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:39.201125    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="693562c8-a5e4-4462-92b9-8aca51cb1d27" containerName="init-config-reloader"
May 25 13:08:39.201179 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:39.201132    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="693562c8-a5e4-4462-92b9-8aca51cb1d27" containerName="init-config-reloader"
May 25 13:08:39.201179 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:39.201141    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="693562c8-a5e4-4462-92b9-8aca51cb1d27" containerName="alertmanager"
May 25 13:08:39.201179 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:39.201149    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="693562c8-a5e4-4462-92b9-8aca51cb1d27" containerName="alertmanager"
May 25 13:08:39.201542 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:39.201208    2580 memory_manager.go:356] "RemoveStaleState removing state" podUID="693562c8-a5e4-4462-92b9-8aca51cb1d27" containerName="kube-rbac-proxy-web"
May 25 13:08:39.201542 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:39.201220    2580 memory_manager.go:356] "RemoveStaleState removing state" podUID="693562c8-a5e4-4462-92b9-8aca51cb1d27" containerName="config-reloader"
May 25 13:08:39.201542 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:39.201231    2580 memory_manager.go:356] "RemoveStaleState removing state" podUID="693562c8-a5e4-4462-92b9-8aca51cb1d27" containerName="alertmanager"
May 25 13:08:39.201542 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:39.201241    2580 memory_manager.go:356] "RemoveStaleState removing state" podUID="693562c8-a5e4-4462-92b9-8aca51cb1d27" containerName="kube-rbac-proxy"
May 25 13:08:39.201542 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:39.201249    2580 memory_manager.go:356] "RemoveStaleState removing state" podUID="693562c8-a5e4-4462-92b9-8aca51cb1d27" containerName="prom-label-proxy"
May 25 13:08:39.201542 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:39.201257    2580 memory_manager.go:356] "RemoveStaleState removing state" podUID="f21589b0-1b80-46bb-92e8-1252201ed8f3" containerName="registry"
May 25 13:08:39.201542 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:39.201266    2580 memory_manager.go:356] "RemoveStaleState removing state" podUID="2f2c2a43-8334-476b-91ad-49475b5fb7b0" containerName="console"
May 25 13:08:39.201542 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:39.201275    2580 memory_manager.go:356] "RemoveStaleState removing state" podUID="693562c8-a5e4-4462-92b9-8aca51cb1d27" containerName="kube-rbac-proxy-metric"
May 25 13:08:39.205019 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:39.204999    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
May 25 13:08:39.207276 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:39.207254    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
May 25 13:08:39.207397 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:39.207266    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
May 25 13:08:39.207397 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:39.207284    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
May 25 13:08:39.207516 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:39.207396    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
May 25 13:08:39.207516 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:39.207356    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
May 25 13:08:39.207516 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:39.207486    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
May 25 13:08:39.207787 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:39.207768    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
May 25 13:08:39.207787 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:39.207783    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
May 25 13:08:39.207905 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:39.207860    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-q4l9n\""
May 25 13:08:39.214711 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:39.214692    2580 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
May 25 13:08:39.217551 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:39.217529    2580 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
May 25 13:08:39.252832 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:39.252800    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7734bf32-5077-42ae-ad48-28b644b05756-config-out\") pod \"alertmanager-main-0\" (UID: \"7734bf32-5077-42ae-ad48-28b644b05756\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 13:08:39.252832 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:39.252833    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7734bf32-5077-42ae-ad48-28b644b05756-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"7734bf32-5077-42ae-ad48-28b644b05756\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 13:08:39.253020 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:39.252862    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7734bf32-5077-42ae-ad48-28b644b05756-tls-assets\") pod \"alertmanager-main-0\" (UID: \"7734bf32-5077-42ae-ad48-28b644b05756\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 13:08:39.253020 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:39.252889    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/7734bf32-5077-42ae-ad48-28b644b05756-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"7734bf32-5077-42ae-ad48-28b644b05756\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 13:08:39.253020 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:39.252949    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7734bf32-5077-42ae-ad48-28b644b05756-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"7734bf32-5077-42ae-ad48-28b644b05756\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 13:08:39.253020 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:39.252976    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4gn2\" (UniqueName: \"kubernetes.io/projected/7734bf32-5077-42ae-ad48-28b644b05756-kube-api-access-j4gn2\") pod \"alertmanager-main-0\" (UID: \"7734bf32-5077-42ae-ad48-28b644b05756\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 13:08:39.253020 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:39.253011    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/7734bf32-5077-42ae-ad48-28b644b05756-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"7734bf32-5077-42ae-ad48-28b644b05756\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 13:08:39.253164 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:39.253041    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/7734bf32-5077-42ae-ad48-28b644b05756-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"7734bf32-5077-42ae-ad48-28b644b05756\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 13:08:39.253164 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:39.253068    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/7734bf32-5077-42ae-ad48-28b644b05756-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"7734bf32-5077-42ae-ad48-28b644b05756\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 13:08:39.253164 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:39.253087    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/7734bf32-5077-42ae-ad48-28b644b05756-config-volume\") pod \"alertmanager-main-0\" (UID: \"7734bf32-5077-42ae-ad48-28b644b05756\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 13:08:39.253164 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:39.253136    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7734bf32-5077-42ae-ad48-28b644b05756-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"7734bf32-5077-42ae-ad48-28b644b05756\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 13:08:39.253274 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:39.253164    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7734bf32-5077-42ae-ad48-28b644b05756-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"7734bf32-5077-42ae-ad48-28b644b05756\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 13:08:39.253274 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:39.253196    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7734bf32-5077-42ae-ad48-28b644b05756-web-config\") pod \"alertmanager-main-0\" (UID: \"7734bf32-5077-42ae-ad48-28b644b05756\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 13:08:39.354148 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:39.354097    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/7734bf32-5077-42ae-ad48-28b644b05756-config-volume\") pod \"alertmanager-main-0\" (UID: \"7734bf32-5077-42ae-ad48-28b644b05756\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 13:08:39.354339 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:39.354165    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7734bf32-5077-42ae-ad48-28b644b05756-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"7734bf32-5077-42ae-ad48-28b644b05756\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 13:08:39.354339 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:39.354202    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7734bf32-5077-42ae-ad48-28b644b05756-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"7734bf32-5077-42ae-ad48-28b644b05756\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 13:08:39.354339 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:39.354237    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7734bf32-5077-42ae-ad48-28b644b05756-web-config\") pod \"alertmanager-main-0\" (UID: \"7734bf32-5077-42ae-ad48-28b644b05756\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 13:08:39.354339 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:39.354274    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7734bf32-5077-42ae-ad48-28b644b05756-config-out\") pod \"alertmanager-main-0\" (UID: \"7734bf32-5077-42ae-ad48-28b644b05756\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 13:08:39.354339 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:39.354299    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7734bf32-5077-42ae-ad48-28b644b05756-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"7734bf32-5077-42ae-ad48-28b644b05756\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 13:08:39.354339 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:39.354327    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7734bf32-5077-42ae-ad48-28b644b05756-tls-assets\") pod \"alertmanager-main-0\" (UID: \"7734bf32-5077-42ae-ad48-28b644b05756\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 13:08:39.354644 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:39.354350    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/7734bf32-5077-42ae-ad48-28b644b05756-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"7734bf32-5077-42ae-ad48-28b644b05756\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 13:08:39.354644 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:39.354404    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7734bf32-5077-42ae-ad48-28b644b05756-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"7734bf32-5077-42ae-ad48-28b644b05756\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 13:08:39.354644 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:39.354427    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-j4gn2\" (UniqueName: \"kubernetes.io/projected/7734bf32-5077-42ae-ad48-28b644b05756-kube-api-access-j4gn2\") pod \"alertmanager-main-0\" (UID: \"7734bf32-5077-42ae-ad48-28b644b05756\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 13:08:39.354644 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:39.354476    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/7734bf32-5077-42ae-ad48-28b644b05756-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"7734bf32-5077-42ae-ad48-28b644b05756\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 13:08:39.354644 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:39.354504    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/7734bf32-5077-42ae-ad48-28b644b05756-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"7734bf32-5077-42ae-ad48-28b644b05756\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 13:08:39.354644 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:39.354537    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/7734bf32-5077-42ae-ad48-28b644b05756-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"7734bf32-5077-42ae-ad48-28b644b05756\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 13:08:39.355607 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:39.355545    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/7734bf32-5077-42ae-ad48-28b644b05756-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"7734bf32-5077-42ae-ad48-28b644b05756\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 13:08:39.356164 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:39.356111    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7734bf32-5077-42ae-ad48-28b644b05756-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"7734bf32-5077-42ae-ad48-28b644b05756\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 13:08:39.356284 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:39.356186    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7734bf32-5077-42ae-ad48-28b644b05756-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"7734bf32-5077-42ae-ad48-28b644b05756\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 13:08:39.357578 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:39.357535    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7734bf32-5077-42ae-ad48-28b644b05756-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"7734bf32-5077-42ae-ad48-28b644b05756\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 13:08:39.357753 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:39.357694    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/7734bf32-5077-42ae-ad48-28b644b05756-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"7734bf32-5077-42ae-ad48-28b644b05756\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 13:08:39.357753 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:39.357694    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7734bf32-5077-42ae-ad48-28b644b05756-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"7734bf32-5077-42ae-ad48-28b644b05756\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 13:08:39.357992 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:39.357972    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7734bf32-5077-42ae-ad48-28b644b05756-web-config\") pod \"alertmanager-main-0\" (UID: \"7734bf32-5077-42ae-ad48-28b644b05756\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 13:08:39.358359 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:39.358341    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/7734bf32-5077-42ae-ad48-28b644b05756-config-volume\") pod \"alertmanager-main-0\" (UID: \"7734bf32-5077-42ae-ad48-28b644b05756\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 13:08:39.358834 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:39.358811    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7734bf32-5077-42ae-ad48-28b644b05756-config-out\") pod \"alertmanager-main-0\" (UID: \"7734bf32-5077-42ae-ad48-28b644b05756\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 13:08:39.359073 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:39.359055    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/7734bf32-5077-42ae-ad48-28b644b05756-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"7734bf32-5077-42ae-ad48-28b644b05756\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 13:08:39.359230 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:39.359207    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/7734bf32-5077-42ae-ad48-28b644b05756-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"7734bf32-5077-42ae-ad48-28b644b05756\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 13:08:39.359230 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:39.359224    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7734bf32-5077-42ae-ad48-28b644b05756-tls-assets\") pod \"alertmanager-main-0\" (UID: \"7734bf32-5077-42ae-ad48-28b644b05756\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 13:08:39.363052 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:39.363031    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4gn2\" (UniqueName: \"kubernetes.io/projected/7734bf32-5077-42ae-ad48-28b644b05756-kube-api-access-j4gn2\") pod \"alertmanager-main-0\" (UID: \"7734bf32-5077-42ae-ad48-28b644b05756\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 13:08:39.515335 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:39.515301    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
May 25 13:08:39.652008 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:39.651628    2580 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
May 25 13:08:39.654668 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:08:39.654638    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7734bf32_5077_42ae_ad48_28b644b05756.slice/crio-6f2e64c908bd21a611dd5bf983b35a3728b687b4fb0f8d6720e9fe0b946c967a WatchSource:0}: Error finding container 6f2e64c908bd21a611dd5bf983b35a3728b687b4fb0f8d6720e9fe0b946c967a: Status 404 returned error can't find the container with id 6f2e64c908bd21a611dd5bf983b35a3728b687b4fb0f8d6720e9fe0b946c967a
May 25 13:08:40.149313 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:40.149275    2580 generic.go:358] "Generic (PLEG): container finished" podID="7734bf32-5077-42ae-ad48-28b644b05756" containerID="2d8fd241978a30efeb0c5bf5fec98aa7b184a34e2e9883a05cdb96ffe42138d0" exitCode=0
May 25 13:08:40.149702 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:40.149344    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7734bf32-5077-42ae-ad48-28b644b05756","Type":"ContainerDied","Data":"2d8fd241978a30efeb0c5bf5fec98aa7b184a34e2e9883a05cdb96ffe42138d0"}
May 25 13:08:40.149702 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:40.149401    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7734bf32-5077-42ae-ad48-28b644b05756","Type":"ContainerStarted","Data":"6f2e64c908bd21a611dd5bf983b35a3728b687b4fb0f8d6720e9fe0b946c967a"}
May 25 13:08:40.377946 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:40.377914    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="693562c8-a5e4-4462-92b9-8aca51cb1d27" path="/var/lib/kubelet/pods/693562c8-a5e4-4462-92b9-8aca51cb1d27/volumes"
May 25 13:08:40.993938 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:40.993884    2580 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-5dcdc9c674-ml8pd"]
May 25 13:08:40.996473 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:40.996449    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5dcdc9c674-ml8pd"
May 25 13:08:41.000685 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:41.000659    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
May 25 13:08:41.000685 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:41.000670    2580 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
May 25 13:08:41.001174 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:41.001152    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
May 25 13:08:41.001444 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:41.001426    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
May 25 13:08:41.001752 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:41.001733    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-h6s7n\""
May 25 13:08:41.002185 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:41.002008    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
May 25 13:08:41.006542 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:41.006522    2580 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
May 25 13:08:41.012850 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:41.012824    2580 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5dcdc9c674-ml8pd"]
May 25 13:08:41.068194 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:41.068158    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e6b50ea7-d401-4f36-8fd3-08c48aad8efc-metrics-client-ca\") pod \"telemeter-client-5dcdc9c674-ml8pd\" (UID: \"e6b50ea7-d401-4f36-8fd3-08c48aad8efc\") " pod="openshift-monitoring/telemeter-client-5dcdc9c674-ml8pd"
May 25 13:08:41.068362 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:41.068199    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/e6b50ea7-d401-4f36-8fd3-08c48aad8efc-federate-client-tls\") pod \"telemeter-client-5dcdc9c674-ml8pd\" (UID: \"e6b50ea7-d401-4f36-8fd3-08c48aad8efc\") " pod="openshift-monitoring/telemeter-client-5dcdc9c674-ml8pd"
May 25 13:08:41.068362 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:41.068237    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e6b50ea7-d401-4f36-8fd3-08c48aad8efc-serving-certs-ca-bundle\") pod \"telemeter-client-5dcdc9c674-ml8pd\" (UID: \"e6b50ea7-d401-4f36-8fd3-08c48aad8efc\") " pod="openshift-monitoring/telemeter-client-5dcdc9c674-ml8pd"
May 25 13:08:41.068362 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:41.068331    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e6b50ea7-d401-4f36-8fd3-08c48aad8efc-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5dcdc9c674-ml8pd\" (UID: \"e6b50ea7-d401-4f36-8fd3-08c48aad8efc\") " pod="openshift-monitoring/telemeter-client-5dcdc9c674-ml8pd"
May 25 13:08:41.068562 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:41.068390    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srdb8\" (UniqueName: \"kubernetes.io/projected/e6b50ea7-d401-4f36-8fd3-08c48aad8efc-kube-api-access-srdb8\") pod \"telemeter-client-5dcdc9c674-ml8pd\" (UID: \"e6b50ea7-d401-4f36-8fd3-08c48aad8efc\") " pod="openshift-monitoring/telemeter-client-5dcdc9c674-ml8pd"
May 25 13:08:41.068562 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:41.068451    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/e6b50ea7-d401-4f36-8fd3-08c48aad8efc-telemeter-client-tls\") pod \"telemeter-client-5dcdc9c674-ml8pd\" (UID: \"e6b50ea7-d401-4f36-8fd3-08c48aad8efc\") " pod="openshift-monitoring/telemeter-client-5dcdc9c674-ml8pd"
May 25 13:08:41.068562 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:41.068486    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e6b50ea7-d401-4f36-8fd3-08c48aad8efc-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5dcdc9c674-ml8pd\" (UID: \"e6b50ea7-d401-4f36-8fd3-08c48aad8efc\") " pod="openshift-monitoring/telemeter-client-5dcdc9c674-ml8pd"
May 25 13:08:41.068562 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:41.068528    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/e6b50ea7-d401-4f36-8fd3-08c48aad8efc-secret-telemeter-client\") pod \"telemeter-client-5dcdc9c674-ml8pd\" (UID: \"e6b50ea7-d401-4f36-8fd3-08c48aad8efc\") " pod="openshift-monitoring/telemeter-client-5dcdc9c674-ml8pd"
May 25 13:08:41.156607 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:41.156564    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7734bf32-5077-42ae-ad48-28b644b05756","Type":"ContainerStarted","Data":"dbce81d7818f64bac0355de04172669e62e191119b25efc365b89c93fa11313e"}
May 25 13:08:41.157010 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:41.156632    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7734bf32-5077-42ae-ad48-28b644b05756","Type":"ContainerStarted","Data":"fc06aee26a42ed8eec88933502ee4edc1abb744f319b2679d0ea4c152796cf1c"}
May 25 13:08:41.157010 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:41.156654    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7734bf32-5077-42ae-ad48-28b644b05756","Type":"ContainerStarted","Data":"ef9a5d44554a084cc34771ef2fbce762358db09c5e0647071960258907bc2491"}
May 25 13:08:41.157010 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:41.156670    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7734bf32-5077-42ae-ad48-28b644b05756","Type":"ContainerStarted","Data":"c3f9d7d064495bbe9c214296e85e9a073f01bf158cb666b1a05df36e6415cb8a"}
May 25 13:08:41.157010 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:41.156680    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7734bf32-5077-42ae-ad48-28b644b05756","Type":"ContainerStarted","Data":"8d8bcfc44dffdae85bc9a295e2f3bc5e9eea50cb50a465c5e3aab68341998c95"}
May 25 13:08:41.157010 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:41.156688    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7734bf32-5077-42ae-ad48-28b644b05756","Type":"ContainerStarted","Data":"d19173d38342a760c07bb1912ed797fa23dc58c89f920c44487108d1c3b14004"}
May 25 13:08:41.169305 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:41.169268    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e6b50ea7-d401-4f36-8fd3-08c48aad8efc-serving-certs-ca-bundle\") pod \"telemeter-client-5dcdc9c674-ml8pd\" (UID: \"e6b50ea7-d401-4f36-8fd3-08c48aad8efc\") " pod="openshift-monitoring/telemeter-client-5dcdc9c674-ml8pd"
May 25 13:08:41.169509 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:41.169338    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e6b50ea7-d401-4f36-8fd3-08c48aad8efc-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5dcdc9c674-ml8pd\" (UID: \"e6b50ea7-d401-4f36-8fd3-08c48aad8efc\") " pod="openshift-monitoring/telemeter-client-5dcdc9c674-ml8pd"
May 25 13:08:41.169509 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:41.169392    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-srdb8\" (UniqueName: \"kubernetes.io/projected/e6b50ea7-d401-4f36-8fd3-08c48aad8efc-kube-api-access-srdb8\") pod \"telemeter-client-5dcdc9c674-ml8pd\" (UID: \"e6b50ea7-d401-4f36-8fd3-08c48aad8efc\") " pod="openshift-monitoring/telemeter-client-5dcdc9c674-ml8pd"
May 25 13:08:41.169509 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:41.169447    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/e6b50ea7-d401-4f36-8fd3-08c48aad8efc-telemeter-client-tls\") pod \"telemeter-client-5dcdc9c674-ml8pd\" (UID: \"e6b50ea7-d401-4f36-8fd3-08c48aad8efc\") " pod="openshift-monitoring/telemeter-client-5dcdc9c674-ml8pd"
May 25 13:08:41.169509 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:41.169471    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e6b50ea7-d401-4f36-8fd3-08c48aad8efc-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5dcdc9c674-ml8pd\" (UID: \"e6b50ea7-d401-4f36-8fd3-08c48aad8efc\") " pod="openshift-monitoring/telemeter-client-5dcdc9c674-ml8pd"
May 25 13:08:41.169716 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:41.169508    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/e6b50ea7-d401-4f36-8fd3-08c48aad8efc-secret-telemeter-client\") pod \"telemeter-client-5dcdc9c674-ml8pd\" (UID: \"e6b50ea7-d401-4f36-8fd3-08c48aad8efc\") " pod="openshift-monitoring/telemeter-client-5dcdc9c674-ml8pd"
May 25 13:08:41.169716 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:41.169539    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e6b50ea7-d401-4f36-8fd3-08c48aad8efc-metrics-client-ca\") pod \"telemeter-client-5dcdc9c674-ml8pd\" (UID: \"e6b50ea7-d401-4f36-8fd3-08c48aad8efc\") " pod="openshift-monitoring/telemeter-client-5dcdc9c674-ml8pd"
May 25 13:08:41.169807 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:41.169717    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/e6b50ea7-d401-4f36-8fd3-08c48aad8efc-federate-client-tls\") pod \"telemeter-client-5dcdc9c674-ml8pd\" (UID: \"e6b50ea7-d401-4f36-8fd3-08c48aad8efc\") " pod="openshift-monitoring/telemeter-client-5dcdc9c674-ml8pd"
May 25 13:08:41.170289 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:41.170264    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e6b50ea7-d401-4f36-8fd3-08c48aad8efc-metrics-client-ca\") pod \"telemeter-client-5dcdc9c674-ml8pd\" (UID: \"e6b50ea7-d401-4f36-8fd3-08c48aad8efc\") " pod="openshift-monitoring/telemeter-client-5dcdc9c674-ml8pd"
May 25 13:08:41.170467 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:41.170305    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e6b50ea7-d401-4f36-8fd3-08c48aad8efc-serving-certs-ca-bundle\") pod \"telemeter-client-5dcdc9c674-ml8pd\" (UID: \"e6b50ea7-d401-4f36-8fd3-08c48aad8efc\") " pod="openshift-monitoring/telemeter-client-5dcdc9c674-ml8pd"
May 25 13:08:41.171243 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:41.171182    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e6b50ea7-d401-4f36-8fd3-08c48aad8efc-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5dcdc9c674-ml8pd\" (UID: \"e6b50ea7-d401-4f36-8fd3-08c48aad8efc\") " pod="openshift-monitoring/telemeter-client-5dcdc9c674-ml8pd"
May 25 13:08:41.173149 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:41.173120    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/e6b50ea7-d401-4f36-8fd3-08c48aad8efc-secret-telemeter-client\") pod \"telemeter-client-5dcdc9c674-ml8pd\" (UID: \"e6b50ea7-d401-4f36-8fd3-08c48aad8efc\") " pod="openshift-monitoring/telemeter-client-5dcdc9c674-ml8pd"
May 25 13:08:41.173642 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:41.173617    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e6b50ea7-d401-4f36-8fd3-08c48aad8efc-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5dcdc9c674-ml8pd\" (UID: \"e6b50ea7-d401-4f36-8fd3-08c48aad8efc\") " pod="openshift-monitoring/telemeter-client-5dcdc9c674-ml8pd"
May 25 13:08:41.173642 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:41.173635    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/e6b50ea7-d401-4f36-8fd3-08c48aad8efc-federate-client-tls\") pod \"telemeter-client-5dcdc9c674-ml8pd\" (UID: \"e6b50ea7-d401-4f36-8fd3-08c48aad8efc\") " pod="openshift-monitoring/telemeter-client-5dcdc9c674-ml8pd"
May 25 13:08:41.174120 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:41.174099    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/e6b50ea7-d401-4f36-8fd3-08c48aad8efc-telemeter-client-tls\") pod \"telemeter-client-5dcdc9c674-ml8pd\" (UID: \"e6b50ea7-d401-4f36-8fd3-08c48aad8efc\") " pod="openshift-monitoring/telemeter-client-5dcdc9c674-ml8pd"
May 25 13:08:41.178286 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:41.178266    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-srdb8\" (UniqueName: \"kubernetes.io/projected/e6b50ea7-d401-4f36-8fd3-08c48aad8efc-kube-api-access-srdb8\") pod \"telemeter-client-5dcdc9c674-ml8pd\" (UID: \"e6b50ea7-d401-4f36-8fd3-08c48aad8efc\") " pod="openshift-monitoring/telemeter-client-5dcdc9c674-ml8pd"
May 25 13:08:41.183122 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:41.183076    2580 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.183059843 podStartE2EDuration="2.183059843s" podCreationTimestamp="2026-05-25 13:08:39 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 13:08:41.182227429 +0000 UTC m=+221.374521790" watchObservedRunningTime="2026-05-25 13:08:41.183059843 +0000 UTC m=+221.375354205"
May 25 13:08:41.308618 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:41.308530    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5dcdc9c674-ml8pd"
May 25 13:08:41.462043 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:41.462010    2580 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5dcdc9c674-ml8pd"]
May 25 13:08:41.464981 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:08:41.464906    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6b50ea7_d401_4f36_8fd3_08c48aad8efc.slice/crio-20f2423219d90cf267630391148c8c9661982fc7a29b0435106c88e124dc504a WatchSource:0}: Error finding container 20f2423219d90cf267630391148c8c9661982fc7a29b0435106c88e124dc504a: Status 404 returned error can't find the container with id 20f2423219d90cf267630391148c8c9661982fc7a29b0435106c88e124dc504a
May 25 13:08:42.165948 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:42.165902    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5dcdc9c674-ml8pd" event={"ID":"e6b50ea7-d401-4f36-8fd3-08c48aad8efc","Type":"ContainerStarted","Data":"20f2423219d90cf267630391148c8c9661982fc7a29b0435106c88e124dc504a"}
May 25 13:08:43.170325 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:43.170299    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5dcdc9c674-ml8pd" event={"ID":"e6b50ea7-d401-4f36-8fd3-08c48aad8efc","Type":"ContainerStarted","Data":"ce88f677f406991dd5ca35fd2eb4214990db7a87a8135c4397f43617d28382fe"}
May 25 13:08:43.170610 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:43.170334    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5dcdc9c674-ml8pd" event={"ID":"e6b50ea7-d401-4f36-8fd3-08c48aad8efc","Type":"ContainerStarted","Data":"61d48f74932aac1683199158935c4cdd3dab9a3bb6dcdd71d256f8e545ea46ae"}
May 25 13:08:44.175311 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:44.175271    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5dcdc9c674-ml8pd" event={"ID":"e6b50ea7-d401-4f36-8fd3-08c48aad8efc","Type":"ContainerStarted","Data":"951dbe9dc1e98af3dcb971e9b54e03ba54ab6def3607ede334765df57fc5e40c"}
May 25 13:08:44.197329 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:44.197262    2580 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-5dcdc9c674-ml8pd" podStartSLOduration=2.6084411320000003 podStartE2EDuration="4.197247023s" podCreationTimestamp="2026-05-25 13:08:40 +0000 UTC" firstStartedPulling="2026-05-25 13:08:41.46701849 +0000 UTC m=+221.659312829" lastFinishedPulling="2026-05-25 13:08:43.055824379 +0000 UTC m=+223.248118720" observedRunningTime="2026-05-25 13:08:44.195528061 +0000 UTC m=+224.387822421" watchObservedRunningTime="2026-05-25 13:08:44.197247023 +0000 UTC m=+224.389541383"
May 25 13:08:44.813170 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:44.813138    2580 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console/console-5bdcffdd98-9v5kx"]
May 25 13:08:44.815089 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:44.815073    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5bdcffdd98-9v5kx"
May 25 13:08:44.825227 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:44.825200    2580 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5bdcffdd98-9v5kx"]
May 25 13:08:44.904442 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:44.904409    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d413aef7-390e-441a-a383-8196d1615d69-console-config\") pod \"console-5bdcffdd98-9v5kx\" (UID: \"d413aef7-390e-441a-a383-8196d1615d69\") " pod="openshift-console/console-5bdcffdd98-9v5kx"
May 25 13:08:44.904442 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:44.904442    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d413aef7-390e-441a-a383-8196d1615d69-oauth-serving-cert\") pod \"console-5bdcffdd98-9v5kx\" (UID: \"d413aef7-390e-441a-a383-8196d1615d69\") " pod="openshift-console/console-5bdcffdd98-9v5kx"
May 25 13:08:44.904643 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:44.904486    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d413aef7-390e-441a-a383-8196d1615d69-console-serving-cert\") pod \"console-5bdcffdd98-9v5kx\" (UID: \"d413aef7-390e-441a-a383-8196d1615d69\") " pod="openshift-console/console-5bdcffdd98-9v5kx"
May 25 13:08:44.904643 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:44.904512    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d413aef7-390e-441a-a383-8196d1615d69-service-ca\") pod \"console-5bdcffdd98-9v5kx\" (UID: \"d413aef7-390e-441a-a383-8196d1615d69\") " pod="openshift-console/console-5bdcffdd98-9v5kx"
May 25 13:08:44.904643 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:44.904537    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d413aef7-390e-441a-a383-8196d1615d69-trusted-ca-bundle\") pod \"console-5bdcffdd98-9v5kx\" (UID: \"d413aef7-390e-441a-a383-8196d1615d69\") " pod="openshift-console/console-5bdcffdd98-9v5kx"
May 25 13:08:44.904643 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:44.904600    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpjqb\" (UniqueName: \"kubernetes.io/projected/d413aef7-390e-441a-a383-8196d1615d69-kube-api-access-zpjqb\") pod \"console-5bdcffdd98-9v5kx\" (UID: \"d413aef7-390e-441a-a383-8196d1615d69\") " pod="openshift-console/console-5bdcffdd98-9v5kx"
May 25 13:08:44.904643 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:44.904626    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d413aef7-390e-441a-a383-8196d1615d69-console-oauth-config\") pod \"console-5bdcffdd98-9v5kx\" (UID: \"d413aef7-390e-441a-a383-8196d1615d69\") " pod="openshift-console/console-5bdcffdd98-9v5kx"
May 25 13:08:45.005790 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:45.005743    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-zpjqb\" (UniqueName: \"kubernetes.io/projected/d413aef7-390e-441a-a383-8196d1615d69-kube-api-access-zpjqb\") pod \"console-5bdcffdd98-9v5kx\" (UID: \"d413aef7-390e-441a-a383-8196d1615d69\") " pod="openshift-console/console-5bdcffdd98-9v5kx"
May 25 13:08:45.005989 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:45.005801    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d413aef7-390e-441a-a383-8196d1615d69-console-oauth-config\") pod \"console-5bdcffdd98-9v5kx\" (UID: \"d413aef7-390e-441a-a383-8196d1615d69\") " pod="openshift-console/console-5bdcffdd98-9v5kx"
May 25 13:08:45.005989 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:45.005831    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d413aef7-390e-441a-a383-8196d1615d69-console-config\") pod \"console-5bdcffdd98-9v5kx\" (UID: \"d413aef7-390e-441a-a383-8196d1615d69\") " pod="openshift-console/console-5bdcffdd98-9v5kx"
May 25 13:08:45.005989 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:45.005859    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d413aef7-390e-441a-a383-8196d1615d69-oauth-serving-cert\") pod \"console-5bdcffdd98-9v5kx\" (UID: \"d413aef7-390e-441a-a383-8196d1615d69\") " pod="openshift-console/console-5bdcffdd98-9v5kx"
May 25 13:08:45.005989 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:45.005900    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d413aef7-390e-441a-a383-8196d1615d69-console-serving-cert\") pod \"console-5bdcffdd98-9v5kx\" (UID: \"d413aef7-390e-441a-a383-8196d1615d69\") " pod="openshift-console/console-5bdcffdd98-9v5kx"
May 25 13:08:45.005989 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:45.005930    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d413aef7-390e-441a-a383-8196d1615d69-service-ca\") pod \"console-5bdcffdd98-9v5kx\" (UID: \"d413aef7-390e-441a-a383-8196d1615d69\") " pod="openshift-console/console-5bdcffdd98-9v5kx"
May 25 13:08:45.005989 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:45.005969    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d413aef7-390e-441a-a383-8196d1615d69-trusted-ca-bundle\") pod \"console-5bdcffdd98-9v5kx\" (UID: \"d413aef7-390e-441a-a383-8196d1615d69\") " pod="openshift-console/console-5bdcffdd98-9v5kx"
May 25 13:08:45.006757 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:45.006695    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d413aef7-390e-441a-a383-8196d1615d69-oauth-serving-cert\") pod \"console-5bdcffdd98-9v5kx\" (UID: \"d413aef7-390e-441a-a383-8196d1615d69\") " pod="openshift-console/console-5bdcffdd98-9v5kx"
May 25 13:08:45.007012 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:45.006948    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d413aef7-390e-441a-a383-8196d1615d69-service-ca\") pod \"console-5bdcffdd98-9v5kx\" (UID: \"d413aef7-390e-441a-a383-8196d1615d69\") " pod="openshift-console/console-5bdcffdd98-9v5kx"
May 25 13:08:45.007112 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:45.007038    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d413aef7-390e-441a-a383-8196d1615d69-trusted-ca-bundle\") pod \"console-5bdcffdd98-9v5kx\" (UID: \"d413aef7-390e-441a-a383-8196d1615d69\") " pod="openshift-console/console-5bdcffdd98-9v5kx"
May 25 13:08:45.007476 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:45.007453    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d413aef7-390e-441a-a383-8196d1615d69-console-config\") pod \"console-5bdcffdd98-9v5kx\" (UID: \"d413aef7-390e-441a-a383-8196d1615d69\") " pod="openshift-console/console-5bdcffdd98-9v5kx"
May 25 13:08:45.009334 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:45.009311    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d413aef7-390e-441a-a383-8196d1615d69-console-serving-cert\") pod \"console-5bdcffdd98-9v5kx\" (UID: \"d413aef7-390e-441a-a383-8196d1615d69\") " pod="openshift-console/console-5bdcffdd98-9v5kx"
May 25 13:08:45.009502 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:45.009481    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d413aef7-390e-441a-a383-8196d1615d69-console-oauth-config\") pod \"console-5bdcffdd98-9v5kx\" (UID: \"d413aef7-390e-441a-a383-8196d1615d69\") " pod="openshift-console/console-5bdcffdd98-9v5kx"
May 25 13:08:45.014352 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:45.014329    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpjqb\" (UniqueName: \"kubernetes.io/projected/d413aef7-390e-441a-a383-8196d1615d69-kube-api-access-zpjqb\") pod \"console-5bdcffdd98-9v5kx\" (UID: \"d413aef7-390e-441a-a383-8196d1615d69\") " pod="openshift-console/console-5bdcffdd98-9v5kx"
May 25 13:08:45.123645 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:45.123607    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5bdcffdd98-9v5kx"
May 25 13:08:45.269390 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:45.269326    2580 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5bdcffdd98-9v5kx"]
May 25 13:08:45.273115 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:08:45.273086    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd413aef7_390e_441a_a383_8196d1615d69.slice/crio-71435ccff8c05b8449bbbc7db0a06366ba3f713b5ecaf8b519eca2c6eaa934c2 WatchSource:0}: Error finding container 71435ccff8c05b8449bbbc7db0a06366ba3f713b5ecaf8b519eca2c6eaa934c2: Status 404 returned error can't find the container with id 71435ccff8c05b8449bbbc7db0a06366ba3f713b5ecaf8b519eca2c6eaa934c2
May 25 13:08:46.188172 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:46.188135    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5bdcffdd98-9v5kx" event={"ID":"d413aef7-390e-441a-a383-8196d1615d69","Type":"ContainerStarted","Data":"488f7a764adbfa30527607fa404c2ea033ea40383271b493e9eb5e6c91314b9e"}
May 25 13:08:46.188172 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:46.188173    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5bdcffdd98-9v5kx" event={"ID":"d413aef7-390e-441a-a383-8196d1615d69","Type":"ContainerStarted","Data":"71435ccff8c05b8449bbbc7db0a06366ba3f713b5ecaf8b519eca2c6eaa934c2"}
May 25 13:08:46.205123 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:46.205069    2580 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console/console-5bdcffdd98-9v5kx" podStartSLOduration=2.205050697 podStartE2EDuration="2.205050697s" podCreationTimestamp="2026-05-25 13:08:44 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 13:08:46.204200499 +0000 UTC m=+226.396494859" watchObservedRunningTime="2026-05-25 13:08:46.205050697 +0000 UTC m=+226.397345058"
May 25 13:08:55.123907 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:55.123863    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5bdcffdd98-9v5kx"
May 25 13:08:55.124407 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:55.123932    2580 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5bdcffdd98-9v5kx"
May 25 13:08:55.128739 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:55.128711    2580 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5bdcffdd98-9v5kx"
May 25 13:08:55.217192 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:55.217163    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5bdcffdd98-9v5kx"
May 25 13:08:55.262413 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:08:55.262361    2580 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-74f45df5f9-q5cfk"]
May 25 13:09:20.282897 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:09:20.282783    2580 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-console/console-74f45df5f9-q5cfk" podUID="3f62ecc5-82b8-4f02-ab8e-6ddb496456a0" containerName="console" containerID="cri-o://38ba31e251ea0ecd5463c64830500f8da50015dc47d9a3ef20c239a7e369218f" gracePeriod=15
May 25 13:09:20.537889 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:09:20.537822    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-74f45df5f9-q5cfk_3f62ecc5-82b8-4f02-ab8e-6ddb496456a0/console/0.log"
May 25 13:09:20.538008 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:09:20.537889    2580 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74f45df5f9-q5cfk"
May 25 13:09:20.589685 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:09:20.589646    2580 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ldmss\" (UniqueName: \"kubernetes.io/projected/3f62ecc5-82b8-4f02-ab8e-6ddb496456a0-kube-api-access-ldmss\") pod \"3f62ecc5-82b8-4f02-ab8e-6ddb496456a0\" (UID: \"3f62ecc5-82b8-4f02-ab8e-6ddb496456a0\") "
May 25 13:09:20.589879 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:09:20.589701    2580 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3f62ecc5-82b8-4f02-ab8e-6ddb496456a0-console-oauth-config\") pod \"3f62ecc5-82b8-4f02-ab8e-6ddb496456a0\" (UID: \"3f62ecc5-82b8-4f02-ab8e-6ddb496456a0\") "
May 25 13:09:20.589879 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:09:20.589747    2580 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3f62ecc5-82b8-4f02-ab8e-6ddb496456a0-service-ca\") pod \"3f62ecc5-82b8-4f02-ab8e-6ddb496456a0\" (UID: \"3f62ecc5-82b8-4f02-ab8e-6ddb496456a0\") "
May 25 13:09:20.589879 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:09:20.589847    2580 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3f62ecc5-82b8-4f02-ab8e-6ddb496456a0-trusted-ca-bundle\") pod \"3f62ecc5-82b8-4f02-ab8e-6ddb496456a0\" (UID: \"3f62ecc5-82b8-4f02-ab8e-6ddb496456a0\") "
May 25 13:09:20.590045 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:09:20.589902    2580 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3f62ecc5-82b8-4f02-ab8e-6ddb496456a0-oauth-serving-cert\") pod \"3f62ecc5-82b8-4f02-ab8e-6ddb496456a0\" (UID: \"3f62ecc5-82b8-4f02-ab8e-6ddb496456a0\") "
May 25 13:09:20.590045 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:09:20.589941    2580 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3f62ecc5-82b8-4f02-ab8e-6ddb496456a0-console-config\") pod \"3f62ecc5-82b8-4f02-ab8e-6ddb496456a0\" (UID: \"3f62ecc5-82b8-4f02-ab8e-6ddb496456a0\") "
May 25 13:09:20.590045 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:09:20.589982    2580 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3f62ecc5-82b8-4f02-ab8e-6ddb496456a0-console-serving-cert\") pod \"3f62ecc5-82b8-4f02-ab8e-6ddb496456a0\" (UID: \"3f62ecc5-82b8-4f02-ab8e-6ddb496456a0\") "
May 25 13:09:20.590191 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:09:20.590137    2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f62ecc5-82b8-4f02-ab8e-6ddb496456a0-service-ca" (OuterVolumeSpecName: "service-ca") pod "3f62ecc5-82b8-4f02-ab8e-6ddb496456a0" (UID: "3f62ecc5-82b8-4f02-ab8e-6ddb496456a0"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 13:09:20.590266 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:09:20.590240    2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f62ecc5-82b8-4f02-ab8e-6ddb496456a0-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "3f62ecc5-82b8-4f02-ab8e-6ddb496456a0" (UID: "3f62ecc5-82b8-4f02-ab8e-6ddb496456a0"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 13:09:20.590502 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:09:20.590451    2580 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3f62ecc5-82b8-4f02-ab8e-6ddb496456a0-service-ca\") on node \"ip-10-0-135-103.ec2.internal\" DevicePath \"\""
May 25 13:09:20.590502 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:09:20.590477    2580 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3f62ecc5-82b8-4f02-ab8e-6ddb496456a0-trusted-ca-bundle\") on node \"ip-10-0-135-103.ec2.internal\" DevicePath \"\""
May 25 13:09:20.590660 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:09:20.590632    2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f62ecc5-82b8-4f02-ab8e-6ddb496456a0-console-config" (OuterVolumeSpecName: "console-config") pod "3f62ecc5-82b8-4f02-ab8e-6ddb496456a0" (UID: "3f62ecc5-82b8-4f02-ab8e-6ddb496456a0"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 13:09:20.590824 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:09:20.590803    2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f62ecc5-82b8-4f02-ab8e-6ddb496456a0-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "3f62ecc5-82b8-4f02-ab8e-6ddb496456a0" (UID: "3f62ecc5-82b8-4f02-ab8e-6ddb496456a0"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 13:09:20.592056 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:09:20.592029    2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f62ecc5-82b8-4f02-ab8e-6ddb496456a0-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "3f62ecc5-82b8-4f02-ab8e-6ddb496456a0" (UID: "3f62ecc5-82b8-4f02-ab8e-6ddb496456a0"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 13:09:20.592156 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:09:20.592062    2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f62ecc5-82b8-4f02-ab8e-6ddb496456a0-kube-api-access-ldmss" (OuterVolumeSpecName: "kube-api-access-ldmss") pod "3f62ecc5-82b8-4f02-ab8e-6ddb496456a0" (UID: "3f62ecc5-82b8-4f02-ab8e-6ddb496456a0"). InnerVolumeSpecName "kube-api-access-ldmss". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 13:09:20.592198 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:09:20.592147    2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f62ecc5-82b8-4f02-ab8e-6ddb496456a0-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "3f62ecc5-82b8-4f02-ab8e-6ddb496456a0" (UID: "3f62ecc5-82b8-4f02-ab8e-6ddb496456a0"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 13:09:20.691596 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:09:20.691560    2580 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3f62ecc5-82b8-4f02-ab8e-6ddb496456a0-console-serving-cert\") on node \"ip-10-0-135-103.ec2.internal\" DevicePath \"\""
May 25 13:09:20.691596 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:09:20.691590    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ldmss\" (UniqueName: \"kubernetes.io/projected/3f62ecc5-82b8-4f02-ab8e-6ddb496456a0-kube-api-access-ldmss\") on node \"ip-10-0-135-103.ec2.internal\" DevicePath \"\""
May 25 13:09:20.691596 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:09:20.691602    2580 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3f62ecc5-82b8-4f02-ab8e-6ddb496456a0-console-oauth-config\") on node \"ip-10-0-135-103.ec2.internal\" DevicePath \"\""
May 25 13:09:20.691833 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:09:20.691612    2580 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3f62ecc5-82b8-4f02-ab8e-6ddb496456a0-oauth-serving-cert\") on node \"ip-10-0-135-103.ec2.internal\" DevicePath \"\""
May 25 13:09:20.691833 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:09:20.691621    2580 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3f62ecc5-82b8-4f02-ab8e-6ddb496456a0-console-config\") on node \"ip-10-0-135-103.ec2.internal\" DevicePath \"\""
May 25 13:09:21.285423 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:09:21.285397    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-74f45df5f9-q5cfk_3f62ecc5-82b8-4f02-ab8e-6ddb496456a0/console/0.log"
May 25 13:09:21.285851 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:09:21.285437    2580 generic.go:358] "Generic (PLEG): container finished" podID="3f62ecc5-82b8-4f02-ab8e-6ddb496456a0" containerID="38ba31e251ea0ecd5463c64830500f8da50015dc47d9a3ef20c239a7e369218f" exitCode=2
May 25 13:09:21.285851 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:09:21.285472    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74f45df5f9-q5cfk" event={"ID":"3f62ecc5-82b8-4f02-ab8e-6ddb496456a0","Type":"ContainerDied","Data":"38ba31e251ea0ecd5463c64830500f8da50015dc47d9a3ef20c239a7e369218f"}
May 25 13:09:21.285851 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:09:21.285513    2580 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74f45df5f9-q5cfk"
May 25 13:09:21.285851 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:09:21.285525    2580 scope.go:117] "RemoveContainer" containerID="38ba31e251ea0ecd5463c64830500f8da50015dc47d9a3ef20c239a7e369218f"
May 25 13:09:21.285851 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:09:21.285515    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74f45df5f9-q5cfk" event={"ID":"3f62ecc5-82b8-4f02-ab8e-6ddb496456a0","Type":"ContainerDied","Data":"ad8976dc638b09c02e0c690dcb7ed722aa7332b3fe42cf9d1ce5180d3fdc3c42"}
May 25 13:09:21.294049 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:09:21.294033    2580 scope.go:117] "RemoveContainer" containerID="38ba31e251ea0ecd5463c64830500f8da50015dc47d9a3ef20c239a7e369218f"
May 25 13:09:21.294303 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:09:21.294282    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38ba31e251ea0ecd5463c64830500f8da50015dc47d9a3ef20c239a7e369218f\": container with ID starting with 38ba31e251ea0ecd5463c64830500f8da50015dc47d9a3ef20c239a7e369218f not found: ID does not exist" containerID="38ba31e251ea0ecd5463c64830500f8da50015dc47d9a3ef20c239a7e369218f"
May 25 13:09:21.294354 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:09:21.294312    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38ba31e251ea0ecd5463c64830500f8da50015dc47d9a3ef20c239a7e369218f"} err="failed to get container status \"38ba31e251ea0ecd5463c64830500f8da50015dc47d9a3ef20c239a7e369218f\": rpc error: code = NotFound desc = could not find container \"38ba31e251ea0ecd5463c64830500f8da50015dc47d9a3ef20c239a7e369218f\": container with ID starting with 38ba31e251ea0ecd5463c64830500f8da50015dc47d9a3ef20c239a7e369218f not found: ID does not exist"
May 25 13:09:21.306007 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:09:21.305961    2580 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-74f45df5f9-q5cfk"]
May 25 13:09:21.308446 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:09:21.308422    2580 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-74f45df5f9-q5cfk"]
May 25 13:09:22.376806 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:09:22.376774    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3f62ecc5-82b8-4f02-ab8e-6ddb496456a0" path="/var/lib/kubelet/pods/3f62ecc5-82b8-4f02-ab8e-6ddb496456a0/volumes"
May 25 13:09:58.475593 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:09:58.475558    2580 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console/console-6648679c85-9sc5q"]
May 25 13:09:58.476038 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:09:58.475839    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f62ecc5-82b8-4f02-ab8e-6ddb496456a0" containerName="console"
May 25 13:09:58.476038 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:09:58.475849    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f62ecc5-82b8-4f02-ab8e-6ddb496456a0" containerName="console"
May 25 13:09:58.476038 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:09:58.475908    2580 memory_manager.go:356] "RemoveStaleState removing state" podUID="3f62ecc5-82b8-4f02-ab8e-6ddb496456a0" containerName="console"
May 25 13:09:58.478976 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:09:58.478958    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6648679c85-9sc5q"
May 25 13:09:58.495279 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:09:58.495241    2580 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6648679c85-9sc5q"]
May 25 13:09:58.590233 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:09:58.590197    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c005f04e-2380-496b-a715-f4451c441717-console-serving-cert\") pod \"console-6648679c85-9sc5q\" (UID: \"c005f04e-2380-496b-a715-f4451c441717\") " pod="openshift-console/console-6648679c85-9sc5q"
May 25 13:09:58.590412 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:09:58.590239    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b47kd\" (UniqueName: \"kubernetes.io/projected/c005f04e-2380-496b-a715-f4451c441717-kube-api-access-b47kd\") pod \"console-6648679c85-9sc5q\" (UID: \"c005f04e-2380-496b-a715-f4451c441717\") " pod="openshift-console/console-6648679c85-9sc5q"
May 25 13:09:58.590412 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:09:58.590278    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c005f04e-2380-496b-a715-f4451c441717-service-ca\") pod \"console-6648679c85-9sc5q\" (UID: \"c005f04e-2380-496b-a715-f4451c441717\") " pod="openshift-console/console-6648679c85-9sc5q"
May 25 13:09:58.590538 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:09:58.590473    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c005f04e-2380-496b-a715-f4451c441717-oauth-serving-cert\") pod \"console-6648679c85-9sc5q\" (UID: \"c005f04e-2380-496b-a715-f4451c441717\") " pod="openshift-console/console-6648679c85-9sc5q"
May 25 13:09:58.590538 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:09:58.590514    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c005f04e-2380-496b-a715-f4451c441717-console-config\") pod \"console-6648679c85-9sc5q\" (UID: \"c005f04e-2380-496b-a715-f4451c441717\") " pod="openshift-console/console-6648679c85-9sc5q"
May 25 13:09:58.590538 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:09:58.590536    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c005f04e-2380-496b-a715-f4451c441717-trusted-ca-bundle\") pod \"console-6648679c85-9sc5q\" (UID: \"c005f04e-2380-496b-a715-f4451c441717\") " pod="openshift-console/console-6648679c85-9sc5q"
May 25 13:09:58.590669 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:09:58.590572    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c005f04e-2380-496b-a715-f4451c441717-console-oauth-config\") pod \"console-6648679c85-9sc5q\" (UID: \"c005f04e-2380-496b-a715-f4451c441717\") " pod="openshift-console/console-6648679c85-9sc5q"
May 25 13:09:58.691617 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:09:58.691573    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c005f04e-2380-496b-a715-f4451c441717-service-ca\") pod \"console-6648679c85-9sc5q\" (UID: \"c005f04e-2380-496b-a715-f4451c441717\") " pod="openshift-console/console-6648679c85-9sc5q"
May 25 13:09:58.691823 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:09:58.691633    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c005f04e-2380-496b-a715-f4451c441717-oauth-serving-cert\") pod \"console-6648679c85-9sc5q\" (UID: \"c005f04e-2380-496b-a715-f4451c441717\") " pod="openshift-console/console-6648679c85-9sc5q"
May 25 13:09:58.691823 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:09:58.691656    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c005f04e-2380-496b-a715-f4451c441717-console-config\") pod \"console-6648679c85-9sc5q\" (UID: \"c005f04e-2380-496b-a715-f4451c441717\") " pod="openshift-console/console-6648679c85-9sc5q"
May 25 13:09:58.691823 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:09:58.691677    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c005f04e-2380-496b-a715-f4451c441717-trusted-ca-bundle\") pod \"console-6648679c85-9sc5q\" (UID: \"c005f04e-2380-496b-a715-f4451c441717\") " pod="openshift-console/console-6648679c85-9sc5q"
May 25 13:09:58.691823 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:09:58.691694    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c005f04e-2380-496b-a715-f4451c441717-console-oauth-config\") pod \"console-6648679c85-9sc5q\" (UID: \"c005f04e-2380-496b-a715-f4451c441717\") " pod="openshift-console/console-6648679c85-9sc5q"
May 25 13:09:58.691823 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:09:58.691719    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c005f04e-2380-496b-a715-f4451c441717-console-serving-cert\") pod \"console-6648679c85-9sc5q\" (UID: \"c005f04e-2380-496b-a715-f4451c441717\") " pod="openshift-console/console-6648679c85-9sc5q"
May 25 13:09:58.691823 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:09:58.691736    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-b47kd\" (UniqueName: \"kubernetes.io/projected/c005f04e-2380-496b-a715-f4451c441717-kube-api-access-b47kd\") pod \"console-6648679c85-9sc5q\" (UID: \"c005f04e-2380-496b-a715-f4451c441717\") " pod="openshift-console/console-6648679c85-9sc5q"
May 25 13:09:58.692498 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:09:58.692472    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c005f04e-2380-496b-a715-f4451c441717-service-ca\") pod \"console-6648679c85-9sc5q\" (UID: \"c005f04e-2380-496b-a715-f4451c441717\") " pod="openshift-console/console-6648679c85-9sc5q"
May 25 13:09:58.692602 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:09:58.692570    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c005f04e-2380-496b-a715-f4451c441717-trusted-ca-bundle\") pod \"console-6648679c85-9sc5q\" (UID: \"c005f04e-2380-496b-a715-f4451c441717\") " pod="openshift-console/console-6648679c85-9sc5q"
May 25 13:09:58.692696 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:09:58.692676    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c005f04e-2380-496b-a715-f4451c441717-oauth-serving-cert\") pod \"console-6648679c85-9sc5q\" (UID: \"c005f04e-2380-496b-a715-f4451c441717\") " pod="openshift-console/console-6648679c85-9sc5q"
May 25 13:09:58.692799 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:09:58.692777    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c005f04e-2380-496b-a715-f4451c441717-console-config\") pod \"console-6648679c85-9sc5q\" (UID: \"c005f04e-2380-496b-a715-f4451c441717\") " pod="openshift-console/console-6648679c85-9sc5q"
May 25 13:09:58.694441 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:09:58.694409    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c005f04e-2380-496b-a715-f4451c441717-console-oauth-config\") pod \"console-6648679c85-9sc5q\" (UID: \"c005f04e-2380-496b-a715-f4451c441717\") " pod="openshift-console/console-6648679c85-9sc5q"
May 25 13:09:58.694521 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:09:58.694447    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c005f04e-2380-496b-a715-f4451c441717-console-serving-cert\") pod \"console-6648679c85-9sc5q\" (UID: \"c005f04e-2380-496b-a715-f4451c441717\") " pod="openshift-console/console-6648679c85-9sc5q"
May 25 13:09:58.699728 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:09:58.699709    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b47kd\" (UniqueName: \"kubernetes.io/projected/c005f04e-2380-496b-a715-f4451c441717-kube-api-access-b47kd\") pod \"console-6648679c85-9sc5q\" (UID: \"c005f04e-2380-496b-a715-f4451c441717\") " pod="openshift-console/console-6648679c85-9sc5q"
May 25 13:09:58.788023 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:09:58.787928    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6648679c85-9sc5q"
May 25 13:09:58.914527 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:09:58.914491    2580 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6648679c85-9sc5q"]
May 25 13:09:58.917802 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:09:58.917770    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc005f04e_2380_496b_a715_f4451c441717.slice/crio-a0983c02786453abba606e12b77f38c8b09d0003697360d9f93383515509871f WatchSource:0}: Error finding container a0983c02786453abba606e12b77f38c8b09d0003697360d9f93383515509871f: Status 404 returned error can't find the container with id a0983c02786453abba606e12b77f38c8b09d0003697360d9f93383515509871f
May 25 13:09:59.391905 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:09:59.391866    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6648679c85-9sc5q" event={"ID":"c005f04e-2380-496b-a715-f4451c441717","Type":"ContainerStarted","Data":"c788a2fe8a00d1d34c815394681cc37d21dc775b995a33d4a1cfadf71221bb3b"}
May 25 13:09:59.391905 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:09:59.391905    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6648679c85-9sc5q" event={"ID":"c005f04e-2380-496b-a715-f4451c441717","Type":"ContainerStarted","Data":"a0983c02786453abba606e12b77f38c8b09d0003697360d9f93383515509871f"}
May 25 13:09:59.409914 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:09:59.409867    2580 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console/console-6648679c85-9sc5q" podStartSLOduration=1.409851249 podStartE2EDuration="1.409851249s" podCreationTimestamp="2026-05-25 13:09:58 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 13:09:59.409067842 +0000 UTC m=+299.601362213" watchObservedRunningTime="2026-05-25 13:09:59.409851249 +0000 UTC m=+299.602145609"
May 25 13:10:00.253166 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:10:00.253134    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-646r6_59b905b8-5ce3-49f6-b47c-b5b3a525fe06/ovn-acl-logging/0.log"
May 25 13:10:00.254283 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:10:00.254257    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-646r6_59b905b8-5ce3-49f6-b47c-b5b3a525fe06/ovn-acl-logging/0.log"
May 25 13:10:00.259880 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:10:00.259863    2580 kubelet.go:1628] "Image garbage collection succeeded"
May 25 13:10:08.788231 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:10:08.788192    2580 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6648679c85-9sc5q"
May 25 13:10:08.788231 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:10:08.788230    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6648679c85-9sc5q"
May 25 13:10:08.792983 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:10:08.792964    2580 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6648679c85-9sc5q"
May 25 13:10:09.423672 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:10:09.423637    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6648679c85-9sc5q"
May 25 13:10:09.473818 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:10:09.473787    2580 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5bdcffdd98-9v5kx"]
May 25 13:10:34.494940 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:10:34.494873    2580 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-console/console-5bdcffdd98-9v5kx" podUID="d413aef7-390e-441a-a383-8196d1615d69" containerName="console" containerID="cri-o://488f7a764adbfa30527607fa404c2ea033ea40383271b493e9eb5e6c91314b9e" gracePeriod=15
May 25 13:10:34.736935 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:10:34.736911    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-5bdcffdd98-9v5kx_d413aef7-390e-441a-a383-8196d1615d69/console/0.log"
May 25 13:10:34.737062 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:10:34.736983    2580 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5bdcffdd98-9v5kx"
May 25 13:10:34.784821 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:10:34.784748    2580 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d413aef7-390e-441a-a383-8196d1615d69-trusted-ca-bundle\") pod \"d413aef7-390e-441a-a383-8196d1615d69\" (UID: \"d413aef7-390e-441a-a383-8196d1615d69\") "
May 25 13:10:34.784821 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:10:34.784804    2580 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d413aef7-390e-441a-a383-8196d1615d69-oauth-serving-cert\") pod \"d413aef7-390e-441a-a383-8196d1615d69\" (UID: \"d413aef7-390e-441a-a383-8196d1615d69\") "
May 25 13:10:34.785026 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:10:34.784827    2580 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d413aef7-390e-441a-a383-8196d1615d69-service-ca\") pod \"d413aef7-390e-441a-a383-8196d1615d69\" (UID: \"d413aef7-390e-441a-a383-8196d1615d69\") "
May 25 13:10:34.785026 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:10:34.784853    2580 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d413aef7-390e-441a-a383-8196d1615d69-console-oauth-config\") pod \"d413aef7-390e-441a-a383-8196d1615d69\" (UID: \"d413aef7-390e-441a-a383-8196d1615d69\") "
May 25 13:10:34.785026 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:10:34.784897    2580 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zpjqb\" (UniqueName: \"kubernetes.io/projected/d413aef7-390e-441a-a383-8196d1615d69-kube-api-access-zpjqb\") pod \"d413aef7-390e-441a-a383-8196d1615d69\" (UID: \"d413aef7-390e-441a-a383-8196d1615d69\") "
May 25 13:10:34.785026 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:10:34.784944    2580 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d413aef7-390e-441a-a383-8196d1615d69-console-config\") pod \"d413aef7-390e-441a-a383-8196d1615d69\" (UID: \"d413aef7-390e-441a-a383-8196d1615d69\") "
May 25 13:10:34.785026 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:10:34.784982    2580 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d413aef7-390e-441a-a383-8196d1615d69-console-serving-cert\") pod \"d413aef7-390e-441a-a383-8196d1615d69\" (UID: \"d413aef7-390e-441a-a383-8196d1615d69\") "
May 25 13:10:34.785267 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:10:34.785154    2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d413aef7-390e-441a-a383-8196d1615d69-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "d413aef7-390e-441a-a383-8196d1615d69" (UID: "d413aef7-390e-441a-a383-8196d1615d69"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 13:10:34.785267 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:10:34.785228    2580 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d413aef7-390e-441a-a383-8196d1615d69-trusted-ca-bundle\") on node \"ip-10-0-135-103.ec2.internal\" DevicePath \"\""
May 25 13:10:34.785428 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:10:34.785215    2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d413aef7-390e-441a-a383-8196d1615d69-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "d413aef7-390e-441a-a383-8196d1615d69" (UID: "d413aef7-390e-441a-a383-8196d1615d69"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 13:10:34.785428 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:10:34.785388    2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d413aef7-390e-441a-a383-8196d1615d69-console-config" (OuterVolumeSpecName: "console-config") pod "d413aef7-390e-441a-a383-8196d1615d69" (UID: "d413aef7-390e-441a-a383-8196d1615d69"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 13:10:34.785498 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:10:34.785479    2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d413aef7-390e-441a-a383-8196d1615d69-service-ca" (OuterVolumeSpecName: "service-ca") pod "d413aef7-390e-441a-a383-8196d1615d69" (UID: "d413aef7-390e-441a-a383-8196d1615d69"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 13:10:34.787332 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:10:34.787303    2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d413aef7-390e-441a-a383-8196d1615d69-kube-api-access-zpjqb" (OuterVolumeSpecName: "kube-api-access-zpjqb") pod "d413aef7-390e-441a-a383-8196d1615d69" (UID: "d413aef7-390e-441a-a383-8196d1615d69"). InnerVolumeSpecName "kube-api-access-zpjqb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 13:10:34.787332 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:10:34.787323    2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d413aef7-390e-441a-a383-8196d1615d69-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "d413aef7-390e-441a-a383-8196d1615d69" (UID: "d413aef7-390e-441a-a383-8196d1615d69"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 13:10:34.787522 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:10:34.787343    2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d413aef7-390e-441a-a383-8196d1615d69-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "d413aef7-390e-441a-a383-8196d1615d69" (UID: "d413aef7-390e-441a-a383-8196d1615d69"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 13:10:34.886035 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:10:34.885999    2580 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d413aef7-390e-441a-a383-8196d1615d69-oauth-serving-cert\") on node \"ip-10-0-135-103.ec2.internal\" DevicePath \"\""
May 25 13:10:34.886035 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:10:34.886030    2580 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d413aef7-390e-441a-a383-8196d1615d69-service-ca\") on node \"ip-10-0-135-103.ec2.internal\" DevicePath \"\""
May 25 13:10:34.886035 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:10:34.886040    2580 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d413aef7-390e-441a-a383-8196d1615d69-console-oauth-config\") on node \"ip-10-0-135-103.ec2.internal\" DevicePath \"\""
May 25 13:10:34.886253 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:10:34.886048    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zpjqb\" (UniqueName: \"kubernetes.io/projected/d413aef7-390e-441a-a383-8196d1615d69-kube-api-access-zpjqb\") on node \"ip-10-0-135-103.ec2.internal\" DevicePath \"\""
May 25 13:10:34.886253 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:10:34.886057    2580 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d413aef7-390e-441a-a383-8196d1615d69-console-config\") on node \"ip-10-0-135-103.ec2.internal\" DevicePath \"\""
May 25 13:10:34.886253 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:10:34.886066    2580 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d413aef7-390e-441a-a383-8196d1615d69-console-serving-cert\") on node \"ip-10-0-135-103.ec2.internal\" DevicePath \"\""
May 25 13:10:35.493698 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:10:35.493664    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-5bdcffdd98-9v5kx_d413aef7-390e-441a-a383-8196d1615d69/console/0.log"
May 25 13:10:35.493889 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:10:35.493712    2580 generic.go:358] "Generic (PLEG): container finished" podID="d413aef7-390e-441a-a383-8196d1615d69" containerID="488f7a764adbfa30527607fa404c2ea033ea40383271b493e9eb5e6c91314b9e" exitCode=2
May 25 13:10:35.493889 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:10:35.493743    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5bdcffdd98-9v5kx" event={"ID":"d413aef7-390e-441a-a383-8196d1615d69","Type":"ContainerDied","Data":"488f7a764adbfa30527607fa404c2ea033ea40383271b493e9eb5e6c91314b9e"}
May 25 13:10:35.493889 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:10:35.493783    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5bdcffdd98-9v5kx" event={"ID":"d413aef7-390e-441a-a383-8196d1615d69","Type":"ContainerDied","Data":"71435ccff8c05b8449bbbc7db0a06366ba3f713b5ecaf8b519eca2c6eaa934c2"}
May 25 13:10:35.493889 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:10:35.493783    2580 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5bdcffdd98-9v5kx"
May 25 13:10:35.493889 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:10:35.493797    2580 scope.go:117] "RemoveContainer" containerID="488f7a764adbfa30527607fa404c2ea033ea40383271b493e9eb5e6c91314b9e"
May 25 13:10:35.502096 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:10:35.501955    2580 scope.go:117] "RemoveContainer" containerID="488f7a764adbfa30527607fa404c2ea033ea40383271b493e9eb5e6c91314b9e"
May 25 13:10:35.502288 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:10:35.502202    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"488f7a764adbfa30527607fa404c2ea033ea40383271b493e9eb5e6c91314b9e\": container with ID starting with 488f7a764adbfa30527607fa404c2ea033ea40383271b493e9eb5e6c91314b9e not found: ID does not exist" containerID="488f7a764adbfa30527607fa404c2ea033ea40383271b493e9eb5e6c91314b9e"
May 25 13:10:35.502288 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:10:35.502225    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"488f7a764adbfa30527607fa404c2ea033ea40383271b493e9eb5e6c91314b9e"} err="failed to get container status \"488f7a764adbfa30527607fa404c2ea033ea40383271b493e9eb5e6c91314b9e\": rpc error: code = NotFound desc = could not find container \"488f7a764adbfa30527607fa404c2ea033ea40383271b493e9eb5e6c91314b9e\": container with ID starting with 488f7a764adbfa30527607fa404c2ea033ea40383271b493e9eb5e6c91314b9e not found: ID does not exist"
May 25 13:10:35.513182 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:10:35.513158    2580 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5bdcffdd98-9v5kx"]
May 25 13:10:35.518251 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:10:35.518232    2580 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5bdcffdd98-9v5kx"]
May 25 13:10:36.376714 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:10:36.376680    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d413aef7-390e-441a-a383-8196d1615d69" path="/var/lib/kubelet/pods/d413aef7-390e-441a-a383-8196d1615d69/volumes"
May 25 13:10:42.549083 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:10:42.549047    2580 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-rg7x8"]
May 25 13:10:42.549470 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:10:42.549325    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d413aef7-390e-441a-a383-8196d1615d69" containerName="console"
May 25 13:10:42.549470 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:10:42.549335    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="d413aef7-390e-441a-a383-8196d1615d69" containerName="console"
May 25 13:10:42.549470 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:10:42.549412    2580 memory_manager.go:356] "RemoveStaleState removing state" podUID="d413aef7-390e-441a-a383-8196d1615d69" containerName="console"
May 25 13:10:42.552148 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:10:42.552131    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rg7x8"
May 25 13:10:42.554861 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:10:42.554841    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
May 25 13:10:42.561381 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:10:42.561342    2580 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-rg7x8"]
May 25 13:10:42.651427 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:10:42.651334    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1cdfba04-46b6-40d5-9207-cce82288ca53-dbus\") pod \"global-pull-secret-syncer-rg7x8\" (UID: \"1cdfba04-46b6-40d5-9207-cce82288ca53\") " pod="kube-system/global-pull-secret-syncer-rg7x8"
May 25 13:10:42.651427 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:10:42.651422    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1cdfba04-46b6-40d5-9207-cce82288ca53-kubelet-config\") pod \"global-pull-secret-syncer-rg7x8\" (UID: \"1cdfba04-46b6-40d5-9207-cce82288ca53\") " pod="kube-system/global-pull-secret-syncer-rg7x8"
May 25 13:10:42.651616 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:10:42.651452    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1cdfba04-46b6-40d5-9207-cce82288ca53-original-pull-secret\") pod \"global-pull-secret-syncer-rg7x8\" (UID: \"1cdfba04-46b6-40d5-9207-cce82288ca53\") " pod="kube-system/global-pull-secret-syncer-rg7x8"
May 25 13:10:42.752122 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:10:42.752085    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1cdfba04-46b6-40d5-9207-cce82288ca53-original-pull-secret\") pod \"global-pull-secret-syncer-rg7x8\" (UID: \"1cdfba04-46b6-40d5-9207-cce82288ca53\") " pod="kube-system/global-pull-secret-syncer-rg7x8"
May 25 13:10:42.752294 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:10:42.752132    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1cdfba04-46b6-40d5-9207-cce82288ca53-dbus\") pod \"global-pull-secret-syncer-rg7x8\" (UID: \"1cdfba04-46b6-40d5-9207-cce82288ca53\") " pod="kube-system/global-pull-secret-syncer-rg7x8"
May 25 13:10:42.752294 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:10:42.752173    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1cdfba04-46b6-40d5-9207-cce82288ca53-kubelet-config\") pod \"global-pull-secret-syncer-rg7x8\" (UID: \"1cdfba04-46b6-40d5-9207-cce82288ca53\") " pod="kube-system/global-pull-secret-syncer-rg7x8"
May 25 13:10:42.752294 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:10:42.752249    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1cdfba04-46b6-40d5-9207-cce82288ca53-kubelet-config\") pod \"global-pull-secret-syncer-rg7x8\" (UID: \"1cdfba04-46b6-40d5-9207-cce82288ca53\") " pod="kube-system/global-pull-secret-syncer-rg7x8"
May 25 13:10:42.752478 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:10:42.752397    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1cdfba04-46b6-40d5-9207-cce82288ca53-dbus\") pod \"global-pull-secret-syncer-rg7x8\" (UID: \"1cdfba04-46b6-40d5-9207-cce82288ca53\") " pod="kube-system/global-pull-secret-syncer-rg7x8"
May 25 13:10:42.754602 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:10:42.754575    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1cdfba04-46b6-40d5-9207-cce82288ca53-original-pull-secret\") pod \"global-pull-secret-syncer-rg7x8\" (UID: \"1cdfba04-46b6-40d5-9207-cce82288ca53\") " pod="kube-system/global-pull-secret-syncer-rg7x8"
May 25 13:10:42.862004 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:10:42.861973    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rg7x8"
May 25 13:10:42.988652 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:10:42.988577    2580 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-rg7x8"]
May 25 13:10:42.991284 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:10:42.991253    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cdfba04_46b6_40d5_9207_cce82288ca53.slice/crio-31e50129b29e93ba7c1f93f789576db24bbf998fa86297249b99da63636888e8 WatchSource:0}: Error finding container 31e50129b29e93ba7c1f93f789576db24bbf998fa86297249b99da63636888e8: Status 404 returned error can't find the container with id 31e50129b29e93ba7c1f93f789576db24bbf998fa86297249b99da63636888e8
May 25 13:10:42.992834 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:10:42.992819    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 25 13:10:43.517924 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:10:43.517890    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-rg7x8" event={"ID":"1cdfba04-46b6-40d5-9207-cce82288ca53","Type":"ContainerStarted","Data":"31e50129b29e93ba7c1f93f789576db24bbf998fa86297249b99da63636888e8"}
May 25 13:10:47.532219 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:10:47.532126    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-rg7x8" event={"ID":"1cdfba04-46b6-40d5-9207-cce82288ca53","Type":"ContainerStarted","Data":"2823cbce72ed7d17b484901a3cbc16fa70f2cfeed62000644cb70c9b13390075"}
May 25 13:10:47.547360 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:10:47.547310    2580 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-rg7x8" podStartSLOduration=1.427001371 podStartE2EDuration="5.547295433s" podCreationTimestamp="2026-05-25 13:10:42 +0000 UTC" firstStartedPulling="2026-05-25 13:10:42.99293814 +0000 UTC m=+343.185232478" lastFinishedPulling="2026-05-25 13:10:47.113232199 +0000 UTC m=+347.305526540" observedRunningTime="2026-05-25 13:10:47.545710977 +0000 UTC m=+347.738005336" watchObservedRunningTime="2026-05-25 13:10:47.547295433 +0000 UTC m=+347.739589792"
May 25 13:11:02.035823 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:11:02.035785    2580 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cqvttk"]
May 25 13:11:02.040742 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:11:02.040721    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cqvttk"
May 25 13:11:02.043098 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:11:02.043072    2580 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
May 25 13:11:02.043839 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:11:02.043817    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-j8g9n\""
May 25 13:11:02.043839 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:11:02.043828    2580 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
May 25 13:11:02.046640 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:11:02.046616    2580 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cqvttk"]
May 25 13:11:02.217624 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:11:02.217585    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j46wz\" (UniqueName: \"kubernetes.io/projected/a2526c5a-4d8f-4dce-9078-e5e5711b7109-kube-api-access-j46wz\") pod \"59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cqvttk\" (UID: \"a2526c5a-4d8f-4dce-9078-e5e5711b7109\") " pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cqvttk"
May 25 13:11:02.217793 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:11:02.217654    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a2526c5a-4d8f-4dce-9078-e5e5711b7109-bundle\") pod \"59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cqvttk\" (UID: \"a2526c5a-4d8f-4dce-9078-e5e5711b7109\") " pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cqvttk"
May 25 13:11:02.217793 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:11:02.217679    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a2526c5a-4d8f-4dce-9078-e5e5711b7109-util\") pod \"59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cqvttk\" (UID: \"a2526c5a-4d8f-4dce-9078-e5e5711b7109\") " pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cqvttk"
May 25 13:11:02.318652 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:11:02.318569    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-j46wz\" (UniqueName: \"kubernetes.io/projected/a2526c5a-4d8f-4dce-9078-e5e5711b7109-kube-api-access-j46wz\") pod \"59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cqvttk\" (UID: \"a2526c5a-4d8f-4dce-9078-e5e5711b7109\") " pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cqvttk"
May 25 13:11:02.318652 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:11:02.318645    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a2526c5a-4d8f-4dce-9078-e5e5711b7109-bundle\") pod \"59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cqvttk\" (UID: \"a2526c5a-4d8f-4dce-9078-e5e5711b7109\") " pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cqvttk"
May 25 13:11:02.318815 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:11:02.318665    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a2526c5a-4d8f-4dce-9078-e5e5711b7109-util\") pod \"59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cqvttk\" (UID: \"a2526c5a-4d8f-4dce-9078-e5e5711b7109\") " pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cqvttk"
May 25 13:11:02.319091 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:11:02.319069    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a2526c5a-4d8f-4dce-9078-e5e5711b7109-bundle\") pod \"59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cqvttk\" (UID: \"a2526c5a-4d8f-4dce-9078-e5e5711b7109\") " pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cqvttk"
May 25 13:11:02.319131 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:11:02.319080    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a2526c5a-4d8f-4dce-9078-e5e5711b7109-util\") pod \"59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cqvttk\" (UID: \"a2526c5a-4d8f-4dce-9078-e5e5711b7109\") " pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cqvttk"
May 25 13:11:02.326773 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:11:02.326745    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j46wz\" (UniqueName: \"kubernetes.io/projected/a2526c5a-4d8f-4dce-9078-e5e5711b7109-kube-api-access-j46wz\") pod \"59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cqvttk\" (UID: \"a2526c5a-4d8f-4dce-9078-e5e5711b7109\") " pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cqvttk"
May 25 13:11:02.350215 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:11:02.350193    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cqvttk"
May 25 13:11:02.475437 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:11:02.475404    2580 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cqvttk"]
May 25 13:11:02.478541 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:11:02.478505    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2526c5a_4d8f_4dce_9078_e5e5711b7109.slice/crio-95d658eea2f5904d2a554a96735b31e4be774af8db21ab2794b9d9d162b8b96b WatchSource:0}: Error finding container 95d658eea2f5904d2a554a96735b31e4be774af8db21ab2794b9d9d162b8b96b: Status 404 returned error can't find the container with id 95d658eea2f5904d2a554a96735b31e4be774af8db21ab2794b9d9d162b8b96b
May 25 13:11:02.576929 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:11:02.576853    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cqvttk" event={"ID":"a2526c5a-4d8f-4dce-9078-e5e5711b7109","Type":"ContainerStarted","Data":"95d658eea2f5904d2a554a96735b31e4be774af8db21ab2794b9d9d162b8b96b"}
May 25 13:11:10.602296 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:11:10.602260    2580 generic.go:358] "Generic (PLEG): container finished" podID="a2526c5a-4d8f-4dce-9078-e5e5711b7109" containerID="39ecf4bb4a7ad27a3c63e18904b097b8e759c36ee80b2106e7a7604b56ed73a6" exitCode=0
May 25 13:11:10.602705 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:11:10.602307    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cqvttk" event={"ID":"a2526c5a-4d8f-4dce-9078-e5e5711b7109","Type":"ContainerDied","Data":"39ecf4bb4a7ad27a3c63e18904b097b8e759c36ee80b2106e7a7604b56ed73a6"}
May 25 13:11:12.610347 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:11:12.610302    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cqvttk" event={"ID":"a2526c5a-4d8f-4dce-9078-e5e5711b7109","Type":"ContainerStarted","Data":"e381d2c3d9b85d005fe0fdd8c5c7e65a419d6a546bfbadaf024c5089585dbfc5"}
May 25 13:11:13.614338 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:11:13.614301    2580 generic.go:358] "Generic (PLEG): container finished" podID="a2526c5a-4d8f-4dce-9078-e5e5711b7109" containerID="e381d2c3d9b85d005fe0fdd8c5c7e65a419d6a546bfbadaf024c5089585dbfc5" exitCode=0
May 25 13:11:13.614734 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:11:13.614348    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cqvttk" event={"ID":"a2526c5a-4d8f-4dce-9078-e5e5711b7109","Type":"ContainerDied","Data":"e381d2c3d9b85d005fe0fdd8c5c7e65a419d6a546bfbadaf024c5089585dbfc5"}
May 25 13:11:21.640834 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:11:21.640795    2580 generic.go:358] "Generic (PLEG): container finished" podID="a2526c5a-4d8f-4dce-9078-e5e5711b7109" containerID="312be22eb959e1622fe0d1e683811e637500575bffc3a5ae33ef23c08e9c53ba" exitCode=0
May 25 13:11:21.641201 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:11:21.640844    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cqvttk" event={"ID":"a2526c5a-4d8f-4dce-9078-e5e5711b7109","Type":"ContainerDied","Data":"312be22eb959e1622fe0d1e683811e637500575bffc3a5ae33ef23c08e9c53ba"}
May 25 13:11:22.767449 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:11:22.767426    2580 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cqvttk"
May 25 13:11:22.875453 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:11:22.875410    2580 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a2526c5a-4d8f-4dce-9078-e5e5711b7109-util\") pod \"a2526c5a-4d8f-4dce-9078-e5e5711b7109\" (UID: \"a2526c5a-4d8f-4dce-9078-e5e5711b7109\") "
May 25 13:11:22.875640 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:11:22.875467    2580 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j46wz\" (UniqueName: \"kubernetes.io/projected/a2526c5a-4d8f-4dce-9078-e5e5711b7109-kube-api-access-j46wz\") pod \"a2526c5a-4d8f-4dce-9078-e5e5711b7109\" (UID: \"a2526c5a-4d8f-4dce-9078-e5e5711b7109\") "
May 25 13:11:22.875640 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:11:22.875625    2580 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a2526c5a-4d8f-4dce-9078-e5e5711b7109-bundle\") pod \"a2526c5a-4d8f-4dce-9078-e5e5711b7109\" (UID: \"a2526c5a-4d8f-4dce-9078-e5e5711b7109\") "
May 25 13:11:22.876142 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:11:22.876117    2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2526c5a-4d8f-4dce-9078-e5e5711b7109-bundle" (OuterVolumeSpecName: "bundle") pod "a2526c5a-4d8f-4dce-9078-e5e5711b7109" (UID: "a2526c5a-4d8f-4dce-9078-e5e5711b7109"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 13:11:22.877758 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:11:22.877737    2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2526c5a-4d8f-4dce-9078-e5e5711b7109-kube-api-access-j46wz" (OuterVolumeSpecName: "kube-api-access-j46wz") pod "a2526c5a-4d8f-4dce-9078-e5e5711b7109" (UID: "a2526c5a-4d8f-4dce-9078-e5e5711b7109"). InnerVolumeSpecName "kube-api-access-j46wz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 13:11:22.880218 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:11:22.880195    2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2526c5a-4d8f-4dce-9078-e5e5711b7109-util" (OuterVolumeSpecName: "util") pod "a2526c5a-4d8f-4dce-9078-e5e5711b7109" (UID: "a2526c5a-4d8f-4dce-9078-e5e5711b7109"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 13:11:22.976450 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:11:22.976357    2580 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a2526c5a-4d8f-4dce-9078-e5e5711b7109-util\") on node \"ip-10-0-135-103.ec2.internal\" DevicePath \"\""
May 25 13:11:22.976450 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:11:22.976406    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j46wz\" (UniqueName: \"kubernetes.io/projected/a2526c5a-4d8f-4dce-9078-e5e5711b7109-kube-api-access-j46wz\") on node \"ip-10-0-135-103.ec2.internal\" DevicePath \"\""
May 25 13:11:22.976450 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:11:22.976417    2580 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a2526c5a-4d8f-4dce-9078-e5e5711b7109-bundle\") on node \"ip-10-0-135-103.ec2.internal\" DevicePath \"\""
May 25 13:11:23.648224 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:11:23.648184    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cqvttk" event={"ID":"a2526c5a-4d8f-4dce-9078-e5e5711b7109","Type":"ContainerDied","Data":"95d658eea2f5904d2a554a96735b31e4be774af8db21ab2794b9d9d162b8b96b"}
May 25 13:11:23.648224 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:11:23.648220    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="95d658eea2f5904d2a554a96735b31e4be774af8db21ab2794b9d9d162b8b96b"
May 25 13:11:23.648224 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:11:23.648230    2580 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29cqvttk"
May 25 13:11:28.777238 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:11:28.777205    2580 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-6llh9"]
May 25 13:11:28.777764 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:11:28.777537    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2526c5a-4d8f-4dce-9078-e5e5711b7109" containerName="extract"
May 25 13:11:28.777764 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:11:28.777551    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2526c5a-4d8f-4dce-9078-e5e5711b7109" containerName="extract"
May 25 13:11:28.777764 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:11:28.777559    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2526c5a-4d8f-4dce-9078-e5e5711b7109" containerName="util"
May 25 13:11:28.777764 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:11:28.777566    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2526c5a-4d8f-4dce-9078-e5e5711b7109" containerName="util"
May 25 13:11:28.777764 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:11:28.777587    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2526c5a-4d8f-4dce-9078-e5e5711b7109" containerName="pull"
May 25 13:11:28.777764 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:11:28.777592    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2526c5a-4d8f-4dce-9078-e5e5711b7109" containerName="pull"
May 25 13:11:28.777764 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:11:28.777677    2580 memory_manager.go:356] "RemoveStaleState removing state" podUID="a2526c5a-4d8f-4dce-9078-e5e5711b7109" containerName="extract"
May 25 13:11:28.782836 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:11:28.782817    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-6llh9"
May 25 13:11:28.785627 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:11:28.785593    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
May 25 13:11:28.785749 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:11:28.785626    2580 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
May 25 13:11:28.785749 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:11:28.785714    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"custom-metrics-autoscaler-operator-dockercfg-qht5q\""
May 25 13:11:28.785846 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:11:28.785831    2580 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
May 25 13:11:28.792552 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:11:28.792521    2580 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-6llh9"]
May 25 13:11:28.824632 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:11:28.824596    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p68ks\" (UniqueName: \"kubernetes.io/projected/a6f9916c-67e4-4af4-8649-b2e50dc290c4-kube-api-access-p68ks\") pod \"custom-metrics-autoscaler-operator-bbf89fd5d-6llh9\" (UID: \"a6f9916c-67e4-4af4-8649-b2e50dc290c4\") " pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-6llh9"
May 25 13:11:28.824806 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:11:28.824644    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/a6f9916c-67e4-4af4-8649-b2e50dc290c4-certificates\") pod \"custom-metrics-autoscaler-operator-bbf89fd5d-6llh9\" (UID: \"a6f9916c-67e4-4af4-8649-b2e50dc290c4\") " pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-6llh9"
May 25 13:11:28.925959 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:11:28.925913    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-p68ks\" (UniqueName: \"kubernetes.io/projected/a6f9916c-67e4-4af4-8649-b2e50dc290c4-kube-api-access-p68ks\") pod \"custom-metrics-autoscaler-operator-bbf89fd5d-6llh9\" (UID: \"a6f9916c-67e4-4af4-8649-b2e50dc290c4\") " pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-6llh9"
May 25 13:11:28.925959 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:11:28.925961    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/a6f9916c-67e4-4af4-8649-b2e50dc290c4-certificates\") pod \"custom-metrics-autoscaler-operator-bbf89fd5d-6llh9\" (UID: \"a6f9916c-67e4-4af4-8649-b2e50dc290c4\") " pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-6llh9"
May 25 13:11:28.928476 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:11:28.928451    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/a6f9916c-67e4-4af4-8649-b2e50dc290c4-certificates\") pod \"custom-metrics-autoscaler-operator-bbf89fd5d-6llh9\" (UID: \"a6f9916c-67e4-4af4-8649-b2e50dc290c4\") " pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-6llh9"
May 25 13:11:28.934733 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:11:28.934707    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p68ks\" (UniqueName: \"kubernetes.io/projected/a6f9916c-67e4-4af4-8649-b2e50dc290c4-kube-api-access-p68ks\") pod \"custom-metrics-autoscaler-operator-bbf89fd5d-6llh9\" (UID: \"a6f9916c-67e4-4af4-8649-b2e50dc290c4\") " pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-6llh9"
May 25 13:11:29.093788 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:11:29.093759    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-6llh9"
May 25 13:11:29.221792 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:11:29.221758    2580 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-6llh9"]
May 25 13:11:29.225334 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:11:29.225307    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6f9916c_67e4_4af4_8649_b2e50dc290c4.slice/crio-47c2da19d50054a4a06240bb26f7fa559069e798036f13cf735c7474a672df01 WatchSource:0}: Error finding container 47c2da19d50054a4a06240bb26f7fa559069e798036f13cf735c7474a672df01: Status 404 returned error can't find the container with id 47c2da19d50054a4a06240bb26f7fa559069e798036f13cf735c7474a672df01
May 25 13:11:29.666778 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:11:29.666736    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-6llh9" event={"ID":"a6f9916c-67e4-4af4-8649-b2e50dc290c4","Type":"ContainerStarted","Data":"47c2da19d50054a4a06240bb26f7fa559069e798036f13cf735c7474a672df01"}
May 25 13:11:33.482136 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:11:33.482103    2580 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-operator-ffbb595cb-frsll"]
May 25 13:11:33.485339 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:11:33.485322    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-ffbb595cb-frsll"
May 25 13:11:33.488741 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:11:33.488716    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-vljsq\""
May 25 13:11:33.489121 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:11:33.489100    2580 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
May 25 13:11:33.489969 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:11:33.489953    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-certs\""
May 25 13:11:33.496476 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:11:33.496453    2580 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-ffbb595cb-frsll"]
May 25 13:11:33.566672 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:11:33.566643    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/82717ff0-ee68-4f21-8bed-5dd7ed8a0eff-certificates\") pod \"keda-operator-ffbb595cb-frsll\" (UID: \"82717ff0-ee68-4f21-8bed-5dd7ed8a0eff\") " pod="openshift-keda/keda-operator-ffbb595cb-frsll"
May 25 13:11:33.566834 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:11:33.566707    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/82717ff0-ee68-4f21-8bed-5dd7ed8a0eff-cabundle0\") pod \"keda-operator-ffbb595cb-frsll\" (UID: \"82717ff0-ee68-4f21-8bed-5dd7ed8a0eff\") " pod="openshift-keda/keda-operator-ffbb595cb-frsll"
May 25 13:11:33.566834 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:11:33.566739    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jx5p\" (UniqueName: \"kubernetes.io/projected/82717ff0-ee68-4f21-8bed-5dd7ed8a0eff-kube-api-access-6jx5p\") pod \"keda-operator-ffbb595cb-frsll\" (UID: \"82717ff0-ee68-4f21-8bed-5dd7ed8a0eff\") " pod="openshift-keda/keda-operator-ffbb595cb-frsll"
May 25 13:11:33.668116 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:11:33.668082    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/82717ff0-ee68-4f21-8bed-5dd7ed8a0eff-cabundle0\") pod \"keda-operator-ffbb595cb-frsll\" (UID: \"82717ff0-ee68-4f21-8bed-5dd7ed8a0eff\") " pod="openshift-keda/keda-operator-ffbb595cb-frsll"
May 25 13:11:33.668284 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:11:33.668133    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-6jx5p\" (UniqueName: \"kubernetes.io/projected/82717ff0-ee68-4f21-8bed-5dd7ed8a0eff-kube-api-access-6jx5p\") pod \"keda-operator-ffbb595cb-frsll\" (UID: \"82717ff0-ee68-4f21-8bed-5dd7ed8a0eff\") " pod="openshift-keda/keda-operator-ffbb595cb-frsll"
May 25 13:11:33.668284 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:11:33.668159    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/82717ff0-ee68-4f21-8bed-5dd7ed8a0eff-certificates\") pod \"keda-operator-ffbb595cb-frsll\" (UID: \"82717ff0-ee68-4f21-8bed-5dd7ed8a0eff\") " pod="openshift-keda/keda-operator-ffbb595cb-frsll"
May 25 13:11:33.668284 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:11:33.668261    2580 secret.go:281] references non-existent secret key: ca.crt
May 25 13:11:33.668284 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:11:33.668272    2580 projected.go:277] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
May 25 13:11:33.668284 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:11:33.668282    2580 projected.go:194] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-ffbb595cb-frsll: references non-existent secret key: ca.crt
May 25 13:11:33.668560 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:11:33.668343    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/82717ff0-ee68-4f21-8bed-5dd7ed8a0eff-certificates podName:82717ff0-ee68-4f21-8bed-5dd7ed8a0eff nodeName:}" failed. No retries permitted until 2026-05-25 13:11:34.168324178 +0000 UTC m=+394.360618517 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/82717ff0-ee68-4f21-8bed-5dd7ed8a0eff-certificates") pod "keda-operator-ffbb595cb-frsll" (UID: "82717ff0-ee68-4f21-8bed-5dd7ed8a0eff") : references non-existent secret key: ca.crt
May 25 13:11:33.668875 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:11:33.668850    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/82717ff0-ee68-4f21-8bed-5dd7ed8a0eff-cabundle0\") pod \"keda-operator-ffbb595cb-frsll\" (UID: \"82717ff0-ee68-4f21-8bed-5dd7ed8a0eff\") " pod="openshift-keda/keda-operator-ffbb595cb-frsll"
May 25 13:11:33.677023 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:11:33.676999    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jx5p\" (UniqueName: \"kubernetes.io/projected/82717ff0-ee68-4f21-8bed-5dd7ed8a0eff-kube-api-access-6jx5p\") pod \"keda-operator-ffbb595cb-frsll\" (UID: \"82717ff0-ee68-4f21-8bed-5dd7ed8a0eff\") " pod="openshift-keda/keda-operator-ffbb595cb-frsll"
May 25 13:11:33.691077 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:11:33.691037    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-6llh9" event={"ID":"a6f9916c-67e4-4af4-8649-b2e50dc290c4","Type":"ContainerStarted","Data":"07362dc2ea25e416a700d9b5550be9d06777c757d62d604e900dd0c9099daf7f"}
May 25 13:11:33.691252 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:11:33.691159    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-6llh9"
May 25 13:11:33.718294 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:11:33.718246    2580 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-6llh9" podStartSLOduration=2.044166356 podStartE2EDuration="5.718230167s" podCreationTimestamp="2026-05-25 13:11:28 +0000 UTC" firstStartedPulling="2026-05-25 13:11:29.227086034 +0000 UTC m=+389.419380372" lastFinishedPulling="2026-05-25 13:11:32.901149845 +0000 UTC m=+393.093444183" observedRunningTime="2026-05-25 13:11:33.715759485 +0000 UTC m=+393.908053845" watchObservedRunningTime="2026-05-25 13:11:33.718230167 +0000 UTC m=+393.910524523"
May 25 13:11:33.752394 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:11:33.752304    2580 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-metrics-apiserver-7c9f485588-pvtrl"]
May 25 13:11:33.755751 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:11:33.755733    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-7c9f485588-pvtrl"
May 25 13:11:33.757974 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:11:33.757950    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-metrics-apiserver-certs\""
May 25 13:11:33.764403 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:11:33.764360    2580 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-7c9f485588-pvtrl"]
May 25 13:11:33.870101 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:11:33.870048    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfltl\" (UniqueName: \"kubernetes.io/projected/178a9080-1099-4003-8280-7fa311921612-kube-api-access-sfltl\") pod \"keda-metrics-apiserver-7c9f485588-pvtrl\" (UID: \"178a9080-1099-4003-8280-7fa311921612\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-pvtrl"
May 25 13:11:33.870101 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:11:33.870120    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/178a9080-1099-4003-8280-7fa311921612-certificates\") pod \"keda-metrics-apiserver-7c9f485588-pvtrl\" (UID: \"178a9080-1099-4003-8280-7fa311921612\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-pvtrl"
May 25 13:11:33.870352 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:11:33.870224    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/178a9080-1099-4003-8280-7fa311921612-temp-vol\") pod \"keda-metrics-apiserver-7c9f485588-pvtrl\" (UID: \"178a9080-1099-4003-8280-7fa311921612\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-pvtrl"
May 25 13:11:33.970632 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:11:33.970595    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/178a9080-1099-4003-8280-7fa311921612-temp-vol\") pod \"keda-metrics-apiserver-7c9f485588-pvtrl\" (UID: \"178a9080-1099-4003-8280-7fa311921612\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-pvtrl"
May 25 13:11:33.970819 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:11:33.970658    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-sfltl\" (UniqueName: \"kubernetes.io/projected/178a9080-1099-4003-8280-7fa311921612-kube-api-access-sfltl\") pod \"keda-metrics-apiserver-7c9f485588-pvtrl\" (UID: \"178a9080-1099-4003-8280-7fa311921612\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-pvtrl"
May 25 13:11:33.970819 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:11:33.970695    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/178a9080-1099-4003-8280-7fa311921612-certificates\") pod \"keda-metrics-apiserver-7c9f485588-pvtrl\" (UID: \"178a9080-1099-4003-8280-7fa311921612\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-pvtrl"
May 25 13:11:33.970929 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:11:33.970826    2580 secret.go:281] references non-existent secret key: tls.crt
May 25 13:11:33.970929 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:11:33.970842    2580 projected.go:277] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: tls.crt
May 25 13:11:33.970929 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:11:33.970873    2580 projected.go:194] Error preparing data for projected volume certificates for pod openshift-keda/keda-metrics-apiserver-7c9f485588-pvtrl: references non-existent secret key: tls.crt
May 25 13:11:33.971038 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:11:33.970940    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/178a9080-1099-4003-8280-7fa311921612-certificates podName:178a9080-1099-4003-8280-7fa311921612 nodeName:}" failed. No retries permitted until 2026-05-25 13:11:34.470921101 +0000 UTC m=+394.663215460 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/178a9080-1099-4003-8280-7fa311921612-certificates") pod "keda-metrics-apiserver-7c9f485588-pvtrl" (UID: "178a9080-1099-4003-8280-7fa311921612") : references non-existent secret key: tls.crt
May 25 13:11:33.971097 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:11:33.971060    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/178a9080-1099-4003-8280-7fa311921612-temp-vol\") pod \"keda-metrics-apiserver-7c9f485588-pvtrl\" (UID: \"178a9080-1099-4003-8280-7fa311921612\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-pvtrl"
May 25 13:11:33.985860 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:11:33.985827    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfltl\" (UniqueName: \"kubernetes.io/projected/178a9080-1099-4003-8280-7fa311921612-kube-api-access-sfltl\") pod \"keda-metrics-apiserver-7c9f485588-pvtrl\" (UID: \"178a9080-1099-4003-8280-7fa311921612\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-pvtrl"
May 25 13:11:34.172243 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:11:34.172201    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/82717ff0-ee68-4f21-8bed-5dd7ed8a0eff-certificates\") pod \"keda-operator-ffbb595cb-frsll\" (UID: \"82717ff0-ee68-4f21-8bed-5dd7ed8a0eff\") " pod="openshift-keda/keda-operator-ffbb595cb-frsll"
May 25 13:11:34.172447 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:11:34.172409    2580 secret.go:281] references non-existent secret key: ca.crt
May 25 13:11:34.172447 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:11:34.172436    2580 projected.go:277] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
May 25 13:11:34.172523 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:11:34.172449    2580 projected.go:194] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-ffbb595cb-frsll: references non-existent secret key: ca.crt
May 25 13:11:34.172523 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:11:34.172517    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/82717ff0-ee68-4f21-8bed-5dd7ed8a0eff-certificates podName:82717ff0-ee68-4f21-8bed-5dd7ed8a0eff nodeName:}" failed. No retries permitted until 2026-05-25 13:11:35.172496209 +0000 UTC m=+395.364790553 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/82717ff0-ee68-4f21-8bed-5dd7ed8a0eff-certificates") pod "keda-operator-ffbb595cb-frsll" (UID: "82717ff0-ee68-4f21-8bed-5dd7ed8a0eff") : references non-existent secret key: ca.crt
May 25 13:11:34.475085 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:11:34.474992    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/178a9080-1099-4003-8280-7fa311921612-certificates\") pod \"keda-metrics-apiserver-7c9f485588-pvtrl\" (UID: \"178a9080-1099-4003-8280-7fa311921612\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-pvtrl"
May 25 13:11:34.475241 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:11:34.475145    2580 secret.go:281] references non-existent secret key: tls.crt
May 25 13:11:34.475241 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:11:34.475159    2580 projected.go:277] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: tls.crt
May 25 13:11:34.475241 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:11:34.475176    2580 projected.go:194] Error preparing data for projected volume certificates for pod openshift-keda/keda-metrics-apiserver-7c9f485588-pvtrl: references non-existent secret key: tls.crt
May 25 13:11:34.475241 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:11:34.475228    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/178a9080-1099-4003-8280-7fa311921612-certificates podName:178a9080-1099-4003-8280-7fa311921612 nodeName:}" failed. No retries permitted until 2026-05-25 13:11:35.475214172 +0000 UTC m=+395.667508510 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/178a9080-1099-4003-8280-7fa311921612-certificates") pod "keda-metrics-apiserver-7c9f485588-pvtrl" (UID: "178a9080-1099-4003-8280-7fa311921612") : references non-existent secret key: tls.crt
May 25 13:11:35.180576 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:11:35.180539    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/82717ff0-ee68-4f21-8bed-5dd7ed8a0eff-certificates\") pod \"keda-operator-ffbb595cb-frsll\" (UID: \"82717ff0-ee68-4f21-8bed-5dd7ed8a0eff\") " pod="openshift-keda/keda-operator-ffbb595cb-frsll"
May 25 13:11:35.181038 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:11:35.180703    2580 secret.go:281] references non-existent secret key: ca.crt
May 25 13:11:35.181038 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:11:35.180729    2580 projected.go:277] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
May 25 13:11:35.181038 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:11:35.180743    2580 projected.go:194] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-ffbb595cb-frsll: references non-existent secret key: ca.crt
May 25 13:11:35.181038 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:11:35.180826    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/82717ff0-ee68-4f21-8bed-5dd7ed8a0eff-certificates podName:82717ff0-ee68-4f21-8bed-5dd7ed8a0eff nodeName:}" failed. No retries permitted until 2026-05-25 13:11:37.180804413 +0000 UTC m=+397.373098755 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/82717ff0-ee68-4f21-8bed-5dd7ed8a0eff-certificates") pod "keda-operator-ffbb595cb-frsll" (UID: "82717ff0-ee68-4f21-8bed-5dd7ed8a0eff") : references non-existent secret key: ca.crt
May 25 13:11:35.483316 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:11:35.483231    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/178a9080-1099-4003-8280-7fa311921612-certificates\") pod \"keda-metrics-apiserver-7c9f485588-pvtrl\" (UID: \"178a9080-1099-4003-8280-7fa311921612\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-pvtrl"
May 25 13:11:35.483509 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:11:35.483416    2580 secret.go:281] references non-existent secret key: tls.crt
May 25 13:11:35.483509 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:11:35.483444    2580 projected.go:277] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: tls.crt
May 25 13:11:35.483509 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:11:35.483467    2580 projected.go:194] Error preparing data for projected volume certificates for pod openshift-keda/keda-metrics-apiserver-7c9f485588-pvtrl: references non-existent secret key: tls.crt
May 25 13:11:35.483671 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:11:35.483532    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/178a9080-1099-4003-8280-7fa311921612-certificates podName:178a9080-1099-4003-8280-7fa311921612 nodeName:}" failed. No retries permitted until 2026-05-25 13:11:37.483512674 +0000 UTC m=+397.675807019 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/178a9080-1099-4003-8280-7fa311921612-certificates") pod "keda-metrics-apiserver-7c9f485588-pvtrl" (UID: "178a9080-1099-4003-8280-7fa311921612") : references non-existent secret key: tls.crt
May 25 13:11:37.199453 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:11:37.199411    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/82717ff0-ee68-4f21-8bed-5dd7ed8a0eff-certificates\") pod \"keda-operator-ffbb595cb-frsll\" (UID: \"82717ff0-ee68-4f21-8bed-5dd7ed8a0eff\") " pod="openshift-keda/keda-operator-ffbb595cb-frsll"
May 25 13:11:37.199819 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:11:37.199550    2580 secret.go:281] references non-existent secret key: ca.crt
May 25 13:11:37.199819 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:11:37.199564    2580 projected.go:277] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
May 25 13:11:37.199819 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:11:37.199573    2580 projected.go:194] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-ffbb595cb-frsll: references non-existent secret key: ca.crt
May 25 13:11:37.199819 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:11:37.199627    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/82717ff0-ee68-4f21-8bed-5dd7ed8a0eff-certificates podName:82717ff0-ee68-4f21-8bed-5dd7ed8a0eff nodeName:}" failed. No retries permitted until 2026-05-25 13:11:41.199608661 +0000 UTC m=+401.391903010 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/82717ff0-ee68-4f21-8bed-5dd7ed8a0eff-certificates") pod "keda-operator-ffbb595cb-frsll" (UID: "82717ff0-ee68-4f21-8bed-5dd7ed8a0eff") : references non-existent secret key: ca.crt
May 25 13:11:37.502763 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:11:37.502667    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/178a9080-1099-4003-8280-7fa311921612-certificates\") pod \"keda-metrics-apiserver-7c9f485588-pvtrl\" (UID: \"178a9080-1099-4003-8280-7fa311921612\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-pvtrl"
May 25 13:11:37.502929 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:11:37.502805    2580 secret.go:281] references non-existent secret key: tls.crt
May 25 13:11:37.502929 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:11:37.502827    2580 projected.go:277] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: tls.crt
May 25 13:11:37.502929 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:11:37.502846    2580 projected.go:194] Error preparing data for projected volume certificates for pod openshift-keda/keda-metrics-apiserver-7c9f485588-pvtrl: references non-existent secret key: tls.crt
May 25 13:11:37.502929 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:11:37.502909    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/178a9080-1099-4003-8280-7fa311921612-certificates podName:178a9080-1099-4003-8280-7fa311921612 nodeName:}" failed. No retries permitted until 2026-05-25 13:11:41.50289333 +0000 UTC m=+401.695187668 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/178a9080-1099-4003-8280-7fa311921612-certificates") pod "keda-metrics-apiserver-7c9f485588-pvtrl" (UID: "178a9080-1099-4003-8280-7fa311921612") : references non-existent secret key: tls.crt
May 25 13:11:41.236147 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:11:41.236109    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/82717ff0-ee68-4f21-8bed-5dd7ed8a0eff-certificates\") pod \"keda-operator-ffbb595cb-frsll\" (UID: \"82717ff0-ee68-4f21-8bed-5dd7ed8a0eff\") " pod="openshift-keda/keda-operator-ffbb595cb-frsll"
May 25 13:11:41.238604 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:11:41.238579    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/82717ff0-ee68-4f21-8bed-5dd7ed8a0eff-certificates\") pod \"keda-operator-ffbb595cb-frsll\" (UID: \"82717ff0-ee68-4f21-8bed-5dd7ed8a0eff\") " pod="openshift-keda/keda-operator-ffbb595cb-frsll"
May 25 13:11:41.295540 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:11:41.295519    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-ffbb595cb-frsll"
May 25 13:11:41.422130 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:11:41.422104    2580 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-ffbb595cb-frsll"]
May 25 13:11:41.424578 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:11:41.424549    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82717ff0_ee68_4f21_8bed_5dd7ed8a0eff.slice/crio-1b6998cb7e98b78523afe709617a3ab7aaab2893fe7107e44a0d89bb157c52fe WatchSource:0}: Error finding container 1b6998cb7e98b78523afe709617a3ab7aaab2893fe7107e44a0d89bb157c52fe: Status 404 returned error can't find the container with id 1b6998cb7e98b78523afe709617a3ab7aaab2893fe7107e44a0d89bb157c52fe
May 25 13:11:41.539428 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:11:41.539340    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/178a9080-1099-4003-8280-7fa311921612-certificates\") pod \"keda-metrics-apiserver-7c9f485588-pvtrl\" (UID: \"178a9080-1099-4003-8280-7fa311921612\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-pvtrl"
May 25 13:11:41.542030 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:11:41.541997    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/178a9080-1099-4003-8280-7fa311921612-certificates\") pod \"keda-metrics-apiserver-7c9f485588-pvtrl\" (UID: \"178a9080-1099-4003-8280-7fa311921612\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-pvtrl"
May 25 13:11:41.566806 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:11:41.566767    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-7c9f485588-pvtrl"
May 25 13:11:41.689875 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:11:41.689850    2580 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-7c9f485588-pvtrl"]
May 25 13:11:41.691564 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:11:41.691533    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod178a9080_1099_4003_8280_7fa311921612.slice/crio-3a078f2ded9aedc72ece3f30b5679f34b7f07bc482269ce0c2e1f560c0e3d2e7 WatchSource:0}: Error finding container 3a078f2ded9aedc72ece3f30b5679f34b7f07bc482269ce0c2e1f560c0e3d2e7: Status 404 returned error can't find the container with id 3a078f2ded9aedc72ece3f30b5679f34b7f07bc482269ce0c2e1f560c0e3d2e7
May 25 13:11:41.722284 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:11:41.722246    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-ffbb595cb-frsll" event={"ID":"82717ff0-ee68-4f21-8bed-5dd7ed8a0eff","Type":"ContainerStarted","Data":"1b6998cb7e98b78523afe709617a3ab7aaab2893fe7107e44a0d89bb157c52fe"}
May 25 13:11:41.723251 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:11:41.723219    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-7c9f485588-pvtrl" event={"ID":"178a9080-1099-4003-8280-7fa311921612","Type":"ContainerStarted","Data":"3a078f2ded9aedc72ece3f30b5679f34b7f07bc482269ce0c2e1f560c0e3d2e7"}
May 25 13:11:46.743212 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:11:46.743166    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-7c9f485588-pvtrl" event={"ID":"178a9080-1099-4003-8280-7fa311921612","Type":"ContainerStarted","Data":"2cc28d6f3ad3cb0f70602932153ad8c8823a96d55077f10bafbc4c0970b8c8ad"}
May 25 13:11:46.743728 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:11:46.743331    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-metrics-apiserver-7c9f485588-pvtrl"
May 25 13:11:46.744639 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:11:46.744612    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-ffbb595cb-frsll" event={"ID":"82717ff0-ee68-4f21-8bed-5dd7ed8a0eff","Type":"ContainerStarted","Data":"2fecd17f9ecbc25bd4daad9401491d82c70b2d311d8ddbab0d6992d87fa150b3"}
May 25 13:11:46.744779 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:11:46.744730    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-operator-ffbb595cb-frsll"
May 25 13:11:46.763469 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:11:46.763423    2580 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-keda/keda-metrics-apiserver-7c9f485588-pvtrl" podStartSLOduration=9.461032409 podStartE2EDuration="13.763410281s" podCreationTimestamp="2026-05-25 13:11:33 +0000 UTC" firstStartedPulling="2026-05-25 13:11:41.692826703 +0000 UTC m=+401.885121041" lastFinishedPulling="2026-05-25 13:11:45.995204561 +0000 UTC m=+406.187498913" observedRunningTime="2026-05-25 13:11:46.761769861 +0000 UTC m=+406.954064222" watchObservedRunningTime="2026-05-25 13:11:46.763410281 +0000 UTC m=+406.955704640"
May 25 13:11:46.781940 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:11:46.781898    2580 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-keda/keda-operator-ffbb595cb-frsll" podStartSLOduration=9.201096679 podStartE2EDuration="13.781881279s" podCreationTimestamp="2026-05-25 13:11:33 +0000 UTC" firstStartedPulling="2026-05-25 13:11:41.425775177 +0000 UTC m=+401.618069516" lastFinishedPulling="2026-05-25 13:11:46.006559775 +0000 UTC m=+406.198854116" observedRunningTime="2026-05-25 13:11:46.779579203 +0000 UTC m=+406.971873581" watchObservedRunningTime="2026-05-25 13:11:46.781881279 +0000 UTC m=+406.974175640"
May 25 13:11:54.695909 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:11:54.695876    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-6llh9"
May 25 13:11:57.753089 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:11:57.753064    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-metrics-apiserver-7c9f485588-pvtrl"
May 25 13:12:07.749527 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:12:07.749493    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-operator-ffbb595cb-frsll"
May 25 13:12:44.102891 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:12:44.102855    2580 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-79f457656b-p928f"]
May 25 13:12:44.105442 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:12:44.105421    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-79f457656b-p928f"
May 25 13:12:44.109138 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:12:44.109119    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
May 25 13:12:44.109285 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:12:44.109259    2580 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
May 25 13:12:44.110160 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:12:44.110143    2580 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
May 25 13:12:44.110248 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:12:44.110146    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-r99bh\""
May 25 13:12:44.128269 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:12:44.128242    2580 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-79f457656b-p928f"]
May 25 13:12:44.269915 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:12:44.269861    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/1a80a0f5-44b8-43b6-a873-f7943feed70f-data\") pod \"seaweedfs-79f457656b-p928f\" (UID: \"1a80a0f5-44b8-43b6-a873-f7943feed70f\") " pod="kserve/seaweedfs-79f457656b-p928f"
May 25 13:12:44.270090 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:12:44.269970    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h98pd\" (UniqueName: \"kubernetes.io/projected/1a80a0f5-44b8-43b6-a873-f7943feed70f-kube-api-access-h98pd\") pod \"seaweedfs-79f457656b-p928f\" (UID: \"1a80a0f5-44b8-43b6-a873-f7943feed70f\") " pod="kserve/seaweedfs-79f457656b-p928f"
May 25 13:12:44.371414 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:12:44.371317    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/1a80a0f5-44b8-43b6-a873-f7943feed70f-data\") pod \"seaweedfs-79f457656b-p928f\" (UID: \"1a80a0f5-44b8-43b6-a873-f7943feed70f\") " pod="kserve/seaweedfs-79f457656b-p928f"
May 25 13:12:44.371414 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:12:44.371395    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-h98pd\" (UniqueName: \"kubernetes.io/projected/1a80a0f5-44b8-43b6-a873-f7943feed70f-kube-api-access-h98pd\") pod \"seaweedfs-79f457656b-p928f\" (UID: \"1a80a0f5-44b8-43b6-a873-f7943feed70f\") " pod="kserve/seaweedfs-79f457656b-p928f"
May 25 13:12:44.371694 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:12:44.371672    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/1a80a0f5-44b8-43b6-a873-f7943feed70f-data\") pod \"seaweedfs-79f457656b-p928f\" (UID: \"1a80a0f5-44b8-43b6-a873-f7943feed70f\") " pod="kserve/seaweedfs-79f457656b-p928f"
May 25 13:12:44.379959 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:12:44.379933    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h98pd\" (UniqueName: \"kubernetes.io/projected/1a80a0f5-44b8-43b6-a873-f7943feed70f-kube-api-access-h98pd\") pod \"seaweedfs-79f457656b-p928f\" (UID: \"1a80a0f5-44b8-43b6-a873-f7943feed70f\") " pod="kserve/seaweedfs-79f457656b-p928f"
May 25 13:12:44.414779 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:12:44.414751    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-79f457656b-p928f"
May 25 13:12:44.539680 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:12:44.539655    2580 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-79f457656b-p928f"]
May 25 13:12:44.544788 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:12:44.544757    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a80a0f5_44b8_43b6_a873_f7943feed70f.slice/crio-01b2a4d58a5510055613ed1151563a21b8d86ee9562a49070529b9ac5634ff45 WatchSource:0}: Error finding container 01b2a4d58a5510055613ed1151563a21b8d86ee9562a49070529b9ac5634ff45: Status 404 returned error can't find the container with id 01b2a4d58a5510055613ed1151563a21b8d86ee9562a49070529b9ac5634ff45
May 25 13:12:44.946332 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:12:44.946295    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-79f457656b-p928f" event={"ID":"1a80a0f5-44b8-43b6-a873-f7943feed70f","Type":"ContainerStarted","Data":"01b2a4d58a5510055613ed1151563a21b8d86ee9562a49070529b9ac5634ff45"}
May 25 13:12:48.961729 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:12:48.961691    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-79f457656b-p928f" event={"ID":"1a80a0f5-44b8-43b6-a873-f7943feed70f","Type":"ContainerStarted","Data":"4df26e5ef91857a5a7986a0ffa5575e78a562badc2d78765739903ec7e7bf463"}
May 25 13:12:48.962144 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:12:48.961916    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-79f457656b-p928f"
May 25 13:12:48.981526 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:12:48.981468    2580 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve/seaweedfs-79f457656b-p928f" podStartSLOduration=1.5027695859999999 podStartE2EDuration="4.981454914s" podCreationTimestamp="2026-05-25 13:12:44 +0000 UTC" firstStartedPulling="2026-05-25 13:12:44.546204499 +0000 UTC m=+464.738498853" lastFinishedPulling="2026-05-25 13:12:48.024889842 +0000 UTC m=+468.217184181" observedRunningTime="2026-05-25 13:12:48.98080035 +0000 UTC m=+469.173094710" watchObservedRunningTime="2026-05-25 13:12:48.981454914 +0000 UTC m=+469.173749273"
May 25 13:12:54.966374 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:12:54.966341    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-79f457656b-p928f"
May 25 13:13:50.859663 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:13:50.859585    2580 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console/console-6c7dc66496-6dvf5"]
May 25 13:13:50.862829 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:13:50.862813    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c7dc66496-6dvf5"
May 25 13:13:50.875470 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:13:50.875446    2580 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6c7dc66496-6dvf5"]
May 25 13:13:51.026570 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:13:51.026534    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9fffd9b4-fef3-4dd5-ad3a-3fa68d4c5fad-oauth-serving-cert\") pod \"console-6c7dc66496-6dvf5\" (UID: \"9fffd9b4-fef3-4dd5-ad3a-3fa68d4c5fad\") " pod="openshift-console/console-6c7dc66496-6dvf5"
May 25 13:13:51.026752 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:13:51.026597    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9fffd9b4-fef3-4dd5-ad3a-3fa68d4c5fad-console-serving-cert\") pod \"console-6c7dc66496-6dvf5\" (UID: \"9fffd9b4-fef3-4dd5-ad3a-3fa68d4c5fad\") " pod="openshift-console/console-6c7dc66496-6dvf5"
May 25 13:13:51.026752 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:13:51.026623    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9fffd9b4-fef3-4dd5-ad3a-3fa68d4c5fad-console-oauth-config\") pod \"console-6c7dc66496-6dvf5\" (UID: \"9fffd9b4-fef3-4dd5-ad3a-3fa68d4c5fad\") " pod="openshift-console/console-6c7dc66496-6dvf5"
May 25 13:13:51.026752 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:13:51.026642    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9fffd9b4-fef3-4dd5-ad3a-3fa68d4c5fad-console-config\") pod \"console-6c7dc66496-6dvf5\" (UID: \"9fffd9b4-fef3-4dd5-ad3a-3fa68d4c5fad\") " pod="openshift-console/console-6c7dc66496-6dvf5"
May 25 13:13:51.026752 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:13:51.026671    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbrsx\" (UniqueName: \"kubernetes.io/projected/9fffd9b4-fef3-4dd5-ad3a-3fa68d4c5fad-kube-api-access-nbrsx\") pod \"console-6c7dc66496-6dvf5\" (UID: \"9fffd9b4-fef3-4dd5-ad3a-3fa68d4c5fad\") " pod="openshift-console/console-6c7dc66496-6dvf5"
May 25 13:13:51.026752 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:13:51.026734    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9fffd9b4-fef3-4dd5-ad3a-3fa68d4c5fad-trusted-ca-bundle\") pod \"console-6c7dc66496-6dvf5\" (UID: \"9fffd9b4-fef3-4dd5-ad3a-3fa68d4c5fad\") " pod="openshift-console/console-6c7dc66496-6dvf5"
May 25 13:13:51.026966 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:13:51.026770    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9fffd9b4-fef3-4dd5-ad3a-3fa68d4c5fad-service-ca\") pod \"console-6c7dc66496-6dvf5\" (UID: \"9fffd9b4-fef3-4dd5-ad3a-3fa68d4c5fad\") " pod="openshift-console/console-6c7dc66496-6dvf5"
May 25 13:13:51.127563 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:13:51.127470    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-nbrsx\" (UniqueName: \"kubernetes.io/projected/9fffd9b4-fef3-4dd5-ad3a-3fa68d4c5fad-kube-api-access-nbrsx\") pod \"console-6c7dc66496-6dvf5\" (UID: \"9fffd9b4-fef3-4dd5-ad3a-3fa68d4c5fad\") " pod="openshift-console/console-6c7dc66496-6dvf5"
May 25 13:13:51.127563 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:13:51.127535    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9fffd9b4-fef3-4dd5-ad3a-3fa68d4c5fad-trusted-ca-bundle\") pod \"console-6c7dc66496-6dvf5\" (UID: \"9fffd9b4-fef3-4dd5-ad3a-3fa68d4c5fad\") " pod="openshift-console/console-6c7dc66496-6dvf5"
May 25 13:13:51.127778 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:13:51.127577    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9fffd9b4-fef3-4dd5-ad3a-3fa68d4c5fad-service-ca\") pod \"console-6c7dc66496-6dvf5\" (UID: \"9fffd9b4-fef3-4dd5-ad3a-3fa68d4c5fad\") " pod="openshift-console/console-6c7dc66496-6dvf5"
May 25 13:13:51.127778 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:13:51.127602    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9fffd9b4-fef3-4dd5-ad3a-3fa68d4c5fad-oauth-serving-cert\") pod \"console-6c7dc66496-6dvf5\" (UID: \"9fffd9b4-fef3-4dd5-ad3a-3fa68d4c5fad\") " pod="openshift-console/console-6c7dc66496-6dvf5"
May 25 13:13:51.127778 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:13:51.127652    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9fffd9b4-fef3-4dd5-ad3a-3fa68d4c5fad-console-serving-cert\") pod \"console-6c7dc66496-6dvf5\" (UID: \"9fffd9b4-fef3-4dd5-ad3a-3fa68d4c5fad\") " pod="openshift-console/console-6c7dc66496-6dvf5"
May 25 13:13:51.127778 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:13:51.127675    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9fffd9b4-fef3-4dd5-ad3a-3fa68d4c5fad-console-oauth-config\") pod \"console-6c7dc66496-6dvf5\" (UID: \"9fffd9b4-fef3-4dd5-ad3a-3fa68d4c5fad\") " pod="openshift-console/console-6c7dc66496-6dvf5"
May 25 13:13:51.127778 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:13:51.127695    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9fffd9b4-fef3-4dd5-ad3a-3fa68d4c5fad-console-config\") pod \"console-6c7dc66496-6dvf5\" (UID: \"9fffd9b4-fef3-4dd5-ad3a-3fa68d4c5fad\") " pod="openshift-console/console-6c7dc66496-6dvf5"
May 25 13:13:51.128284 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:13:51.128249    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9fffd9b4-fef3-4dd5-ad3a-3fa68d4c5fad-service-ca\") pod \"console-6c7dc66496-6dvf5\" (UID: \"9fffd9b4-fef3-4dd5-ad3a-3fa68d4c5fad\") " pod="openshift-console/console-6c7dc66496-6dvf5"
May 25 13:13:51.128495 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:13:51.128467    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9fffd9b4-fef3-4dd5-ad3a-3fa68d4c5fad-trusted-ca-bundle\") pod \"console-6c7dc66496-6dvf5\" (UID: \"9fffd9b4-fef3-4dd5-ad3a-3fa68d4c5fad\") " pod="openshift-console/console-6c7dc66496-6dvf5"
May 25 13:13:51.128606 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:13:51.128471    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9fffd9b4-fef3-4dd5-ad3a-3fa68d4c5fad-oauth-serving-cert\") pod \"console-6c7dc66496-6dvf5\" (UID: \"9fffd9b4-fef3-4dd5-ad3a-3fa68d4c5fad\") " pod="openshift-console/console-6c7dc66496-6dvf5"
May 25 13:13:51.128830 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:13:51.128808    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9fffd9b4-fef3-4dd5-ad3a-3fa68d4c5fad-console-config\") pod \"console-6c7dc66496-6dvf5\" (UID: \"9fffd9b4-fef3-4dd5-ad3a-3fa68d4c5fad\") " pod="openshift-console/console-6c7dc66496-6dvf5"
May 25 13:13:51.130291 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:13:51.130264    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9fffd9b4-fef3-4dd5-ad3a-3fa68d4c5fad-console-serving-cert\") pod \"console-6c7dc66496-6dvf5\" (UID: \"9fffd9b4-fef3-4dd5-ad3a-3fa68d4c5fad\") " pod="openshift-console/console-6c7dc66496-6dvf5"
May 25 13:13:51.130386 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:13:51.130268    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9fffd9b4-fef3-4dd5-ad3a-3fa68d4c5fad-console-oauth-config\") pod \"console-6c7dc66496-6dvf5\" (UID: \"9fffd9b4-fef3-4dd5-ad3a-3fa68d4c5fad\") " pod="openshift-console/console-6c7dc66496-6dvf5"
May 25 13:13:51.135956 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:13:51.135938    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbrsx\" (UniqueName: \"kubernetes.io/projected/9fffd9b4-fef3-4dd5-ad3a-3fa68d4c5fad-kube-api-access-nbrsx\") pod \"console-6c7dc66496-6dvf5\" (UID: \"9fffd9b4-fef3-4dd5-ad3a-3fa68d4c5fad\") " pod="openshift-console/console-6c7dc66496-6dvf5"
May 25 13:13:51.172023 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:13:51.171995    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c7dc66496-6dvf5"
May 25 13:13:51.301764 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:13:51.301561    2580 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6c7dc66496-6dvf5"]
May 25 13:13:51.304577 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:13:51.304550    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fffd9b4_fef3_4dd5_ad3a_3fa68d4c5fad.slice/crio-adbb1f6ba06ba85884f577633e69b9050fc88f6adb5b05658da6f0d02cc23743 WatchSource:0}: Error finding container adbb1f6ba06ba85884f577633e69b9050fc88f6adb5b05658da6f0d02cc23743: Status 404 returned error can't find the container with id adbb1f6ba06ba85884f577633e69b9050fc88f6adb5b05658da6f0d02cc23743
May 25 13:13:52.168647 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:13:52.168609    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c7dc66496-6dvf5" event={"ID":"9fffd9b4-fef3-4dd5-ad3a-3fa68d4c5fad","Type":"ContainerStarted","Data":"9d289479c1e5fec454677ce7efaa5b1e5082c178e319c61fb05ed43c4995945a"}
May 25 13:13:52.168647 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:13:52.168650    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c7dc66496-6dvf5" event={"ID":"9fffd9b4-fef3-4dd5-ad3a-3fa68d4c5fad","Type":"ContainerStarted","Data":"adbb1f6ba06ba85884f577633e69b9050fc88f6adb5b05658da6f0d02cc23743"}
May 25 13:13:52.190701 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:13:52.190654    2580 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console/console-6c7dc66496-6dvf5" podStartSLOduration=2.190640857 podStartE2EDuration="2.190640857s" podCreationTimestamp="2026-05-25 13:13:50 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 13:13:52.190037482 +0000 UTC m=+532.382331842" watchObservedRunningTime="2026-05-25 13:13:52.190640857 +0000 UTC m=+532.382935217"
May 25 13:13:54.739076 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:13:54.739039    2580 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-86f7b4b499-vxhdm"]
May 25 13:13:54.742476 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:13:54.742458    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-86f7b4b499-vxhdm"
May 25 13:13:54.744932 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:13:54.744910    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-cq6gq\""
May 25 13:13:54.745050 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:13:54.744941    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
May 25 13:13:54.752909 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:13:54.752882    2580 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-86f7b4b499-vxhdm"]
May 25 13:13:54.760100 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:13:54.760071    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56sf4\" (UniqueName: \"kubernetes.io/projected/eaf98502-5fa4-474f-8567-ffa3d161e84e-kube-api-access-56sf4\") pod \"model-serving-api-86f7b4b499-vxhdm\" (UID: \"eaf98502-5fa4-474f-8567-ffa3d161e84e\") " pod="kserve/model-serving-api-86f7b4b499-vxhdm"
May 25 13:13:54.760202 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:13:54.760115    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eaf98502-5fa4-474f-8567-ffa3d161e84e-tls-certs\") pod \"model-serving-api-86f7b4b499-vxhdm\" (UID: \"eaf98502-5fa4-474f-8567-ffa3d161e84e\") " pod="kserve/model-serving-api-86f7b4b499-vxhdm"
May 25 13:13:54.860631 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:13:54.860598    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-56sf4\" (UniqueName: \"kubernetes.io/projected/eaf98502-5fa4-474f-8567-ffa3d161e84e-kube-api-access-56sf4\") pod \"model-serving-api-86f7b4b499-vxhdm\" (UID: \"eaf98502-5fa4-474f-8567-ffa3d161e84e\") " pod="kserve/model-serving-api-86f7b4b499-vxhdm"
May 25 13:13:54.860786 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:13:54.860640    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eaf98502-5fa4-474f-8567-ffa3d161e84e-tls-certs\") pod \"model-serving-api-86f7b4b499-vxhdm\" (UID: \"eaf98502-5fa4-474f-8567-ffa3d161e84e\") " pod="kserve/model-serving-api-86f7b4b499-vxhdm"
May 25 13:13:54.860786 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:13:54.860766    2580 secret.go:189] Couldn't get secret kserve/model-serving-api-tls: secret "model-serving-api-tls" not found
May 25 13:13:54.860855 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:13:54.860826    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eaf98502-5fa4-474f-8567-ffa3d161e84e-tls-certs podName:eaf98502-5fa4-474f-8567-ffa3d161e84e nodeName:}" failed. No retries permitted until 2026-05-25 13:13:55.360808692 +0000 UTC m=+535.553103029 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/eaf98502-5fa4-474f-8567-ffa3d161e84e-tls-certs") pod "model-serving-api-86f7b4b499-vxhdm" (UID: "eaf98502-5fa4-474f-8567-ffa3d161e84e") : secret "model-serving-api-tls" not found
May 25 13:13:54.871407 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:13:54.871353    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-56sf4\" (UniqueName: \"kubernetes.io/projected/eaf98502-5fa4-474f-8567-ffa3d161e84e-kube-api-access-56sf4\") pod \"model-serving-api-86f7b4b499-vxhdm\" (UID: \"eaf98502-5fa4-474f-8567-ffa3d161e84e\") " pod="kserve/model-serving-api-86f7b4b499-vxhdm"
May 25 13:13:55.365269 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:13:55.365230    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eaf98502-5fa4-474f-8567-ffa3d161e84e-tls-certs\") pod \"model-serving-api-86f7b4b499-vxhdm\" (UID: \"eaf98502-5fa4-474f-8567-ffa3d161e84e\") " pod="kserve/model-serving-api-86f7b4b499-vxhdm"
May 25 13:13:55.367767 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:13:55.367746    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eaf98502-5fa4-474f-8567-ffa3d161e84e-tls-certs\") pod \"model-serving-api-86f7b4b499-vxhdm\" (UID: \"eaf98502-5fa4-474f-8567-ffa3d161e84e\") " pod="kserve/model-serving-api-86f7b4b499-vxhdm"
May 25 13:13:55.653594 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:13:55.653506    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-86f7b4b499-vxhdm"
May 25 13:13:55.782722 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:13:55.782697    2580 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-86f7b4b499-vxhdm"]
May 25 13:13:55.785328 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:13:55.785293    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeaf98502_5fa4_474f_8567_ffa3d161e84e.slice/crio-02169b5b34ba6e5e507b48acab24620c40cb8797d135a90ef6e0eef15f41156a WatchSource:0}: Error finding container 02169b5b34ba6e5e507b48acab24620c40cb8797d135a90ef6e0eef15f41156a: Status 404 returned error can't find the container with id 02169b5b34ba6e5e507b48acab24620c40cb8797d135a90ef6e0eef15f41156a
May 25 13:13:56.183418 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:13:56.183358    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-86f7b4b499-vxhdm" event={"ID":"eaf98502-5fa4-474f-8567-ffa3d161e84e","Type":"ContainerStarted","Data":"02169b5b34ba6e5e507b48acab24620c40cb8797d135a90ef6e0eef15f41156a"}
May 25 13:13:59.194541 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:13:59.194503    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-86f7b4b499-vxhdm" event={"ID":"eaf98502-5fa4-474f-8567-ffa3d161e84e","Type":"ContainerStarted","Data":"03a0b22397a129de11864c947e2e1386cd576067c2f05c36a80b596641c4fc6d"}
May 25 13:13:59.194945 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:13:59.194656    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-86f7b4b499-vxhdm"
May 25 13:13:59.211528 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:13:59.211487    2580 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve/model-serving-api-86f7b4b499-vxhdm" podStartSLOduration=2.814621319 podStartE2EDuration="5.211475102s" podCreationTimestamp="2026-05-25 13:13:54 +0000 UTC" firstStartedPulling="2026-05-25 13:13:55.786958119 +0000 UTC m=+535.979252458" lastFinishedPulling="2026-05-25 13:13:58.183811896 +0000 UTC m=+538.376106241" observedRunningTime="2026-05-25 13:13:59.21013493 +0000 UTC m=+539.402429303" watchObservedRunningTime="2026-05-25 13:13:59.211475102 +0000 UTC m=+539.403769462"
May 25 13:14:01.172906 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:14:01.172872    2580 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6c7dc66496-6dvf5"
May 25 13:14:01.172906 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:14:01.172913    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6c7dc66496-6dvf5"
May 25 13:14:01.177252 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:14:01.177231    2580 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6c7dc66496-6dvf5"
May 25 13:14:01.204201 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:14:01.204169    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6c7dc66496-6dvf5"
May 25 13:14:01.254057 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:14:01.254012    2580 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6648679c85-9sc5q"]
May 25 13:14:10.202307 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:14:10.202277    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-86f7b4b499-vxhdm"
May 25 13:14:26.277744 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:14:26.277700    2580 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-console/console-6648679c85-9sc5q" podUID="c005f04e-2380-496b-a715-f4451c441717" containerName="console" containerID="cri-o://c788a2fe8a00d1d34c815394681cc37d21dc775b995a33d4a1cfadf71221bb3b" gracePeriod=15
May 25 13:14:26.523347 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:14:26.523314    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-6648679c85-9sc5q_c005f04e-2380-496b-a715-f4451c441717/console/0.log"
May 25 13:14:26.523497 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:14:26.523407    2580 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6648679c85-9sc5q"
May 25 13:14:26.637810 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:14:26.637775    2580 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c005f04e-2380-496b-a715-f4451c441717-trusted-ca-bundle\") pod \"c005f04e-2380-496b-a715-f4451c441717\" (UID: \"c005f04e-2380-496b-a715-f4451c441717\") "
May 25 13:14:26.638041 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:14:26.637822    2580 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c005f04e-2380-496b-a715-f4451c441717-oauth-serving-cert\") pod \"c005f04e-2380-496b-a715-f4451c441717\" (UID: \"c005f04e-2380-496b-a715-f4451c441717\") "
May 25 13:14:26.638041 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:14:26.637842    2580 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c005f04e-2380-496b-a715-f4451c441717-console-serving-cert\") pod \"c005f04e-2380-496b-a715-f4451c441717\" (UID: \"c005f04e-2380-496b-a715-f4451c441717\") "
May 25 13:14:26.638041 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:14:26.637873    2580 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c005f04e-2380-496b-a715-f4451c441717-service-ca\") pod \"c005f04e-2380-496b-a715-f4451c441717\" (UID: \"c005f04e-2380-496b-a715-f4451c441717\") "
May 25 13:14:26.638041 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:14:26.637939    2580 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b47kd\" (UniqueName: \"kubernetes.io/projected/c005f04e-2380-496b-a715-f4451c441717-kube-api-access-b47kd\") pod \"c005f04e-2380-496b-a715-f4451c441717\" (UID: \"c005f04e-2380-496b-a715-f4451c441717\") "
May 25 13:14:26.638041 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:14:26.637955    2580 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c005f04e-2380-496b-a715-f4451c441717-console-oauth-config\") pod \"c005f04e-2380-496b-a715-f4451c441717\" (UID: \"c005f04e-2380-496b-a715-f4451c441717\") "
May 25 13:14:26.638041 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:14:26.637993    2580 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c005f04e-2380-496b-a715-f4451c441717-console-config\") pod \"c005f04e-2380-496b-a715-f4451c441717\" (UID: \"c005f04e-2380-496b-a715-f4451c441717\") "
May 25 13:14:26.638451 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:14:26.638319    2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c005f04e-2380-496b-a715-f4451c441717-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "c005f04e-2380-496b-a715-f4451c441717" (UID: "c005f04e-2380-496b-a715-f4451c441717"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 13:14:26.638451 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:14:26.638340    2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c005f04e-2380-496b-a715-f4451c441717-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "c005f04e-2380-496b-a715-f4451c441717" (UID: "c005f04e-2380-496b-a715-f4451c441717"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 13:14:26.638553 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:14:26.638430    2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c005f04e-2380-496b-a715-f4451c441717-service-ca" (OuterVolumeSpecName: "service-ca") pod "c005f04e-2380-496b-a715-f4451c441717" (UID: "c005f04e-2380-496b-a715-f4451c441717"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 13:14:26.638619 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:14:26.638597    2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c005f04e-2380-496b-a715-f4451c441717-console-config" (OuterVolumeSpecName: "console-config") pod "c005f04e-2380-496b-a715-f4451c441717" (UID: "c005f04e-2380-496b-a715-f4451c441717"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 13:14:26.640204 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:14:26.640179    2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c005f04e-2380-496b-a715-f4451c441717-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "c005f04e-2380-496b-a715-f4451c441717" (UID: "c005f04e-2380-496b-a715-f4451c441717"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 13:14:26.640356 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:14:26.640328    2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c005f04e-2380-496b-a715-f4451c441717-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "c005f04e-2380-496b-a715-f4451c441717" (UID: "c005f04e-2380-496b-a715-f4451c441717"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 13:14:26.640456 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:14:26.640352    2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c005f04e-2380-496b-a715-f4451c441717-kube-api-access-b47kd" (OuterVolumeSpecName: "kube-api-access-b47kd") pod "c005f04e-2380-496b-a715-f4451c441717" (UID: "c005f04e-2380-496b-a715-f4451c441717"). InnerVolumeSpecName "kube-api-access-b47kd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 13:14:26.739195 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:14:26.739163    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b47kd\" (UniqueName: \"kubernetes.io/projected/c005f04e-2380-496b-a715-f4451c441717-kube-api-access-b47kd\") on node \"ip-10-0-135-103.ec2.internal\" DevicePath \"\""
May 25 13:14:26.739195 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:14:26.739192    2580 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c005f04e-2380-496b-a715-f4451c441717-console-oauth-config\") on node \"ip-10-0-135-103.ec2.internal\" DevicePath \"\""
May 25 13:14:26.739195 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:14:26.739202    2580 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c005f04e-2380-496b-a715-f4451c441717-console-config\") on node \"ip-10-0-135-103.ec2.internal\" DevicePath \"\""
May 25 13:14:26.739443 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:14:26.739214    2580 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c005f04e-2380-496b-a715-f4451c441717-trusted-ca-bundle\") on node \"ip-10-0-135-103.ec2.internal\" DevicePath \"\""
May 25 13:14:26.739443 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:14:26.739223    2580 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c005f04e-2380-496b-a715-f4451c441717-oauth-serving-cert\") on node \"ip-10-0-135-103.ec2.internal\" DevicePath \"\""
May 25 13:14:26.739443 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:14:26.739232    2580 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c005f04e-2380-496b-a715-f4451c441717-console-serving-cert\") on node \"ip-10-0-135-103.ec2.internal\" DevicePath \"\""
May 25 13:14:26.739443 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:14:26.739240    2580 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c005f04e-2380-496b-a715-f4451c441717-service-ca\") on node \"ip-10-0-135-103.ec2.internal\" DevicePath \"\""
May 25 13:14:27.287638 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:14:27.287605    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-6648679c85-9sc5q_c005f04e-2380-496b-a715-f4451c441717/console/0.log"
May 25 13:14:27.288049 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:14:27.287651    2580 generic.go:358] "Generic (PLEG): container finished" podID="c005f04e-2380-496b-a715-f4451c441717" containerID="c788a2fe8a00d1d34c815394681cc37d21dc775b995a33d4a1cfadf71221bb3b" exitCode=2
May 25 13:14:27.288049 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:14:27.287720    2580 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6648679c85-9sc5q"
May 25 13:14:27.288049 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:14:27.287740    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6648679c85-9sc5q" event={"ID":"c005f04e-2380-496b-a715-f4451c441717","Type":"ContainerDied","Data":"c788a2fe8a00d1d34c815394681cc37d21dc775b995a33d4a1cfadf71221bb3b"}
May 25 13:14:27.288049 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:14:27.287776    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6648679c85-9sc5q" event={"ID":"c005f04e-2380-496b-a715-f4451c441717","Type":"ContainerDied","Data":"a0983c02786453abba606e12b77f38c8b09d0003697360d9f93383515509871f"}
May 25 13:14:27.288049 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:14:27.287790    2580 scope.go:117] "RemoveContainer" containerID="c788a2fe8a00d1d34c815394681cc37d21dc775b995a33d4a1cfadf71221bb3b"
May 25 13:14:27.297229 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:14:27.297208    2580 scope.go:117] "RemoveContainer" containerID="c788a2fe8a00d1d34c815394681cc37d21dc775b995a33d4a1cfadf71221bb3b"
May 25 13:14:27.297555 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:14:27.297535    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c788a2fe8a00d1d34c815394681cc37d21dc775b995a33d4a1cfadf71221bb3b\": container with ID starting with c788a2fe8a00d1d34c815394681cc37d21dc775b995a33d4a1cfadf71221bb3b not found: ID does not exist" containerID="c788a2fe8a00d1d34c815394681cc37d21dc775b995a33d4a1cfadf71221bb3b"
May 25 13:14:27.297631 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:14:27.297566    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c788a2fe8a00d1d34c815394681cc37d21dc775b995a33d4a1cfadf71221bb3b"} err="failed to get container status \"c788a2fe8a00d1d34c815394681cc37d21dc775b995a33d4a1cfadf71221bb3b\": rpc error: code = NotFound desc = could not find container \"c788a2fe8a00d1d34c815394681cc37d21dc775b995a33d4a1cfadf71221bb3b\": container with ID starting with c788a2fe8a00d1d34c815394681cc37d21dc775b995a33d4a1cfadf71221bb3b not found: ID does not exist"
May 25 13:14:27.309776 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:14:27.309744    2580 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6648679c85-9sc5q"]
May 25 13:14:27.313756 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:14:27.313730    2580 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6648679c85-9sc5q"]
May 25 13:14:28.380724 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:14:28.380694    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c005f04e-2380-496b-a715-f4451c441717" path="/var/lib/kubelet/pods/c005f04e-2380-496b-a715-f4451c441717/volumes"
May 25 13:14:30.653085 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:14:30.653047    2580 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-889c1-predictor-746ccd4c5b-pjzvp"]
May 25 13:14:30.653487 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:14:30.653389    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c005f04e-2380-496b-a715-f4451c441717" containerName="console"
May 25 13:14:30.653487 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:14:30.653400    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="c005f04e-2380-496b-a715-f4451c441717" containerName="console"
May 25 13:14:30.653487 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:14:30.653465    2580 memory_manager.go:356] "RemoveStaleState removing state" podUID="c005f04e-2380-496b-a715-f4451c441717" containerName="console"
May 25 13:14:30.657946 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:14:30.657925    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-889c1-predictor-746ccd4c5b-pjzvp"
May 25 13:14:30.665966 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:14:30.665938    2580 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-889c1-kube-rbac-proxy-sar-config\""
May 25 13:14:30.666095 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:14:30.665938    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-889c1-predictor-serving-cert\""
May 25 13:14:30.666095 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:14:30.665994    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-7pmnd\""
May 25 13:14:30.666095 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:14:30.666039    2580 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
May 25 13:14:30.666095 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:14:30.665950    2580 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
May 25 13:14:30.675789 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:14:30.675764    2580 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-889c1-predictor-746ccd4c5b-pjzvp"]
May 25 13:14:30.774596 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:14:30.774559    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"success-200-isvc-889c1-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cbdc46e1-4ffb-4a9d-9eef-6615a72fe025-success-200-isvc-889c1-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-889c1-predictor-746ccd4c5b-pjzvp\" (UID: \"cbdc46e1-4ffb-4a9d-9eef-6615a72fe025\") " pod="kserve-ci-e2e-test/success-200-isvc-889c1-predictor-746ccd4c5b-pjzvp"
May 25 13:14:30.774794 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:14:30.774617    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wd56d\" (UniqueName: \"kubernetes.io/projected/cbdc46e1-4ffb-4a9d-9eef-6615a72fe025-kube-api-access-wd56d\") pod \"success-200-isvc-889c1-predictor-746ccd4c5b-pjzvp\" (UID: \"cbdc46e1-4ffb-4a9d-9eef-6615a72fe025\") " pod="kserve-ci-e2e-test/success-200-isvc-889c1-predictor-746ccd4c5b-pjzvp"
May 25 13:14:30.774794 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:14:30.774697    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cbdc46e1-4ffb-4a9d-9eef-6615a72fe025-proxy-tls\") pod \"success-200-isvc-889c1-predictor-746ccd4c5b-pjzvp\" (UID: \"cbdc46e1-4ffb-4a9d-9eef-6615a72fe025\") " pod="kserve-ci-e2e-test/success-200-isvc-889c1-predictor-746ccd4c5b-pjzvp"
May 25 13:14:30.876126 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:14:30.876086    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"success-200-isvc-889c1-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cbdc46e1-4ffb-4a9d-9eef-6615a72fe025-success-200-isvc-889c1-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-889c1-predictor-746ccd4c5b-pjzvp\" (UID: \"cbdc46e1-4ffb-4a9d-9eef-6615a72fe025\") " pod="kserve-ci-e2e-test/success-200-isvc-889c1-predictor-746ccd4c5b-pjzvp"
May 25 13:14:30.876305 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:14:30.876150    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-wd56d\" (UniqueName: \"kubernetes.io/projected/cbdc46e1-4ffb-4a9d-9eef-6615a72fe025-kube-api-access-wd56d\") pod \"success-200-isvc-889c1-predictor-746ccd4c5b-pjzvp\" (UID: \"cbdc46e1-4ffb-4a9d-9eef-6615a72fe025\") " pod="kserve-ci-e2e-test/success-200-isvc-889c1-predictor-746ccd4c5b-pjzvp"
May 25 13:14:30.876305 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:14:30.876184    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cbdc46e1-4ffb-4a9d-9eef-6615a72fe025-proxy-tls\") pod \"success-200-isvc-889c1-predictor-746ccd4c5b-pjzvp\" (UID: \"cbdc46e1-4ffb-4a9d-9eef-6615a72fe025\") " pod="kserve-ci-e2e-test/success-200-isvc-889c1-predictor-746ccd4c5b-pjzvp"
May 25 13:14:30.876794 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:14:30.876769    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"success-200-isvc-889c1-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cbdc46e1-4ffb-4a9d-9eef-6615a72fe025-success-200-isvc-889c1-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-889c1-predictor-746ccd4c5b-pjzvp\" (UID: \"cbdc46e1-4ffb-4a9d-9eef-6615a72fe025\") " pod="kserve-ci-e2e-test/success-200-isvc-889c1-predictor-746ccd4c5b-pjzvp"
May 25 13:14:30.879359 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:14:30.879316    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cbdc46e1-4ffb-4a9d-9eef-6615a72fe025-proxy-tls\") pod \"success-200-isvc-889c1-predictor-746ccd4c5b-pjzvp\" (UID: \"cbdc46e1-4ffb-4a9d-9eef-6615a72fe025\") " pod="kserve-ci-e2e-test/success-200-isvc-889c1-predictor-746ccd4c5b-pjzvp"
May 25 13:14:30.888247 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:14:30.888217    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wd56d\" (UniqueName: \"kubernetes.io/projected/cbdc46e1-4ffb-4a9d-9eef-6615a72fe025-kube-api-access-wd56d\") pod \"success-200-isvc-889c1-predictor-746ccd4c5b-pjzvp\" (UID: \"cbdc46e1-4ffb-4a9d-9eef-6615a72fe025\") " pod="kserve-ci-e2e-test/success-200-isvc-889c1-predictor-746ccd4c5b-pjzvp"
May 25 13:14:30.964256 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:14:30.964162    2580 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-678784f46b-c6nfj"]
May 25 13:14:30.968133 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:14:30.968106    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-889c1-predictor-746ccd4c5b-pjzvp"
May 25 13:14:30.968247 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:14:30.968144    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-678784f46b-c6nfj"
May 25 13:14:30.972247 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:14:30.972209    2580 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-1-kube-rbac-proxy-sar-config\""
May 25 13:14:30.972247 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:14:30.972209    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-1-predictor-serving-cert\""
May 25 13:14:30.981942 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:14:30.981898    2580 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-678784f46b-c6nfj"]
May 25 13:14:31.078340 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:14:31.078308    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9e671986-36b9-4c8b-bc33-12613fa1f8d9-proxy-tls\") pod \"isvc-sklearn-graph-1-predictor-678784f46b-c6nfj\" (UID: \"9e671986-36b9-4c8b-bc33-12613fa1f8d9\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-678784f46b-c6nfj"
May 25 13:14:31.078531 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:14:31.078348    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9e671986-36b9-4c8b-bc33-12613fa1f8d9-kserve-provision-location\") pod \"isvc-sklearn-graph-1-predictor-678784f46b-c6nfj\" (UID: \"9e671986-36b9-4c8b-bc33-12613fa1f8d9\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-678784f46b-c6nfj"
May 25 13:14:31.078531 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:14:31.078431    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcpqq\" (UniqueName: \"kubernetes.io/projected/9e671986-36b9-4c8b-bc33-12613fa1f8d9-kube-api-access-lcpqq\") pod \"isvc-sklearn-graph-1-predictor-678784f46b-c6nfj\" (UID: \"9e671986-36b9-4c8b-bc33-12613fa1f8d9\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-678784f46b-c6nfj"
May 25 13:14:31.078531 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:14:31.078453    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-sklearn-graph-1-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9e671986-36b9-4c8b-bc33-12613fa1f8d9-isvc-sklearn-graph-1-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-1-predictor-678784f46b-c6nfj\" (UID: \"9e671986-36b9-4c8b-bc33-12613fa1f8d9\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-678784f46b-c6nfj"
May 25 13:14:31.110400 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:14:31.110355    2580 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-889c1-predictor-746ccd4c5b-pjzvp"]
May 25 13:14:31.113143 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:14:31.113109    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbdc46e1_4ffb_4a9d_9eef_6615a72fe025.slice/crio-7b90dfaf1b9546250b6bad749cd061cabaec70bf5a399c6d988bf5e9b4421d64 WatchSource:0}: Error finding container 7b90dfaf1b9546250b6bad749cd061cabaec70bf5a399c6d988bf5e9b4421d64: Status 404 returned error can't find the container with id 7b90dfaf1b9546250b6bad749cd061cabaec70bf5a399c6d988bf5e9b4421d64
May 25 13:14:31.179552 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:14:31.179510    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-lcpqq\" (UniqueName: \"kubernetes.io/projected/9e671986-36b9-4c8b-bc33-12613fa1f8d9-kube-api-access-lcpqq\") pod \"isvc-sklearn-graph-1-predictor-678784f46b-c6nfj\" (UID: \"9e671986-36b9-4c8b-bc33-12613fa1f8d9\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-678784f46b-c6nfj"
May 25 13:14:31.179552 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:14:31.179560    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-sklearn-graph-1-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9e671986-36b9-4c8b-bc33-12613fa1f8d9-isvc-sklearn-graph-1-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-1-predictor-678784f46b-c6nfj\" (UID: \"9e671986-36b9-4c8b-bc33-12613fa1f8d9\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-678784f46b-c6nfj"
May 25 13:14:31.179787 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:14:31.179656    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9e671986-36b9-4c8b-bc33-12613fa1f8d9-proxy-tls\") pod \"isvc-sklearn-graph-1-predictor-678784f46b-c6nfj\" (UID: \"9e671986-36b9-4c8b-bc33-12613fa1f8d9\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-678784f46b-c6nfj"
May 25 13:14:31.179787 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:14:31.179692    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9e671986-36b9-4c8b-bc33-12613fa1f8d9-kserve-provision-location\") pod \"isvc-sklearn-graph-1-predictor-678784f46b-c6nfj\" (UID: \"9e671986-36b9-4c8b-bc33-12613fa1f8d9\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-678784f46b-c6nfj"
May 25 13:14:31.180099 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:14:31.180073    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9e671986-36b9-4c8b-bc33-12613fa1f8d9-kserve-provision-location\") pod \"isvc-sklearn-graph-1-predictor-678784f46b-c6nfj\" (UID: \"9e671986-36b9-4c8b-bc33-12613fa1f8d9\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-678784f46b-c6nfj"
May 25 13:14:31.180316 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:14:31.180296    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-sklearn-graph-1-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9e671986-36b9-4c8b-bc33-12613fa1f8d9-isvc-sklearn-graph-1-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-1-predictor-678784f46b-c6nfj\" (UID: \"9e671986-36b9-4c8b-bc33-12613fa1f8d9\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-678784f46b-c6nfj"
May 25 13:14:31.182121 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:14:31.182103    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9e671986-36b9-4c8b-bc33-12613fa1f8d9-proxy-tls\") pod \"isvc-sklearn-graph-1-predictor-678784f46b-c6nfj\" (UID: \"9e671986-36b9-4c8b-bc33-12613fa1f8d9\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-678784f46b-c6nfj"
May 25 13:14:31.188582 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:14:31.188563    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcpqq\" (UniqueName: \"kubernetes.io/projected/9e671986-36b9-4c8b-bc33-12613fa1f8d9-kube-api-access-lcpqq\") pod \"isvc-sklearn-graph-1-predictor-678784f46b-c6nfj\" (UID: \"9e671986-36b9-4c8b-bc33-12613fa1f8d9\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-678784f46b-c6nfj"
May 25 13:14:31.300300 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:14:31.300203    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-678784f46b-c6nfj"
May 25 13:14:31.303740 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:14:31.303707    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-889c1-predictor-746ccd4c5b-pjzvp" event={"ID":"cbdc46e1-4ffb-4a9d-9eef-6615a72fe025","Type":"ContainerStarted","Data":"7b90dfaf1b9546250b6bad749cd061cabaec70bf5a399c6d988bf5e9b4421d64"}
May 25 13:14:31.436559 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:14:31.436392    2580 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-678784f46b-c6nfj"]
May 25 13:14:31.438536 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:14:31.438498    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e671986_36b9_4c8b_bc33_12613fa1f8d9.slice/crio-d5b1243b5ec5520268ae2d20ac283c26c5057fa726b6951496b2138b7e28a2d9 WatchSource:0}: Error finding container d5b1243b5ec5520268ae2d20ac283c26c5057fa726b6951496b2138b7e28a2d9: Status 404 returned error can't find the container with id d5b1243b5ec5520268ae2d20ac283c26c5057fa726b6951496b2138b7e28a2d9
May 25 13:14:31.912960 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:14:31.912919    2580 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-54c99bfc7d-sxdzl"]
May 25 13:14:31.917616 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:14:31.917588    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-54c99bfc7d-sxdzl"
May 25 13:14:31.920256 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:14:31.920223    2580 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-2-kube-rbac-proxy-sar-config\""
May 25 13:14:31.920808 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:14:31.920625    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-2-predictor-serving-cert\""
May 25 13:14:31.930014 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:14:31.929563    2580 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-54c99bfc7d-sxdzl"]
May 25 13:14:32.089593 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:14:32.089553    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/706b7f73-1a67-41ce-bd5e-0b14892b0acf-proxy-tls\") pod \"isvc-sklearn-graph-2-predictor-54c99bfc7d-sxdzl\" (UID: \"706b7f73-1a67-41ce-bd5e-0b14892b0acf\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-54c99bfc7d-sxdzl"
May 25 13:14:32.089791 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:14:32.089627    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8dq4\" (UniqueName: \"kubernetes.io/projected/706b7f73-1a67-41ce-bd5e-0b14892b0acf-kube-api-access-d8dq4\") pod \"isvc-sklearn-graph-2-predictor-54c99bfc7d-sxdzl\" (UID: \"706b7f73-1a67-41ce-bd5e-0b14892b0acf\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-54c99bfc7d-sxdzl"
May 25 13:14:32.089791 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:14:32.089697    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-sklearn-graph-2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/706b7f73-1a67-41ce-bd5e-0b14892b0acf-isvc-sklearn-graph-2-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-2-predictor-54c99bfc7d-sxdzl\" (UID: \"706b7f73-1a67-41ce-bd5e-0b14892b0acf\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-54c99bfc7d-sxdzl"
May 25 13:14:32.089902 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:14:32.089794    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/706b7f73-1a67-41ce-bd5e-0b14892b0acf-kserve-provision-location\") pod \"isvc-sklearn-graph-2-predictor-54c99bfc7d-sxdzl\" (UID: \"706b7f73-1a67-41ce-bd5e-0b14892b0acf\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-54c99bfc7d-sxdzl"
May 25 13:14:32.190880 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:14:32.190791    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/706b7f73-1a67-41ce-bd5e-0b14892b0acf-proxy-tls\") pod \"isvc-sklearn-graph-2-predictor-54c99bfc7d-sxdzl\" (UID: \"706b7f73-1a67-41ce-bd5e-0b14892b0acf\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-54c99bfc7d-sxdzl"
May 25 13:14:32.190880 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:14:32.190860    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-d8dq4\" (UniqueName: \"kubernetes.io/projected/706b7f73-1a67-41ce-bd5e-0b14892b0acf-kube-api-access-d8dq4\") pod \"isvc-sklearn-graph-2-predictor-54c99bfc7d-sxdzl\" (UID: \"706b7f73-1a67-41ce-bd5e-0b14892b0acf\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-54c99bfc7d-sxdzl"
May 25 13:14:32.191110 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:14:32.190913    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-sklearn-graph-2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/706b7f73-1a67-41ce-bd5e-0b14892b0acf-isvc-sklearn-graph-2-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-2-predictor-54c99bfc7d-sxdzl\" (UID: \"706b7f73-1a67-41ce-bd5e-0b14892b0acf\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-54c99bfc7d-sxdzl"
May 25 13:14:32.191110 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:14:32.190999    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/706b7f73-1a67-41ce-bd5e-0b14892b0acf-kserve-provision-location\") pod \"isvc-sklearn-graph-2-predictor-54c99bfc7d-sxdzl\" (UID: \"706b7f73-1a67-41ce-bd5e-0b14892b0acf\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-54c99bfc7d-sxdzl"
May 25 13:14:32.191565 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:14:32.191526    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/706b7f73-1a67-41ce-bd5e-0b14892b0acf-kserve-provision-location\") pod \"isvc-sklearn-graph-2-predictor-54c99bfc7d-sxdzl\" (UID: \"706b7f73-1a67-41ce-bd5e-0b14892b0acf\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-54c99bfc7d-sxdzl"
May 25 13:14:32.192882 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:14:32.192851    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-sklearn-graph-2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/706b7f73-1a67-41ce-bd5e-0b14892b0acf-isvc-sklearn-graph-2-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-2-predictor-54c99bfc7d-sxdzl\" (UID: \"706b7f73-1a67-41ce-bd5e-0b14892b0acf\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-54c99bfc7d-sxdzl"
May 25 13:14:32.201838 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:14:32.201807    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/706b7f73-1a67-41ce-bd5e-0b14892b0acf-proxy-tls\") pod \"isvc-sklearn-graph-2-predictor-54c99bfc7d-sxdzl\" (UID: \"706b7f73-1a67-41ce-bd5e-0b14892b0acf\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-54c99bfc7d-sxdzl"
May 25 13:14:32.211406 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:14:32.211361    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8dq4\" (UniqueName: \"kubernetes.io/projected/706b7f73-1a67-41ce-bd5e-0b14892b0acf-kube-api-access-d8dq4\") pod \"isvc-sklearn-graph-2-predictor-54c99bfc7d-sxdzl\" (UID: \"706b7f73-1a67-41ce-bd5e-0b14892b0acf\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-54c99bfc7d-sxdzl"
May 25 13:14:32.236168 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:14:32.236134    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-54c99bfc7d-sxdzl"
May 25 13:14:32.320517 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:14:32.320448    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-678784f46b-c6nfj" event={"ID":"9e671986-36b9-4c8b-bc33-12613fa1f8d9","Type":"ContainerStarted","Data":"d5b1243b5ec5520268ae2d20ac283c26c5057fa726b6951496b2138b7e28a2d9"}
May 25 13:14:32.467280 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:14:32.467198    2580 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-54c99bfc7d-sxdzl"]
May 25 13:14:33.252535 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:14:33.252491    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod706b7f73_1a67_41ce_bd5e_0b14892b0acf.slice/crio-0123e3e03fd06ee70e93cc63a822ef9a427a2d8b6d37f76e71045f0407874ea4 WatchSource:0}: Error finding container 0123e3e03fd06ee70e93cc63a822ef9a427a2d8b6d37f76e71045f0407874ea4: Status 404 returned error can't find the container with id 0123e3e03fd06ee70e93cc63a822ef9a427a2d8b6d37f76e71045f0407874ea4
May 25 13:14:33.328653 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:14:33.328611    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-54c99bfc7d-sxdzl" event={"ID":"706b7f73-1a67-41ce-bd5e-0b14892b0acf","Type":"ContainerStarted","Data":"0123e3e03fd06ee70e93cc63a822ef9a427a2d8b6d37f76e71045f0407874ea4"}
May 25 13:14:36.343245 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:14:36.343203    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-678784f46b-c6nfj" event={"ID":"9e671986-36b9-4c8b-bc33-12613fa1f8d9","Type":"ContainerStarted","Data":"b90b29224916cd27e49a5c3a3e6054726e7d624268e81b371840987b00d42de7"}
May 25 13:14:36.345470 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:14:36.345440    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-54c99bfc7d-sxdzl" event={"ID":"706b7f73-1a67-41ce-bd5e-0b14892b0acf","Type":"ContainerStarted","Data":"78084eb143e88c203496c56b586c6ae390eb2ee84c6682908aa8b6181c441727"}
May 25 13:14:40.363245 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:14:40.363207    2580 generic.go:358] "Generic (PLEG): container finished" podID="706b7f73-1a67-41ce-bd5e-0b14892b0acf" containerID="78084eb143e88c203496c56b586c6ae390eb2ee84c6682908aa8b6181c441727" exitCode=0
May 25 13:14:40.363728 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:14:40.363295    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-54c99bfc7d-sxdzl" event={"ID":"706b7f73-1a67-41ce-bd5e-0b14892b0acf","Type":"ContainerDied","Data":"78084eb143e88c203496c56b586c6ae390eb2ee84c6682908aa8b6181c441727"}
May 25 13:14:40.364825 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:14:40.364804    2580 generic.go:358] "Generic (PLEG): container finished" podID="9e671986-36b9-4c8b-bc33-12613fa1f8d9" containerID="b90b29224916cd27e49a5c3a3e6054726e7d624268e81b371840987b00d42de7" exitCode=0
May 25 13:14:40.364941 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:14:40.364885    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-678784f46b-c6nfj" event={"ID":"9e671986-36b9-4c8b-bc33-12613fa1f8d9","Type":"ContainerDied","Data":"b90b29224916cd27e49a5c3a3e6054726e7d624268e81b371840987b00d42de7"}
May 25 13:14:43.384416 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:14:43.384341    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-889c1-predictor-746ccd4c5b-pjzvp" event={"ID":"cbdc46e1-4ffb-4a9d-9eef-6615a72fe025","Type":"ContainerStarted","Data":"0d2d003722bd7769feb5cf3023f41a23b7dd5eb43c8d23e0e65203952ddcbd87"}
May 25 13:14:47.408345 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:14:47.408305    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-889c1-predictor-746ccd4c5b-pjzvp" event={"ID":"cbdc46e1-4ffb-4a9d-9eef-6615a72fe025","Type":"ContainerStarted","Data":"10aca39d22c4c3177bb6251ca5b1f6c3c0e06281f5825d9a554d00beb12b56cd"}
May 25 13:14:47.408824 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:14:47.408545    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-889c1-predictor-746ccd4c5b-pjzvp"
May 25 13:14:47.408824 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:14:47.408674    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-889c1-predictor-746ccd4c5b-pjzvp"
May 25 13:14:47.410130 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:14:47.410100    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-889c1-predictor-746ccd4c5b-pjzvp" podUID="cbdc46e1-4ffb-4a9d-9eef-6615a72fe025" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
May 25 13:14:47.430974 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:14:47.430920    2580 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-889c1-predictor-746ccd4c5b-pjzvp" podStartSLOduration=2.22631034 podStartE2EDuration="17.430906096s" podCreationTimestamp="2026-05-25 13:14:30 +0000 UTC" firstStartedPulling="2026-05-25 13:14:31.114955487 +0000 UTC m=+571.307249831" lastFinishedPulling="2026-05-25 13:14:46.319551222 +0000 UTC m=+586.511845587" observedRunningTime="2026-05-25 13:14:47.42853701 +0000 UTC m=+587.620831380" watchObservedRunningTime="2026-05-25 13:14:47.430906096 +0000 UTC m=+587.623200455"
May 25 13:14:48.412933 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:14:48.412891    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-889c1-predictor-746ccd4c5b-pjzvp" podUID="cbdc46e1-4ffb-4a9d-9eef-6615a72fe025" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
May 25 13:14:50.420998 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:14:50.420955    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-54c99bfc7d-sxdzl" event={"ID":"706b7f73-1a67-41ce-bd5e-0b14892b0acf","Type":"ContainerStarted","Data":"b66bae26ffb8583ab33b0fb5b593f219e483ab600deb3c51e290ec87deeed17a"}
May 25 13:14:50.421413 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:14:50.421008    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-54c99bfc7d-sxdzl" event={"ID":"706b7f73-1a67-41ce-bd5e-0b14892b0acf","Type":"ContainerStarted","Data":"0f4c99c800602aa490b987eb66918aa47e7e86b5d55319f988ba4a3ef771dfa0"}
May 25 13:14:50.421413 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:14:50.421240    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-54c99bfc7d-sxdzl"
May 25 13:14:50.422730 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:14:50.422708    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-678784f46b-c6nfj" event={"ID":"9e671986-36b9-4c8b-bc33-12613fa1f8d9","Type":"ContainerStarted","Data":"d1278c54cd2991e6beee74caeed79057016e4700aada3aca8b251f224a159ebb"}
May 25 13:14:50.422829 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:14:50.422740    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-678784f46b-c6nfj" event={"ID":"9e671986-36b9-4c8b-bc33-12613fa1f8d9","Type":"ContainerStarted","Data":"a7318035547c507d2ad424e17d9b606c2094a4c085e4015223e91792a0d3370b"}
May 25 13:14:50.423108 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:14:50.423079    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-678784f46b-c6nfj"
May 25 13:14:50.423144 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:14:50.423109    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-678784f46b-c6nfj"
May 25 13:14:50.424113 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:14:50.424090    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-678784f46b-c6nfj" podUID="9e671986-36b9-4c8b-bc33-12613fa1f8d9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
May 25 13:14:50.442027 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:14:50.441985    2580 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-54c99bfc7d-sxdzl" podStartSLOduration=2.8013016 podStartE2EDuration="19.441973687s" podCreationTimestamp="2026-05-25 13:14:31 +0000 UTC" firstStartedPulling="2026-05-25 13:14:33.255071075 +0000 UTC m=+573.447365429" lastFinishedPulling="2026-05-25 13:14:49.895743169 +0000 UTC m=+590.088037516" observedRunningTime="2026-05-25 13:14:50.441557008 +0000 UTC m=+590.633851370" watchObservedRunningTime="2026-05-25 13:14:50.441973687 +0000 UTC m=+590.634268047"
May 25 13:14:50.462605 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:14:50.462562    2580 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-678784f46b-c6nfj" podStartSLOduration=1.99026334 podStartE2EDuration="20.462549739s" podCreationTimestamp="2026-05-25 13:14:30 +0000 UTC" firstStartedPulling="2026-05-25 13:14:31.440632172 +0000 UTC m=+571.632926510" lastFinishedPulling="2026-05-25 13:14:49.912918568 +0000 UTC m=+590.105212909" observedRunningTime="2026-05-25 13:14:50.460308473 +0000 UTC m=+590.652602833" watchObservedRunningTime="2026-05-25 13:14:50.462549739 +0000 UTC m=+590.654844100"
May 25 13:14:51.426068 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:14:51.426015    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-678784f46b-c6nfj" podUID="9e671986-36b9-4c8b-bc33-12613fa1f8d9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
May 25 13:14:51.426510 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:14:51.426188    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-54c99bfc7d-sxdzl"
May 25 13:14:51.427072 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:14:51.427048    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-54c99bfc7d-sxdzl" podUID="706b7f73-1a67-41ce-bd5e-0b14892b0acf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
May 25 13:14:52.429827 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:14:52.429790    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-54c99bfc7d-sxdzl" podUID="706b7f73-1a67-41ce-bd5e-0b14892b0acf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
May 25 13:14:53.417096 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:14:53.417065    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-889c1-predictor-746ccd4c5b-pjzvp"
May 25 13:14:53.417715 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:14:53.417688    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-889c1-predictor-746ccd4c5b-pjzvp" podUID="cbdc46e1-4ffb-4a9d-9eef-6615a72fe025" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
May 25 13:14:56.430667 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:14:56.430635    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-678784f46b-c6nfj"
May 25 13:14:56.431275 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:14:56.431248    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-678784f46b-c6nfj" podUID="9e671986-36b9-4c8b-bc33-12613fa1f8d9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
May 25 13:14:57.434319 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:14:57.434290    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-54c99bfc7d-sxdzl"
May 25 13:14:57.434949 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:14:57.434922    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-54c99bfc7d-sxdzl" podUID="706b7f73-1a67-41ce-bd5e-0b14892b0acf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
May 25 13:15:00.286020 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:15:00.285993    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-646r6_59b905b8-5ce3-49f6-b47c-b5b3a525fe06/ovn-acl-logging/0.log"
May 25 13:15:00.286500 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:15:00.286031    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-646r6_59b905b8-5ce3-49f6-b47c-b5b3a525fe06/ovn-acl-logging/0.log"
May 25 13:15:03.417950 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:15:03.417910    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-889c1-predictor-746ccd4c5b-pjzvp" podUID="cbdc46e1-4ffb-4a9d-9eef-6615a72fe025" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
May 25 13:15:06.431809 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:15:06.431765    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-678784f46b-c6nfj" podUID="9e671986-36b9-4c8b-bc33-12613fa1f8d9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
May 25 13:15:07.435909 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:15:07.435866    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-54c99bfc7d-sxdzl" podUID="706b7f73-1a67-41ce-bd5e-0b14892b0acf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
May 25 13:15:13.418719 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:15:13.418624    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-889c1-predictor-746ccd4c5b-pjzvp" podUID="cbdc46e1-4ffb-4a9d-9eef-6615a72fe025" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
May 25 13:15:16.431785 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:15:16.431737    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-678784f46b-c6nfj" podUID="9e671986-36b9-4c8b-bc33-12613fa1f8d9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
May 25 13:15:17.435704 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:15:17.435656    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-54c99bfc7d-sxdzl" podUID="706b7f73-1a67-41ce-bd5e-0b14892b0acf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
May 25 13:15:23.418618 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:15:23.418577    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-889c1-predictor-746ccd4c5b-pjzvp" podUID="cbdc46e1-4ffb-4a9d-9eef-6615a72fe025" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
May 25 13:15:26.432108 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:15:26.432064    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-678784f46b-c6nfj" podUID="9e671986-36b9-4c8b-bc33-12613fa1f8d9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
May 25 13:15:27.435415 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:15:27.435358    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-54c99bfc7d-sxdzl" podUID="706b7f73-1a67-41ce-bd5e-0b14892b0acf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
May 25 13:15:33.418522 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:15:33.418489    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-889c1-predictor-746ccd4c5b-pjzvp"
May 25 13:15:36.431649 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:15:36.431608    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-678784f46b-c6nfj" podUID="9e671986-36b9-4c8b-bc33-12613fa1f8d9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
May 25 13:15:37.435532 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:15:37.435486    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-54c99bfc7d-sxdzl" podUID="706b7f73-1a67-41ce-bd5e-0b14892b0acf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
May 25 13:15:46.431604 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:15:46.431562    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-678784f46b-c6nfj" podUID="9e671986-36b9-4c8b-bc33-12613fa1f8d9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
May 25 13:15:47.434876 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:15:47.434837    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-54c99bfc7d-sxdzl" podUID="706b7f73-1a67-41ce-bd5e-0b14892b0acf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
May 25 13:15:56.432202 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:15:56.432171    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-678784f46b-c6nfj"
May 25 13:15:57.439577 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:15:57.439541    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-54c99bfc7d-sxdzl"
May 25 13:16:00.958517 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:16:00.958482    2580 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-889c1-predictor-746ccd4c5b-pjzvp"]
May 25 13:16:00.958919 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:16:00.958796    2580 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-889c1-predictor-746ccd4c5b-pjzvp" podUID="cbdc46e1-4ffb-4a9d-9eef-6615a72fe025" containerName="kserve-container" containerID="cri-o://0d2d003722bd7769feb5cf3023f41a23b7dd5eb43c8d23e0e65203952ddcbd87" gracePeriod=30
May 25 13:16:00.958919 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:16:00.958845    2580 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-889c1-predictor-746ccd4c5b-pjzvp" podUID="cbdc46e1-4ffb-4a9d-9eef-6615a72fe025" containerName="kube-rbac-proxy" containerID="cri-o://10aca39d22c4c3177bb6251ca5b1f6c3c0e06281f5825d9a554d00beb12b56cd" gracePeriod=30
May 25 13:16:00.998550 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:16:00.998525    2580 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-339a4-predictor-58f8586498-9r8dr"]
May 25 13:16:01.002162 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:16:01.002143    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-339a4-predictor-58f8586498-9r8dr"
May 25 13:16:01.004903 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:16:01.004882    2580 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-339a4-kube-rbac-proxy-sar-config\""
May 25 13:16:01.004903 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:16:01.004894    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-339a4-predictor-serving-cert\""
May 25 13:16:01.013605 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:16:01.013581    2580 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-339a4-predictor-58f8586498-9r8dr"]
May 25 13:16:01.080997 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:16:01.080969    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a3abc8ef-5a38-4ebc-bdbb-1c16bf6e82a0-proxy-tls\") pod \"success-200-isvc-339a4-predictor-58f8586498-9r8dr\" (UID: \"a3abc8ef-5a38-4ebc-bdbb-1c16bf6e82a0\") " pod="kserve-ci-e2e-test/success-200-isvc-339a4-predictor-58f8586498-9r8dr"
May 25 13:16:01.081140 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:16:01.081011    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"success-200-isvc-339a4-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a3abc8ef-5a38-4ebc-bdbb-1c16bf6e82a0-success-200-isvc-339a4-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-339a4-predictor-58f8586498-9r8dr\" (UID: \"a3abc8ef-5a38-4ebc-bdbb-1c16bf6e82a0\") " pod="kserve-ci-e2e-test/success-200-isvc-339a4-predictor-58f8586498-9r8dr"
May 25 13:16:01.081181 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:16:01.081144    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvk2z\" (UniqueName: \"kubernetes.io/projected/a3abc8ef-5a38-4ebc-bdbb-1c16bf6e82a0-kube-api-access-lvk2z\") pod \"success-200-isvc-339a4-predictor-58f8586498-9r8dr\" (UID: \"a3abc8ef-5a38-4ebc-bdbb-1c16bf6e82a0\") " pod="kserve-ci-e2e-test/success-200-isvc-339a4-predictor-58f8586498-9r8dr"
May 25 13:16:01.182146 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:16:01.182116    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-lvk2z\" (UniqueName: \"kubernetes.io/projected/a3abc8ef-5a38-4ebc-bdbb-1c16bf6e82a0-kube-api-access-lvk2z\") pod \"success-200-isvc-339a4-predictor-58f8586498-9r8dr\" (UID: \"a3abc8ef-5a38-4ebc-bdbb-1c16bf6e82a0\") " pod="kserve-ci-e2e-test/success-200-isvc-339a4-predictor-58f8586498-9r8dr"
May 25 13:16:01.182301 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:16:01.182179    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a3abc8ef-5a38-4ebc-bdbb-1c16bf6e82a0-proxy-tls\") pod \"success-200-isvc-339a4-predictor-58f8586498-9r8dr\" (UID: \"a3abc8ef-5a38-4ebc-bdbb-1c16bf6e82a0\") " pod="kserve-ci-e2e-test/success-200-isvc-339a4-predictor-58f8586498-9r8dr"
May 25 13:16:01.182301 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:16:01.182218    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"success-200-isvc-339a4-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a3abc8ef-5a38-4ebc-bdbb-1c16bf6e82a0-success-200-isvc-339a4-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-339a4-predictor-58f8586498-9r8dr\" (UID: \"a3abc8ef-5a38-4ebc-bdbb-1c16bf6e82a0\") " pod="kserve-ci-e2e-test/success-200-isvc-339a4-predictor-58f8586498-9r8dr"
May 25 13:16:01.182947 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:16:01.182925    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"success-200-isvc-339a4-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a3abc8ef-5a38-4ebc-bdbb-1c16bf6e82a0-success-200-isvc-339a4-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-339a4-predictor-58f8586498-9r8dr\" (UID: \"a3abc8ef-5a38-4ebc-bdbb-1c16bf6e82a0\") " pod="kserve-ci-e2e-test/success-200-isvc-339a4-predictor-58f8586498-9r8dr"
May 25 13:16:01.184826 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:16:01.184805    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a3abc8ef-5a38-4ebc-bdbb-1c16bf6e82a0-proxy-tls\") pod \"success-200-isvc-339a4-predictor-58f8586498-9r8dr\" (UID: \"a3abc8ef-5a38-4ebc-bdbb-1c16bf6e82a0\") " pod="kserve-ci-e2e-test/success-200-isvc-339a4-predictor-58f8586498-9r8dr"
May 25 13:16:01.190967 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:16:01.190940    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvk2z\" (UniqueName: \"kubernetes.io/projected/a3abc8ef-5a38-4ebc-bdbb-1c16bf6e82a0-kube-api-access-lvk2z\") pod \"success-200-isvc-339a4-predictor-58f8586498-9r8dr\" (UID: \"a3abc8ef-5a38-4ebc-bdbb-1c16bf6e82a0\") " pod="kserve-ci-e2e-test/success-200-isvc-339a4-predictor-58f8586498-9r8dr"
May 25 13:16:01.312969 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:16:01.312884    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-339a4-predictor-58f8586498-9r8dr"
May 25 13:16:01.445469 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:16:01.445440    2580 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-339a4-predictor-58f8586498-9r8dr"]
May 25 13:16:01.448164 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:16:01.448138    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3abc8ef_5a38_4ebc_bdbb_1c16bf6e82a0.slice/crio-b4a4797a3e638be1662750d1683bc3f6e9eff9003f061e84cc75250165221083 WatchSource:0}: Error finding container b4a4797a3e638be1662750d1683bc3f6e9eff9003f061e84cc75250165221083: Status 404 returned error can't find the container with id b4a4797a3e638be1662750d1683bc3f6e9eff9003f061e84cc75250165221083
May 25 13:16:01.449932 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:16:01.449914    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 25 13:16:01.668772 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:16:01.668734    2580 generic.go:358] "Generic (PLEG): container finished" podID="cbdc46e1-4ffb-4a9d-9eef-6615a72fe025" containerID="10aca39d22c4c3177bb6251ca5b1f6c3c0e06281f5825d9a554d00beb12b56cd" exitCode=2
May 25 13:16:01.668936 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:16:01.668808    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-889c1-predictor-746ccd4c5b-pjzvp" event={"ID":"cbdc46e1-4ffb-4a9d-9eef-6615a72fe025","Type":"ContainerDied","Data":"10aca39d22c4c3177bb6251ca5b1f6c3c0e06281f5825d9a554d00beb12b56cd"}
May 25 13:16:01.670221 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:16:01.670197    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-339a4-predictor-58f8586498-9r8dr" event={"ID":"a3abc8ef-5a38-4ebc-bdbb-1c16bf6e82a0","Type":"ContainerStarted","Data":"91c99e5baa6d57eb5946b6b581dead9287c07d822087c96935bff84ddcefd5e9"}
May 25 13:16:01.670331 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:16:01.670226    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-339a4-predictor-58f8586498-9r8dr" event={"ID":"a3abc8ef-5a38-4ebc-bdbb-1c16bf6e82a0","Type":"ContainerStarted","Data":"c9cca1805d992e4be768f3d48a5d59173efd6ac11a9318e622bbc078141b4395"}
May 25 13:16:01.670331 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:16:01.670236    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-339a4-predictor-58f8586498-9r8dr" event={"ID":"a3abc8ef-5a38-4ebc-bdbb-1c16bf6e82a0","Type":"ContainerStarted","Data":"b4a4797a3e638be1662750d1683bc3f6e9eff9003f061e84cc75250165221083"}
May 25 13:16:01.670331 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:16:01.670302    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-339a4-predictor-58f8586498-9r8dr"
May 25 13:16:01.688356 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:16:01.688293    2580 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-339a4-predictor-58f8586498-9r8dr" podStartSLOduration=1.688255826 podStartE2EDuration="1.688255826s" podCreationTimestamp="2026-05-25 13:16:00 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 13:16:01.686436978 +0000 UTC m=+661.878731339" watchObservedRunningTime="2026-05-25 13:16:01.688255826 +0000 UTC m=+661.880550183"
May 25 13:16:02.673621 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:16:02.673592    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-339a4-predictor-58f8586498-9r8dr"
May 25 13:16:02.674962 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:16:02.674922    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-339a4-predictor-58f8586498-9r8dr" podUID="a3abc8ef-5a38-4ebc-bdbb-1c16bf6e82a0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
May 25 13:16:03.413836 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:16:03.413782    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-889c1-predictor-746ccd4c5b-pjzvp" podUID="cbdc46e1-4ffb-4a9d-9eef-6615a72fe025" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.32:8643/healthz\": dial tcp 10.134.0.32:8643: connect: connection refused"
May 25 13:16:03.417935 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:16:03.417899    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-889c1-predictor-746ccd4c5b-pjzvp" podUID="cbdc46e1-4ffb-4a9d-9eef-6615a72fe025" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
May 25 13:16:03.676711 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:16:03.676616    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-339a4-predictor-58f8586498-9r8dr" podUID="a3abc8ef-5a38-4ebc-bdbb-1c16bf6e82a0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
May 25 13:16:04.111213 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:16:04.111188    2580 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-889c1-predictor-746ccd4c5b-pjzvp"
May 25 13:16:04.207697 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:16:04.207659    2580 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"success-200-isvc-889c1-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cbdc46e1-4ffb-4a9d-9eef-6615a72fe025-success-200-isvc-889c1-kube-rbac-proxy-sar-config\") pod \"cbdc46e1-4ffb-4a9d-9eef-6615a72fe025\" (UID: \"cbdc46e1-4ffb-4a9d-9eef-6615a72fe025\") "
May 25 13:16:04.207884 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:16:04.207732    2580 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wd56d\" (UniqueName: \"kubernetes.io/projected/cbdc46e1-4ffb-4a9d-9eef-6615a72fe025-kube-api-access-wd56d\") pod \"cbdc46e1-4ffb-4a9d-9eef-6615a72fe025\" (UID: \"cbdc46e1-4ffb-4a9d-9eef-6615a72fe025\") "
May 25 13:16:04.207884 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:16:04.207797    2580 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cbdc46e1-4ffb-4a9d-9eef-6615a72fe025-proxy-tls\") pod \"cbdc46e1-4ffb-4a9d-9eef-6615a72fe025\" (UID: \"cbdc46e1-4ffb-4a9d-9eef-6615a72fe025\") "
May 25 13:16:04.208035 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:16:04.208010    2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbdc46e1-4ffb-4a9d-9eef-6615a72fe025-success-200-isvc-889c1-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "success-200-isvc-889c1-kube-rbac-proxy-sar-config") pod "cbdc46e1-4ffb-4a9d-9eef-6615a72fe025" (UID: "cbdc46e1-4ffb-4a9d-9eef-6615a72fe025"). InnerVolumeSpecName "success-200-isvc-889c1-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 13:16:04.210024 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:16:04.209992    2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cbdc46e1-4ffb-4a9d-9eef-6615a72fe025-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "cbdc46e1-4ffb-4a9d-9eef-6615a72fe025" (UID: "cbdc46e1-4ffb-4a9d-9eef-6615a72fe025"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 13:16:04.210139 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:16:04.210084    2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbdc46e1-4ffb-4a9d-9eef-6615a72fe025-kube-api-access-wd56d" (OuterVolumeSpecName: "kube-api-access-wd56d") pod "cbdc46e1-4ffb-4a9d-9eef-6615a72fe025" (UID: "cbdc46e1-4ffb-4a9d-9eef-6615a72fe025"). InnerVolumeSpecName "kube-api-access-wd56d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 13:16:04.309335 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:16:04.309234    2580 reconciler_common.go:299] "Volume detached for volume \"success-200-isvc-889c1-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cbdc46e1-4ffb-4a9d-9eef-6615a72fe025-success-200-isvc-889c1-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-103.ec2.internal\" DevicePath \"\""
May 25 13:16:04.309335 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:16:04.309273    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wd56d\" (UniqueName: \"kubernetes.io/projected/cbdc46e1-4ffb-4a9d-9eef-6615a72fe025-kube-api-access-wd56d\") on node \"ip-10-0-135-103.ec2.internal\" DevicePath \"\""
May 25 13:16:04.309335 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:16:04.309286    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cbdc46e1-4ffb-4a9d-9eef-6615a72fe025-proxy-tls\") on node \"ip-10-0-135-103.ec2.internal\" DevicePath \"\""
May 25 13:16:04.680748 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:16:04.680714    2580 generic.go:358] "Generic (PLEG): container finished" podID="cbdc46e1-4ffb-4a9d-9eef-6615a72fe025" containerID="0d2d003722bd7769feb5cf3023f41a23b7dd5eb43c8d23e0e65203952ddcbd87" exitCode=0
May 25 13:16:04.681181 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:16:04.680761    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-889c1-predictor-746ccd4c5b-pjzvp" event={"ID":"cbdc46e1-4ffb-4a9d-9eef-6615a72fe025","Type":"ContainerDied","Data":"0d2d003722bd7769feb5cf3023f41a23b7dd5eb43c8d23e0e65203952ddcbd87"}
May 25 13:16:04.681181 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:16:04.680784    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-889c1-predictor-746ccd4c5b-pjzvp" event={"ID":"cbdc46e1-4ffb-4a9d-9eef-6615a72fe025","Type":"ContainerDied","Data":"7b90dfaf1b9546250b6bad749cd061cabaec70bf5a399c6d988bf5e9b4421d64"}
May 25 13:16:04.681181 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:16:04.680800    2580 scope.go:117] "RemoveContainer" containerID="10aca39d22c4c3177bb6251ca5b1f6c3c0e06281f5825d9a554d00beb12b56cd"
May 25 13:16:04.681181 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:16:04.680797    2580 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-889c1-predictor-746ccd4c5b-pjzvp"
May 25 13:16:04.690002 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:16:04.689983    2580 scope.go:117] "RemoveContainer" containerID="0d2d003722bd7769feb5cf3023f41a23b7dd5eb43c8d23e0e65203952ddcbd87"
May 25 13:16:04.697112 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:16:04.697084    2580 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-889c1-predictor-746ccd4c5b-pjzvp"]
May 25 13:16:04.698385 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:16:04.698329    2580 scope.go:117] "RemoveContainer" containerID="10aca39d22c4c3177bb6251ca5b1f6c3c0e06281f5825d9a554d00beb12b56cd"
May 25 13:16:04.698953 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:16:04.698916    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10aca39d22c4c3177bb6251ca5b1f6c3c0e06281f5825d9a554d00beb12b56cd\": container with ID starting with 10aca39d22c4c3177bb6251ca5b1f6c3c0e06281f5825d9a554d00beb12b56cd not found: ID does not exist" containerID="10aca39d22c4c3177bb6251ca5b1f6c3c0e06281f5825d9a554d00beb12b56cd"
May 25 13:16:04.699058 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:16:04.698958    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10aca39d22c4c3177bb6251ca5b1f6c3c0e06281f5825d9a554d00beb12b56cd"} err="failed to get container status \"10aca39d22c4c3177bb6251ca5b1f6c3c0e06281f5825d9a554d00beb12b56cd\": rpc error: code = NotFound desc = could not find container \"10aca39d22c4c3177bb6251ca5b1f6c3c0e06281f5825d9a554d00beb12b56cd\": container with ID starting with 10aca39d22c4c3177bb6251ca5b1f6c3c0e06281f5825d9a554d00beb12b56cd not found: ID does not exist"
May 25 13:16:04.699058 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:16:04.698984    2580 scope.go:117] "RemoveContainer" containerID="0d2d003722bd7769feb5cf3023f41a23b7dd5eb43c8d23e0e65203952ddcbd87"
May 25 13:16:04.699520 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:16:04.699497    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d2d003722bd7769feb5cf3023f41a23b7dd5eb43c8d23e0e65203952ddcbd87\": container with ID starting with 0d2d003722bd7769feb5cf3023f41a23b7dd5eb43c8d23e0e65203952ddcbd87 not found: ID does not exist" containerID="0d2d003722bd7769feb5cf3023f41a23b7dd5eb43c8d23e0e65203952ddcbd87"
May 25 13:16:04.699610 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:16:04.699526    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d2d003722bd7769feb5cf3023f41a23b7dd5eb43c8d23e0e65203952ddcbd87"} err="failed to get container status \"0d2d003722bd7769feb5cf3023f41a23b7dd5eb43c8d23e0e65203952ddcbd87\": rpc error: code = NotFound desc = could not find container \"0d2d003722bd7769feb5cf3023f41a23b7dd5eb43c8d23e0e65203952ddcbd87\": container with ID starting with 0d2d003722bd7769feb5cf3023f41a23b7dd5eb43c8d23e0e65203952ddcbd87 not found: ID does not exist"
May 25 13:16:04.700534 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:16:04.700513    2580 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-889c1-predictor-746ccd4c5b-pjzvp"]
May 25 13:16:06.377621 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:16:06.377584    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cbdc46e1-4ffb-4a9d-9eef-6615a72fe025" path="/var/lib/kubelet/pods/cbdc46e1-4ffb-4a9d-9eef-6615a72fe025/volumes"
May 25 13:16:08.681543 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:16:08.681509    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-339a4-predictor-58f8586498-9r8dr"
May 25 13:16:08.682143 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:16:08.682111    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-339a4-predictor-58f8586498-9r8dr" podUID="a3abc8ef-5a38-4ebc-bdbb-1c16bf6e82a0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
May 25 13:16:18.683026 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:16:18.682985    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-339a4-predictor-58f8586498-9r8dr" podUID="a3abc8ef-5a38-4ebc-bdbb-1c16bf6e82a0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
May 25 13:16:28.682258 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:16:28.682214    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-339a4-predictor-58f8586498-9r8dr" podUID="a3abc8ef-5a38-4ebc-bdbb-1c16bf6e82a0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
May 25 13:16:30.975205 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:16:30.975164    2580 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-54c99bfc7d-sxdzl"]
May 25 13:16:30.975593 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:16:30.975485    2580 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-54c99bfc7d-sxdzl" podUID="706b7f73-1a67-41ce-bd5e-0b14892b0acf" containerName="kserve-container" containerID="cri-o://0f4c99c800602aa490b987eb66918aa47e7e86b5d55319f988ba4a3ef771dfa0" gracePeriod=30
May 25 13:16:30.975593 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:16:30.975544    2580 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-54c99bfc7d-sxdzl" podUID="706b7f73-1a67-41ce-bd5e-0b14892b0acf" containerName="kube-rbac-proxy" containerID="cri-o://b66bae26ffb8583ab33b0fb5b593f219e483ab600deb3c51e290ec87deeed17a" gracePeriod=30
May 25 13:16:31.017303 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:16:31.017268    2580 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-08c99-predictor-cc5d7f476-czxk9"]
May 25 13:16:31.017884 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:16:31.017861    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cbdc46e1-4ffb-4a9d-9eef-6615a72fe025" containerName="kube-rbac-proxy"
May 25 13:16:31.017884 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:16:31.017884    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="cbdc46e1-4ffb-4a9d-9eef-6615a72fe025" containerName="kube-rbac-proxy"
May 25 13:16:31.018042 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:16:31.017903    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cbdc46e1-4ffb-4a9d-9eef-6615a72fe025" containerName="kserve-container"
May 25 13:16:31.018042 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:16:31.017912    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="cbdc46e1-4ffb-4a9d-9eef-6615a72fe025" containerName="kserve-container"
May 25 13:16:31.018042 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:16:31.018012    2580 memory_manager.go:356] "RemoveStaleState removing state" podUID="cbdc46e1-4ffb-4a9d-9eef-6615a72fe025" containerName="kube-rbac-proxy"
May 25 13:16:31.018042 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:16:31.018025    2580 memory_manager.go:356] "RemoveStaleState removing state" podUID="cbdc46e1-4ffb-4a9d-9eef-6615a72fe025" containerName="kserve-container"
May 25 13:16:31.022674 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:16:31.022647    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-08c99-predictor-cc5d7f476-czxk9"
May 25 13:16:31.026703 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:16:31.026676    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-08c99-predictor-serving-cert\""
May 25 13:16:31.026703 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:16:31.026690    2580 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-08c99-kube-rbac-proxy-sar-config\""
May 25 13:16:31.035415 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:16:31.035394    2580 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-08c99-predictor-cc5d7f476-czxk9"]
May 25 13:16:31.035884 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:16:31.035866    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/11a87d3c-4909-445a-ab48-fb162c262188-proxy-tls\") pod \"success-200-isvc-08c99-predictor-cc5d7f476-czxk9\" (UID: \"11a87d3c-4909-445a-ab48-fb162c262188\") " pod="kserve-ci-e2e-test/success-200-isvc-08c99-predictor-cc5d7f476-czxk9"
May 25 13:16:31.035932 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:16:31.035899    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q65s7\" (UniqueName: \"kubernetes.io/projected/11a87d3c-4909-445a-ab48-fb162c262188-kube-api-access-q65s7\") pod \"success-200-isvc-08c99-predictor-cc5d7f476-czxk9\" (UID: \"11a87d3c-4909-445a-ab48-fb162c262188\") " pod="kserve-ci-e2e-test/success-200-isvc-08c99-predictor-cc5d7f476-czxk9"
May 25 13:16:31.035974 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:16:31.035945    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"success-200-isvc-08c99-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/11a87d3c-4909-445a-ab48-fb162c262188-success-200-isvc-08c99-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-08c99-predictor-cc5d7f476-czxk9\" (UID: \"11a87d3c-4909-445a-ab48-fb162c262188\") " pod="kserve-ci-e2e-test/success-200-isvc-08c99-predictor-cc5d7f476-czxk9"
May 25 13:16:31.052379 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:16:31.052341    2580 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-678784f46b-c6nfj"]
May 25 13:16:31.052716 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:16:31.052677    2580 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-678784f46b-c6nfj" podUID="9e671986-36b9-4c8b-bc33-12613fa1f8d9" containerName="kserve-container" containerID="cri-o://a7318035547c507d2ad424e17d9b606c2094a4c085e4015223e91792a0d3370b" gracePeriod=30
May 25 13:16:31.052812 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:16:31.052696    2580 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-678784f46b-c6nfj" podUID="9e671986-36b9-4c8b-bc33-12613fa1f8d9" containerName="kube-rbac-proxy" containerID="cri-o://d1278c54cd2991e6beee74caeed79057016e4700aada3aca8b251f224a159ebb" gracePeriod=30
May 25 13:16:31.137275 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:16:31.137238    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/11a87d3c-4909-445a-ab48-fb162c262188-proxy-tls\") pod \"success-200-isvc-08c99-predictor-cc5d7f476-czxk9\" (UID: \"11a87d3c-4909-445a-ab48-fb162c262188\") " pod="kserve-ci-e2e-test/success-200-isvc-08c99-predictor-cc5d7f476-czxk9"
May 25 13:16:31.137275 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:16:31.137280    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-q65s7\" (UniqueName: \"kubernetes.io/projected/11a87d3c-4909-445a-ab48-fb162c262188-kube-api-access-q65s7\") pod \"success-200-isvc-08c99-predictor-cc5d7f476-czxk9\" (UID: \"11a87d3c-4909-445a-ab48-fb162c262188\") " pod="kserve-ci-e2e-test/success-200-isvc-08c99-predictor-cc5d7f476-czxk9"
May 25 13:16:31.137527 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:16:31.137312    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"success-200-isvc-08c99-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/11a87d3c-4909-445a-ab48-fb162c262188-success-200-isvc-08c99-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-08c99-predictor-cc5d7f476-czxk9\" (UID: \"11a87d3c-4909-445a-ab48-fb162c262188\") " pod="kserve-ci-e2e-test/success-200-isvc-08c99-predictor-cc5d7f476-czxk9"
May 25 13:16:31.137996 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:16:31.137939    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"success-200-isvc-08c99-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/11a87d3c-4909-445a-ab48-fb162c262188-success-200-isvc-08c99-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-08c99-predictor-cc5d7f476-czxk9\" (UID: \"11a87d3c-4909-445a-ab48-fb162c262188\") " pod="kserve-ci-e2e-test/success-200-isvc-08c99-predictor-cc5d7f476-czxk9"
May 25 13:16:31.139903 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:16:31.139880    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/11a87d3c-4909-445a-ab48-fb162c262188-proxy-tls\") pod \"success-200-isvc-08c99-predictor-cc5d7f476-czxk9\" (UID: \"11a87d3c-4909-445a-ab48-fb162c262188\") " pod="kserve-ci-e2e-test/success-200-isvc-08c99-predictor-cc5d7f476-czxk9"
May 25 13:16:31.149907 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:16:31.149873    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q65s7\" (UniqueName: \"kubernetes.io/projected/11a87d3c-4909-445a-ab48-fb162c262188-kube-api-access-q65s7\") pod \"success-200-isvc-08c99-predictor-cc5d7f476-czxk9\" (UID: \"11a87d3c-4909-445a-ab48-fb162c262188\") " pod="kserve-ci-e2e-test/success-200-isvc-08c99-predictor-cc5d7f476-czxk9"
May 25 13:16:31.334239 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:16:31.334208    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-08c99-predictor-cc5d7f476-czxk9"
May 25 13:16:31.427076 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:16:31.427025    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-678784f46b-c6nfj" podUID="9e671986-36b9-4c8b-bc33-12613fa1f8d9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.33:8643/healthz\": dial tcp 10.134.0.33:8643: connect: connection refused"
May 25 13:16:31.471309 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:16:31.471219    2580 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-08c99-predictor-cc5d7f476-czxk9"]
May 25 13:16:31.473504 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:16:31.473475    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11a87d3c_4909_445a_ab48_fb162c262188.slice/crio-f233fe73c0486972d4ae1f02dfb6968bfae1a29f531c8d82d2bd5ee8cb88548f WatchSource:0}: Error finding container f233fe73c0486972d4ae1f02dfb6968bfae1a29f531c8d82d2bd5ee8cb88548f: Status 404 returned error can't find the container with id f233fe73c0486972d4ae1f02dfb6968bfae1a29f531c8d82d2bd5ee8cb88548f
May 25 13:16:31.779105 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:16:31.779074    2580 generic.go:358] "Generic (PLEG): container finished" podID="706b7f73-1a67-41ce-bd5e-0b14892b0acf" containerID="b66bae26ffb8583ab33b0fb5b593f219e483ab600deb3c51e290ec87deeed17a" exitCode=2
May 25 13:16:31.779298 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:16:31.779137    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-54c99bfc7d-sxdzl" event={"ID":"706b7f73-1a67-41ce-bd5e-0b14892b0acf","Type":"ContainerDied","Data":"b66bae26ffb8583ab33b0fb5b593f219e483ab600deb3c51e290ec87deeed17a"}
May 25 13:16:31.780601 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:16:31.780570    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-08c99-predictor-cc5d7f476-czxk9" event={"ID":"11a87d3c-4909-445a-ab48-fb162c262188","Type":"ContainerStarted","Data":"2802e6e9aa0df5e48b1cb24637010f27e6243465f626fe5d1379fcf612ca6aa3"}
May 25 13:16:31.780711 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:16:31.780611    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-08c99-predictor-cc5d7f476-czxk9" event={"ID":"11a87d3c-4909-445a-ab48-fb162c262188","Type":"ContainerStarted","Data":"a2ea57fe73b1bcf01a93b0508130f2f8617c35f48de87203553d255a6c0942a9"}
May 25 13:16:31.780711 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:16:31.780623    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-08c99-predictor-cc5d7f476-czxk9" event={"ID":"11a87d3c-4909-445a-ab48-fb162c262188","Type":"ContainerStarted","Data":"f233fe73c0486972d4ae1f02dfb6968bfae1a29f531c8d82d2bd5ee8cb88548f"}
May 25 13:16:31.780828 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:16:31.780787    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-08c99-predictor-cc5d7f476-czxk9"
May 25 13:16:31.782260 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:16:31.782236    2580 generic.go:358] "Generic (PLEG): container finished" podID="9e671986-36b9-4c8b-bc33-12613fa1f8d9" containerID="d1278c54cd2991e6beee74caeed79057016e4700aada3aca8b251f224a159ebb" exitCode=2
May 25 13:16:31.782392 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:16:31.782313    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-678784f46b-c6nfj" event={"ID":"9e671986-36b9-4c8b-bc33-12613fa1f8d9","Type":"ContainerDied","Data":"d1278c54cd2991e6beee74caeed79057016e4700aada3aca8b251f224a159ebb"}
May 25 13:16:31.797816 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:16:31.797777    2580 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-08c99-predictor-cc5d7f476-czxk9" podStartSLOduration=1.797763948 podStartE2EDuration="1.797763948s" podCreationTimestamp="2026-05-25 13:16:30 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 13:16:31.796580425 +0000 UTC m=+691.988874785" watchObservedRunningTime="2026-05-25 13:16:31.797763948 +0000 UTC m=+691.990058307"
May 25 13:16:32.430830 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:16:32.430793    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-54c99bfc7d-sxdzl" podUID="706b7f73-1a67-41ce-bd5e-0b14892b0acf" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.34:8643/healthz\": dial tcp 10.134.0.34:8643: connect: connection refused"
May 25 13:16:32.786854 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:16:32.786730    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-08c99-predictor-cc5d7f476-czxk9"
May 25 13:16:32.788160 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:16:32.788124    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-08c99-predictor-cc5d7f476-czxk9" podUID="11a87d3c-4909-445a-ab48-fb162c262188" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
May 25 13:16:33.790884 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:16:33.790826    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-08c99-predictor-cc5d7f476-czxk9" podUID="11a87d3c-4909-445a-ab48-fb162c262188" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
May 25 13:16:35.315775 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:16:35.315750    2580 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-678784f46b-c6nfj"
May 25 13:16:35.376675 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:16:35.376599    2580 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9e671986-36b9-4c8b-bc33-12613fa1f8d9-proxy-tls\") pod \"9e671986-36b9-4c8b-bc33-12613fa1f8d9\" (UID: \"9e671986-36b9-4c8b-bc33-12613fa1f8d9\") "
May 25 13:16:35.376675 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:16:35.376655    2580 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-sklearn-graph-1-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9e671986-36b9-4c8b-bc33-12613fa1f8d9-isvc-sklearn-graph-1-kube-rbac-proxy-sar-config\") pod \"9e671986-36b9-4c8b-bc33-12613fa1f8d9\" (UID: \"9e671986-36b9-4c8b-bc33-12613fa1f8d9\") "
May 25 13:16:35.376884 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:16:35.376699    2580 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lcpqq\" (UniqueName: \"kubernetes.io/projected/9e671986-36b9-4c8b-bc33-12613fa1f8d9-kube-api-access-lcpqq\") pod \"9e671986-36b9-4c8b-bc33-12613fa1f8d9\" (UID: \"9e671986-36b9-4c8b-bc33-12613fa1f8d9\") "
May 25 13:16:35.376884 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:16:35.376724    2580 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9e671986-36b9-4c8b-bc33-12613fa1f8d9-kserve-provision-location\") pod \"9e671986-36b9-4c8b-bc33-12613fa1f8d9\" (UID: \"9e671986-36b9-4c8b-bc33-12613fa1f8d9\") "
May 25 13:16:35.377035 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:16:35.377011    2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e671986-36b9-4c8b-bc33-12613fa1f8d9-isvc-sklearn-graph-1-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-sklearn-graph-1-kube-rbac-proxy-sar-config") pod "9e671986-36b9-4c8b-bc33-12613fa1f8d9" (UID: "9e671986-36b9-4c8b-bc33-12613fa1f8d9"). InnerVolumeSpecName "isvc-sklearn-graph-1-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 13:16:35.377089 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:16:35.377056    2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e671986-36b9-4c8b-bc33-12613fa1f8d9-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9e671986-36b9-4c8b-bc33-12613fa1f8d9" (UID: "9e671986-36b9-4c8b-bc33-12613fa1f8d9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 13:16:35.378928 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:16:35.378907    2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e671986-36b9-4c8b-bc33-12613fa1f8d9-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9e671986-36b9-4c8b-bc33-12613fa1f8d9" (UID: "9e671986-36b9-4c8b-bc33-12613fa1f8d9"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 13:16:35.378999 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:16:35.378985    2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e671986-36b9-4c8b-bc33-12613fa1f8d9-kube-api-access-lcpqq" (OuterVolumeSpecName: "kube-api-access-lcpqq") pod "9e671986-36b9-4c8b-bc33-12613fa1f8d9" (UID: "9e671986-36b9-4c8b-bc33-12613fa1f8d9"). InnerVolumeSpecName "kube-api-access-lcpqq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 13:16:35.477881 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:16:35.477831    2580 reconciler_common.go:299] "Volume detached for volume \"isvc-sklearn-graph-1-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9e671986-36b9-4c8b-bc33-12613fa1f8d9-isvc-sklearn-graph-1-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-103.ec2.internal\" DevicePath \"\""
May 25 13:16:35.477881 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:16:35.477866    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lcpqq\" (UniqueName: \"kubernetes.io/projected/9e671986-36b9-4c8b-bc33-12613fa1f8d9-kube-api-access-lcpqq\") on node \"ip-10-0-135-103.ec2.internal\" DevicePath \"\""
May 25 13:16:35.477881 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:16:35.477880    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9e671986-36b9-4c8b-bc33-12613fa1f8d9-kserve-provision-location\") on node \"ip-10-0-135-103.ec2.internal\" DevicePath \"\""
May 25 13:16:35.478161 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:16:35.477898    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9e671986-36b9-4c8b-bc33-12613fa1f8d9-proxy-tls\") on node \"ip-10-0-135-103.ec2.internal\" DevicePath \"\""
May 25 13:16:35.799075 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:16:35.798992    2580 generic.go:358] "Generic (PLEG): container finished" podID="9e671986-36b9-4c8b-bc33-12613fa1f8d9" containerID="a7318035547c507d2ad424e17d9b606c2094a4c085e4015223e91792a0d3370b" exitCode=0
May 25 13:16:35.799208 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:16:35.799078    2580 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-678784f46b-c6nfj"
May 25 13:16:35.799208 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:16:35.799079    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-678784f46b-c6nfj" event={"ID":"9e671986-36b9-4c8b-bc33-12613fa1f8d9","Type":"ContainerDied","Data":"a7318035547c507d2ad424e17d9b606c2094a4c085e4015223e91792a0d3370b"}
May 25 13:16:35.799208 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:16:35.799119    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-678784f46b-c6nfj" event={"ID":"9e671986-36b9-4c8b-bc33-12613fa1f8d9","Type":"ContainerDied","Data":"d5b1243b5ec5520268ae2d20ac283c26c5057fa726b6951496b2138b7e28a2d9"}
May 25 13:16:35.799208 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:16:35.799133    2580 scope.go:117] "RemoveContainer" containerID="d1278c54cd2991e6beee74caeed79057016e4700aada3aca8b251f224a159ebb"
May 25 13:16:35.808292 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:16:35.808276    2580 scope.go:117] "RemoveContainer" containerID="a7318035547c507d2ad424e17d9b606c2094a4c085e4015223e91792a0d3370b"
May 25 13:16:35.815558 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:16:35.815541    2580 scope.go:117] "RemoveContainer" containerID="b90b29224916cd27e49a5c3a3e6054726e7d624268e81b371840987b00d42de7"
May 25 13:16:35.820931 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:16:35.820910    2580 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-678784f46b-c6nfj"]
May 25 13:16:35.823029 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:16:35.822993    2580 scope.go:117] "RemoveContainer" containerID="d1278c54cd2991e6beee74caeed79057016e4700aada3aca8b251f224a159ebb"
May 25 13:16:35.823283 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:16:35.823262    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1278c54cd2991e6beee74caeed79057016e4700aada3aca8b251f224a159ebb\": container with ID starting with d1278c54cd2991e6beee74caeed79057016e4700aada3aca8b251f224a159ebb not found: ID does not exist" containerID="d1278c54cd2991e6beee74caeed79057016e4700aada3aca8b251f224a159ebb"
May 25 13:16:35.823339 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:16:35.823295    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1278c54cd2991e6beee74caeed79057016e4700aada3aca8b251f224a159ebb"} err="failed to get container status \"d1278c54cd2991e6beee74caeed79057016e4700aada3aca8b251f224a159ebb\": rpc error: code = NotFound desc = could not find container \"d1278c54cd2991e6beee74caeed79057016e4700aada3aca8b251f224a159ebb\": container with ID starting with d1278c54cd2991e6beee74caeed79057016e4700aada3aca8b251f224a159ebb not found: ID does not exist"
May 25 13:16:35.823339 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:16:35.823319    2580 scope.go:117] "RemoveContainer" containerID="a7318035547c507d2ad424e17d9b606c2094a4c085e4015223e91792a0d3370b"
May 25 13:16:35.823676 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:16:35.823645    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7318035547c507d2ad424e17d9b606c2094a4c085e4015223e91792a0d3370b\": container with ID starting with a7318035547c507d2ad424e17d9b606c2094a4c085e4015223e91792a0d3370b not found: ID does not exist" containerID="a7318035547c507d2ad424e17d9b606c2094a4c085e4015223e91792a0d3370b"
May 25 13:16:35.823792 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:16:35.823681    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7318035547c507d2ad424e17d9b606c2094a4c085e4015223e91792a0d3370b"} err="failed to get container status \"a7318035547c507d2ad424e17d9b606c2094a4c085e4015223e91792a0d3370b\": rpc error: code = NotFound desc = could not find container \"a7318035547c507d2ad424e17d9b606c2094a4c085e4015223e91792a0d3370b\": container with ID starting with a7318035547c507d2ad424e17d9b606c2094a4c085e4015223e91792a0d3370b not found: ID does not exist"
May 25 13:16:35.823792 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:16:35.823706    2580 scope.go:117] "RemoveContainer" containerID="b90b29224916cd27e49a5c3a3e6054726e7d624268e81b371840987b00d42de7"
May 25 13:16:35.823960 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:16:35.823941    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b90b29224916cd27e49a5c3a3e6054726e7d624268e81b371840987b00d42de7\": container with ID starting with b90b29224916cd27e49a5c3a3e6054726e7d624268e81b371840987b00d42de7 not found: ID does not exist" containerID="b90b29224916cd27e49a5c3a3e6054726e7d624268e81b371840987b00d42de7"
May 25 13:16:35.824018 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:16:35.823965    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b90b29224916cd27e49a5c3a3e6054726e7d624268e81b371840987b00d42de7"} err="failed to get container status \"b90b29224916cd27e49a5c3a3e6054726e7d624268e81b371840987b00d42de7\": rpc error: code = NotFound desc = could not find container \"b90b29224916cd27e49a5c3a3e6054726e7d624268e81b371840987b00d42de7\": container with ID starting with b90b29224916cd27e49a5c3a3e6054726e7d624268e81b371840987b00d42de7 not found: ID does not exist"
May 25 13:16:35.824478 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:16:35.824458    2580 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-678784f46b-c6nfj"]
May 25 13:16:36.129696 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:16:36.129662    2580 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-54c99bfc7d-sxdzl"
May 25 13:16:36.184698 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:16:36.184667    2580 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d8dq4\" (UniqueName: \"kubernetes.io/projected/706b7f73-1a67-41ce-bd5e-0b14892b0acf-kube-api-access-d8dq4\") pod \"706b7f73-1a67-41ce-bd5e-0b14892b0acf\" (UID: \"706b7f73-1a67-41ce-bd5e-0b14892b0acf\") "
May 25 13:16:36.184889 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:16:36.184765    2580 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-sklearn-graph-2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/706b7f73-1a67-41ce-bd5e-0b14892b0acf-isvc-sklearn-graph-2-kube-rbac-proxy-sar-config\") pod \"706b7f73-1a67-41ce-bd5e-0b14892b0acf\" (UID: \"706b7f73-1a67-41ce-bd5e-0b14892b0acf\") "
May 25 13:16:36.184889 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:16:36.184805    2580 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/706b7f73-1a67-41ce-bd5e-0b14892b0acf-proxy-tls\") pod \"706b7f73-1a67-41ce-bd5e-0b14892b0acf\" (UID: \"706b7f73-1a67-41ce-bd5e-0b14892b0acf\") "
May 25 13:16:36.184889 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:16:36.184849    2580 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/706b7f73-1a67-41ce-bd5e-0b14892b0acf-kserve-provision-location\") pod \"706b7f73-1a67-41ce-bd5e-0b14892b0acf\" (UID: \"706b7f73-1a67-41ce-bd5e-0b14892b0acf\") "
May 25 13:16:36.185194 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:16:36.185162    2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/706b7f73-1a67-41ce-bd5e-0b14892b0acf-isvc-sklearn-graph-2-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-sklearn-graph-2-kube-rbac-proxy-sar-config") pod "706b7f73-1a67-41ce-bd5e-0b14892b0acf" (UID: "706b7f73-1a67-41ce-bd5e-0b14892b0acf"). InnerVolumeSpecName "isvc-sklearn-graph-2-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 13:16:36.185387 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:16:36.185335    2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/706b7f73-1a67-41ce-bd5e-0b14892b0acf-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "706b7f73-1a67-41ce-bd5e-0b14892b0acf" (UID: "706b7f73-1a67-41ce-bd5e-0b14892b0acf"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 13:16:36.187208 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:16:36.187179    2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/706b7f73-1a67-41ce-bd5e-0b14892b0acf-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "706b7f73-1a67-41ce-bd5e-0b14892b0acf" (UID: "706b7f73-1a67-41ce-bd5e-0b14892b0acf"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 13:16:36.187544 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:16:36.187522    2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/706b7f73-1a67-41ce-bd5e-0b14892b0acf-kube-api-access-d8dq4" (OuterVolumeSpecName: "kube-api-access-d8dq4") pod "706b7f73-1a67-41ce-bd5e-0b14892b0acf" (UID: "706b7f73-1a67-41ce-bd5e-0b14892b0acf"). InnerVolumeSpecName "kube-api-access-d8dq4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 13:16:36.285723 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:16:36.285692    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/706b7f73-1a67-41ce-bd5e-0b14892b0acf-proxy-tls\") on node \"ip-10-0-135-103.ec2.internal\" DevicePath \"\""
May 25 13:16:36.285723 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:16:36.285719    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/706b7f73-1a67-41ce-bd5e-0b14892b0acf-kserve-provision-location\") on node \"ip-10-0-135-103.ec2.internal\" DevicePath \"\""
May 25 13:16:36.285723 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:16:36.285730    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d8dq4\" (UniqueName: \"kubernetes.io/projected/706b7f73-1a67-41ce-bd5e-0b14892b0acf-kube-api-access-d8dq4\") on node \"ip-10-0-135-103.ec2.internal\" DevicePath \"\""
May 25 13:16:36.285950 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:16:36.285740    2580 reconciler_common.go:299] "Volume detached for volume \"isvc-sklearn-graph-2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/706b7f73-1a67-41ce-bd5e-0b14892b0acf-isvc-sklearn-graph-2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-103.ec2.internal\" DevicePath \"\""
May 25 13:16:36.377352 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:16:36.377273    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9e671986-36b9-4c8b-bc33-12613fa1f8d9" path="/var/lib/kubelet/pods/9e671986-36b9-4c8b-bc33-12613fa1f8d9/volumes"
May 25 13:16:36.804134 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:16:36.804047    2580 generic.go:358] "Generic (PLEG): container finished" podID="706b7f73-1a67-41ce-bd5e-0b14892b0acf" containerID="0f4c99c800602aa490b987eb66918aa47e7e86b5d55319f988ba4a3ef771dfa0" exitCode=0
May 25 13:16:36.804284 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:16:36.804132    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-54c99bfc7d-sxdzl" event={"ID":"706b7f73-1a67-41ce-bd5e-0b14892b0acf","Type":"ContainerDied","Data":"0f4c99c800602aa490b987eb66918aa47e7e86b5d55319f988ba4a3ef771dfa0"}
May 25 13:16:36.804284 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:16:36.804172    2580 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-54c99bfc7d-sxdzl"
May 25 13:16:36.804284 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:16:36.804181    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-54c99bfc7d-sxdzl" event={"ID":"706b7f73-1a67-41ce-bd5e-0b14892b0acf","Type":"ContainerDied","Data":"0123e3e03fd06ee70e93cc63a822ef9a427a2d8b6d37f76e71045f0407874ea4"}
May 25 13:16:36.804284 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:16:36.804203    2580 scope.go:117] "RemoveContainer" containerID="b66bae26ffb8583ab33b0fb5b593f219e483ab600deb3c51e290ec87deeed17a"
May 25 13:16:36.812295 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:16:36.812278    2580 scope.go:117] "RemoveContainer" containerID="0f4c99c800602aa490b987eb66918aa47e7e86b5d55319f988ba4a3ef771dfa0"
May 25 13:16:36.819469 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:16:36.819451    2580 scope.go:117] "RemoveContainer" containerID="78084eb143e88c203496c56b586c6ae390eb2ee84c6682908aa8b6181c441727"
May 25 13:16:36.822611 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:16:36.822591    2580 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-54c99bfc7d-sxdzl"]
May 25 13:16:36.826701 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:16:36.826679    2580 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-54c99bfc7d-sxdzl"]
May 25 13:16:36.827284 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:16:36.827264    2580 scope.go:117] "RemoveContainer" containerID="b66bae26ffb8583ab33b0fb5b593f219e483ab600deb3c51e290ec87deeed17a"
May 25 13:16:36.827568 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:16:36.827543    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b66bae26ffb8583ab33b0fb5b593f219e483ab600deb3c51e290ec87deeed17a\": container with ID starting with b66bae26ffb8583ab33b0fb5b593f219e483ab600deb3c51e290ec87deeed17a not found: ID does not exist" containerID="b66bae26ffb8583ab33b0fb5b593f219e483ab600deb3c51e290ec87deeed17a"
May 25 13:16:36.827626 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:16:36.827578    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b66bae26ffb8583ab33b0fb5b593f219e483ab600deb3c51e290ec87deeed17a"} err="failed to get container status \"b66bae26ffb8583ab33b0fb5b593f219e483ab600deb3c51e290ec87deeed17a\": rpc error: code = NotFound desc = could not find container \"b66bae26ffb8583ab33b0fb5b593f219e483ab600deb3c51e290ec87deeed17a\": container with ID starting with b66bae26ffb8583ab33b0fb5b593f219e483ab600deb3c51e290ec87deeed17a not found: ID does not exist"
May 25 13:16:36.827626 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:16:36.827596    2580 scope.go:117] "RemoveContainer" containerID="0f4c99c800602aa490b987eb66918aa47e7e86b5d55319f988ba4a3ef771dfa0"
May 25 13:16:36.827867 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:16:36.827849    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f4c99c800602aa490b987eb66918aa47e7e86b5d55319f988ba4a3ef771dfa0\": container with ID starting with 0f4c99c800602aa490b987eb66918aa47e7e86b5d55319f988ba4a3ef771dfa0 not found: ID does not exist" containerID="0f4c99c800602aa490b987eb66918aa47e7e86b5d55319f988ba4a3ef771dfa0"
May 25 13:16:36.827931 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:16:36.827873    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f4c99c800602aa490b987eb66918aa47e7e86b5d55319f988ba4a3ef771dfa0"} err="failed to get container status \"0f4c99c800602aa490b987eb66918aa47e7e86b5d55319f988ba4a3ef771dfa0\": rpc error: code = NotFound desc = could not find container \"0f4c99c800602aa490b987eb66918aa47e7e86b5d55319f988ba4a3ef771dfa0\": container with ID starting with 0f4c99c800602aa490b987eb66918aa47e7e86b5d55319f988ba4a3ef771dfa0 not found: ID does not exist"
May 25 13:16:36.827931 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:16:36.827890    2580 scope.go:117] "RemoveContainer" containerID="78084eb143e88c203496c56b586c6ae390eb2ee84c6682908aa8b6181c441727"
May 25 13:16:36.828092 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:16:36.828078    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78084eb143e88c203496c56b586c6ae390eb2ee84c6682908aa8b6181c441727\": container with ID starting with 78084eb143e88c203496c56b586c6ae390eb2ee84c6682908aa8b6181c441727 not found: ID does not exist" containerID="78084eb143e88c203496c56b586c6ae390eb2ee84c6682908aa8b6181c441727"
May 25 13:16:36.828129 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:16:36.828097    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78084eb143e88c203496c56b586c6ae390eb2ee84c6682908aa8b6181c441727"} err="failed to get container status \"78084eb143e88c203496c56b586c6ae390eb2ee84c6682908aa8b6181c441727\": rpc error: code = NotFound desc = could not find container \"78084eb143e88c203496c56b586c6ae390eb2ee84c6682908aa8b6181c441727\": container with ID starting with 78084eb143e88c203496c56b586c6ae390eb2ee84c6682908aa8b6181c441727 not found: ID does not exist"
May 25 13:16:38.377195 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:16:38.377158    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="706b7f73-1a67-41ce-bd5e-0b14892b0acf" path="/var/lib/kubelet/pods/706b7f73-1a67-41ce-bd5e-0b14892b0acf/volumes"
May 25 13:16:38.682159 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:16:38.682067    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-339a4-predictor-58f8586498-9r8dr" podUID="a3abc8ef-5a38-4ebc-bdbb-1c16bf6e82a0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
May 25 13:16:38.795399 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:16:38.795352    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-08c99-predictor-cc5d7f476-czxk9"
May 25 13:16:38.795911 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:16:38.795882    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-08c99-predictor-cc5d7f476-czxk9" podUID="11a87d3c-4909-445a-ab48-fb162c262188" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
May 25 13:16:48.683533 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:16:48.683491    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-339a4-predictor-58f8586498-9r8dr"
May 25 13:16:48.796308 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:16:48.796266    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-08c99-predictor-cc5d7f476-czxk9" podUID="11a87d3c-4909-445a-ab48-fb162c262188" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
May 25 13:16:58.796019 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:16:58.795984    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-08c99-predictor-cc5d7f476-czxk9" podUID="11a87d3c-4909-445a-ab48-fb162c262188" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
May 25 13:17:08.796304 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:17:08.796258    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-08c99-predictor-cc5d7f476-czxk9" podUID="11a87d3c-4909-445a-ab48-fb162c262188" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
May 25 13:17:18.796387 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:17:18.796338    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-08c99-predictor-cc5d7f476-czxk9"
May 25 13:20:00.311713 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:20:00.311678    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-646r6_59b905b8-5ce3-49f6-b47c-b5b3a525fe06/ovn-acl-logging/0.log"
May 25 13:20:00.312998 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:20:00.312974    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-646r6_59b905b8-5ce3-49f6-b47c-b5b3a525fe06/ovn-acl-logging/0.log"
May 25 13:25:00.336059 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:25:00.336029    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-646r6_59b905b8-5ce3-49f6-b47c-b5b3a525fe06/ovn-acl-logging/0.log"
May 25 13:25:00.340282 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:25:00.340262    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-646r6_59b905b8-5ce3-49f6-b47c-b5b3a525fe06/ovn-acl-logging/0.log"
May 25 13:25:15.745359 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:25:15.745325    2580 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-339a4-predictor-58f8586498-9r8dr"]
May 25 13:25:15.745903 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:25:15.745592    2580 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-339a4-predictor-58f8586498-9r8dr" podUID="a3abc8ef-5a38-4ebc-bdbb-1c16bf6e82a0" containerName="kserve-container" containerID="cri-o://c9cca1805d992e4be768f3d48a5d59173efd6ac11a9318e622bbc078141b4395" gracePeriod=30
May 25 13:25:15.745903 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:25:15.745653    2580 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-339a4-predictor-58f8586498-9r8dr" podUID="a3abc8ef-5a38-4ebc-bdbb-1c16bf6e82a0" containerName="kube-rbac-proxy" containerID="cri-o://91c99e5baa6d57eb5946b6b581dead9287c07d822087c96935bff84ddcefd5e9" gracePeriod=30
May 25 13:25:15.844748 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:25:15.844713    2580 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-db114-predictor-586c7dfd-52v5d"]
May 25 13:25:15.845098 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:25:15.845074    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="706b7f73-1a67-41ce-bd5e-0b14892b0acf" containerName="storage-initializer"
May 25 13:25:15.845098 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:25:15.845089    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="706b7f73-1a67-41ce-bd5e-0b14892b0acf" containerName="storage-initializer"
May 25 13:25:15.845098 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:25:15.845097    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e671986-36b9-4c8b-bc33-12613fa1f8d9" containerName="kserve-container"
May 25 13:25:15.845209 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:25:15.845102    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e671986-36b9-4c8b-bc33-12613fa1f8d9" containerName="kserve-container"
May 25 13:25:15.845209 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:25:15.845115    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="706b7f73-1a67-41ce-bd5e-0b14892b0acf" containerName="kube-rbac-proxy"
May 25 13:25:15.845209 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:25:15.845120    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="706b7f73-1a67-41ce-bd5e-0b14892b0acf" containerName="kube-rbac-proxy"
May 25 13:25:15.845209 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:25:15.845126    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="706b7f73-1a67-41ce-bd5e-0b14892b0acf" containerName="kserve-container"
May 25 13:25:15.845209 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:25:15.845137    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="706b7f73-1a67-41ce-bd5e-0b14892b0acf" containerName="kserve-container"
May 25 13:25:15.845209 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:25:15.845154    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e671986-36b9-4c8b-bc33-12613fa1f8d9" containerName="kube-rbac-proxy"
May 25 13:25:15.845209 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:25:15.845162    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e671986-36b9-4c8b-bc33-12613fa1f8d9" containerName="kube-rbac-proxy"
May 25 13:25:15.845209 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:25:15.845176    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e671986-36b9-4c8b-bc33-12613fa1f8d9" containerName="storage-initializer"
May 25 13:25:15.845209 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:25:15.845182    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e671986-36b9-4c8b-bc33-12613fa1f8d9" containerName="storage-initializer"
May 25 13:25:15.845509 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:25:15.845234    2580 memory_manager.go:356] "RemoveStaleState removing state" podUID="706b7f73-1a67-41ce-bd5e-0b14892b0acf" containerName="kube-rbac-proxy"
May 25 13:25:15.845509 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:25:15.845245    2580 memory_manager.go:356] "RemoveStaleState removing state" podUID="9e671986-36b9-4c8b-bc33-12613fa1f8d9" containerName="kube-rbac-proxy"
May 25 13:25:15.845509 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:25:15.845257    2580 memory_manager.go:356] "RemoveStaleState removing state" podUID="706b7f73-1a67-41ce-bd5e-0b14892b0acf" containerName="kserve-container"
May 25 13:25:15.845509 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:25:15.845267    2580 memory_manager.go:356] "RemoveStaleState removing state" podUID="9e671986-36b9-4c8b-bc33-12613fa1f8d9" containerName="kserve-container"
May 25 13:25:15.848480 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:25:15.848464    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-db114-predictor-586c7dfd-52v5d"
May 25 13:25:15.850644 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:25:15.850619    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-db114-predictor-serving-cert\""
May 25 13:25:15.850767 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:25:15.850622    2580 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-db114-kube-rbac-proxy-sar-config\""
May 25 13:25:15.870084 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:25:15.870055    2580 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-db114-predictor-586c7dfd-52v5d"]
May 25 13:25:15.902468 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:25:15.902439    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/830a1fff-253d-45b7-8c19-b896f15c0bc9-proxy-tls\") pod \"success-200-isvc-db114-predictor-586c7dfd-52v5d\" (UID: \"830a1fff-253d-45b7-8c19-b896f15c0bc9\") " pod="kserve-ci-e2e-test/success-200-isvc-db114-predictor-586c7dfd-52v5d"
May 25 13:25:15.902627 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:25:15.902579    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"success-200-isvc-db114-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/830a1fff-253d-45b7-8c19-b896f15c0bc9-success-200-isvc-db114-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-db114-predictor-586c7dfd-52v5d\" (UID: \"830a1fff-253d-45b7-8c19-b896f15c0bc9\") " pod="kserve-ci-e2e-test/success-200-isvc-db114-predictor-586c7dfd-52v5d"
May 25 13:25:15.902627 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:25:15.902613    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hcmp\" (UniqueName: \"kubernetes.io/projected/830a1fff-253d-45b7-8c19-b896f15c0bc9-kube-api-access-8hcmp\") pod \"success-200-isvc-db114-predictor-586c7dfd-52v5d\" (UID: \"830a1fff-253d-45b7-8c19-b896f15c0bc9\") " pod="kserve-ci-e2e-test/success-200-isvc-db114-predictor-586c7dfd-52v5d"
May 25 13:25:16.003435 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:25:16.003334    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"success-200-isvc-db114-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/830a1fff-253d-45b7-8c19-b896f15c0bc9-success-200-isvc-db114-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-db114-predictor-586c7dfd-52v5d\" (UID: \"830a1fff-253d-45b7-8c19-b896f15c0bc9\") " pod="kserve-ci-e2e-test/success-200-isvc-db114-predictor-586c7dfd-52v5d"
May 25 13:25:16.003435 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:25:16.003397    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-8hcmp\" (UniqueName: \"kubernetes.io/projected/830a1fff-253d-45b7-8c19-b896f15c0bc9-kube-api-access-8hcmp\") pod \"success-200-isvc-db114-predictor-586c7dfd-52v5d\" (UID: \"830a1fff-253d-45b7-8c19-b896f15c0bc9\") " pod="kserve-ci-e2e-test/success-200-isvc-db114-predictor-586c7dfd-52v5d"
May 25 13:25:16.003435 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:25:16.003433    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/830a1fff-253d-45b7-8c19-b896f15c0bc9-proxy-tls\") pod \"success-200-isvc-db114-predictor-586c7dfd-52v5d\" (UID: \"830a1fff-253d-45b7-8c19-b896f15c0bc9\") " pod="kserve-ci-e2e-test/success-200-isvc-db114-predictor-586c7dfd-52v5d"
May 25 13:25:16.003943 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:25:16.003922    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"success-200-isvc-db114-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/830a1fff-253d-45b7-8c19-b896f15c0bc9-success-200-isvc-db114-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-db114-predictor-586c7dfd-52v5d\" (UID: \"830a1fff-253d-45b7-8c19-b896f15c0bc9\") " pod="kserve-ci-e2e-test/success-200-isvc-db114-predictor-586c7dfd-52v5d"
May 25 13:25:16.005945 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:25:16.005922    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/830a1fff-253d-45b7-8c19-b896f15c0bc9-proxy-tls\") pod \"success-200-isvc-db114-predictor-586c7dfd-52v5d\" (UID: \"830a1fff-253d-45b7-8c19-b896f15c0bc9\") " pod="kserve-ci-e2e-test/success-200-isvc-db114-predictor-586c7dfd-52v5d"
May 25 13:25:16.011860 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:25:16.011825    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hcmp\" (UniqueName: \"kubernetes.io/projected/830a1fff-253d-45b7-8c19-b896f15c0bc9-kube-api-access-8hcmp\") pod \"success-200-isvc-db114-predictor-586c7dfd-52v5d\" (UID: \"830a1fff-253d-45b7-8c19-b896f15c0bc9\") " pod="kserve-ci-e2e-test/success-200-isvc-db114-predictor-586c7dfd-52v5d"
May 25 13:25:16.158854 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:25:16.158810    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-db114-predictor-586c7dfd-52v5d"
May 25 13:25:16.289231 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:25:16.289201    2580 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-db114-predictor-586c7dfd-52v5d"]
May 25 13:25:16.291398 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:25:16.291353    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod830a1fff_253d_45b7_8c19_b896f15c0bc9.slice/crio-4ef3bd40ff35b9779ca2de969c841c0e6800eb81315c5982a6d31e25a5d8019b WatchSource:0}: Error finding container 4ef3bd40ff35b9779ca2de969c841c0e6800eb81315c5982a6d31e25a5d8019b: Status 404 returned error can't find the container with id 4ef3bd40ff35b9779ca2de969c841c0e6800eb81315c5982a6d31e25a5d8019b
May 25 13:25:16.293181 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:25:16.293164    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 25 13:25:16.577694 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:25:16.577661    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-db114-predictor-586c7dfd-52v5d" event={"ID":"830a1fff-253d-45b7-8c19-b896f15c0bc9","Type":"ContainerStarted","Data":"f8e596ff22470a46f681c3094377c4d587ed88bf747c41d5dee74068b837bda5"}
May 25 13:25:16.577873 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:25:16.577705    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-db114-predictor-586c7dfd-52v5d" event={"ID":"830a1fff-253d-45b7-8c19-b896f15c0bc9","Type":"ContainerStarted","Data":"e6f8e7e839b663edfc3658615af2f6c552c9da5376428fbed4de2366f298f7f4"}
May 25 13:25:16.577873 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:25:16.577720    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-db114-predictor-586c7dfd-52v5d" event={"ID":"830a1fff-253d-45b7-8c19-b896f15c0bc9","Type":"ContainerStarted","Data":"4ef3bd40ff35b9779ca2de969c841c0e6800eb81315c5982a6d31e25a5d8019b"}
May 25 13:25:16.577873 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:25:16.577853    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-db114-predictor-586c7dfd-52v5d"
May 25 13:25:16.578026 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:25:16.577877    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-db114-predictor-586c7dfd-52v5d"
May 25 13:25:16.579428 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:25:16.579403    2580 generic.go:358] "Generic (PLEG): container finished" podID="a3abc8ef-5a38-4ebc-bdbb-1c16bf6e82a0" containerID="91c99e5baa6d57eb5946b6b581dead9287c07d822087c96935bff84ddcefd5e9" exitCode=2
May 25 13:25:16.579556 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:25:16.579468    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-339a4-predictor-58f8586498-9r8dr" event={"ID":"a3abc8ef-5a38-4ebc-bdbb-1c16bf6e82a0","Type":"ContainerDied","Data":"91c99e5baa6d57eb5946b6b581dead9287c07d822087c96935bff84ddcefd5e9"}
May 25 13:25:16.579556 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:25:16.579500    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-db114-predictor-586c7dfd-52v5d" podUID="830a1fff-253d-45b7-8c19-b896f15c0bc9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
May 25 13:25:16.596593 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:25:16.596535    2580 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-db114-predictor-586c7dfd-52v5d" podStartSLOduration=1.596514626 podStartE2EDuration="1.596514626s" podCreationTimestamp="2026-05-25 13:25:15 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 13:25:16.595616384 +0000 UTC m=+1216.787910770" watchObservedRunningTime="2026-05-25 13:25:16.596514626 +0000 UTC m=+1216.788808987"
May 25 13:25:17.582575 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:25:17.582538    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-db114-predictor-586c7dfd-52v5d" podUID="830a1fff-253d-45b7-8c19-b896f15c0bc9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
May 25 13:25:18.676963 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:25:18.676913    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-339a4-predictor-58f8586498-9r8dr" podUID="a3abc8ef-5a38-4ebc-bdbb-1c16bf6e82a0" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.35:8643/healthz\": dial tcp 10.134.0.35:8643: connect: connection refused"
May 25 13:25:18.682092 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:25:18.682062    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-339a4-predictor-58f8586498-9r8dr" podUID="a3abc8ef-5a38-4ebc-bdbb-1c16bf6e82a0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
May 25 13:25:18.896901 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:25:18.896876    2580 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-339a4-predictor-58f8586498-9r8dr"
May 25 13:25:18.923289 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:25:18.923254    2580 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a3abc8ef-5a38-4ebc-bdbb-1c16bf6e82a0-proxy-tls\") pod \"a3abc8ef-5a38-4ebc-bdbb-1c16bf6e82a0\" (UID: \"a3abc8ef-5a38-4ebc-bdbb-1c16bf6e82a0\") "
May 25 13:25:18.923472 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:25:18.923318    2580 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"success-200-isvc-339a4-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a3abc8ef-5a38-4ebc-bdbb-1c16bf6e82a0-success-200-isvc-339a4-kube-rbac-proxy-sar-config\") pod \"a3abc8ef-5a38-4ebc-bdbb-1c16bf6e82a0\" (UID: \"a3abc8ef-5a38-4ebc-bdbb-1c16bf6e82a0\") "
May 25 13:25:18.923472 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:25:18.923362    2580 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lvk2z\" (UniqueName: \"kubernetes.io/projected/a3abc8ef-5a38-4ebc-bdbb-1c16bf6e82a0-kube-api-access-lvk2z\") pod \"a3abc8ef-5a38-4ebc-bdbb-1c16bf6e82a0\" (UID: \"a3abc8ef-5a38-4ebc-bdbb-1c16bf6e82a0\") "
May 25 13:25:18.923639 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:25:18.923615    2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3abc8ef-5a38-4ebc-bdbb-1c16bf6e82a0-success-200-isvc-339a4-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "success-200-isvc-339a4-kube-rbac-proxy-sar-config") pod "a3abc8ef-5a38-4ebc-bdbb-1c16bf6e82a0" (UID: "a3abc8ef-5a38-4ebc-bdbb-1c16bf6e82a0"). InnerVolumeSpecName "success-200-isvc-339a4-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 13:25:18.925526 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:25:18.925500    2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3abc8ef-5a38-4ebc-bdbb-1c16bf6e82a0-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a3abc8ef-5a38-4ebc-bdbb-1c16bf6e82a0" (UID: "a3abc8ef-5a38-4ebc-bdbb-1c16bf6e82a0"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 13:25:18.925817 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:25:18.925792    2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3abc8ef-5a38-4ebc-bdbb-1c16bf6e82a0-kube-api-access-lvk2z" (OuterVolumeSpecName: "kube-api-access-lvk2z") pod "a3abc8ef-5a38-4ebc-bdbb-1c16bf6e82a0" (UID: "a3abc8ef-5a38-4ebc-bdbb-1c16bf6e82a0"). InnerVolumeSpecName "kube-api-access-lvk2z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 13:25:19.024429 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:25:19.024395    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a3abc8ef-5a38-4ebc-bdbb-1c16bf6e82a0-proxy-tls\") on node \"ip-10-0-135-103.ec2.internal\" DevicePath \"\""
May 25 13:25:19.024429 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:25:19.024425    2580 reconciler_common.go:299] "Volume detached for volume \"success-200-isvc-339a4-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a3abc8ef-5a38-4ebc-bdbb-1c16bf6e82a0-success-200-isvc-339a4-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-103.ec2.internal\" DevicePath \"\""
May 25 13:25:19.024429 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:25:19.024438    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lvk2z\" (UniqueName: \"kubernetes.io/projected/a3abc8ef-5a38-4ebc-bdbb-1c16bf6e82a0-kube-api-access-lvk2z\") on node \"ip-10-0-135-103.ec2.internal\" DevicePath \"\""
May 25 13:25:19.591429 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:25:19.591394    2580 generic.go:358] "Generic (PLEG): container finished" podID="a3abc8ef-5a38-4ebc-bdbb-1c16bf6e82a0" containerID="c9cca1805d992e4be768f3d48a5d59173efd6ac11a9318e622bbc078141b4395" exitCode=0
May 25 13:25:19.591596 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:25:19.591474    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-339a4-predictor-58f8586498-9r8dr" event={"ID":"a3abc8ef-5a38-4ebc-bdbb-1c16bf6e82a0","Type":"ContainerDied","Data":"c9cca1805d992e4be768f3d48a5d59173efd6ac11a9318e622bbc078141b4395"}
May 25 13:25:19.591596 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:25:19.591502    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-339a4-predictor-58f8586498-9r8dr" event={"ID":"a3abc8ef-5a38-4ebc-bdbb-1c16bf6e82a0","Type":"ContainerDied","Data":"b4a4797a3e638be1662750d1683bc3f6e9eff9003f061e84cc75250165221083"}
May 25 13:25:19.591596 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:25:19.591497    2580 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-339a4-predictor-58f8586498-9r8dr"
May 25 13:25:19.591596 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:25:19.591518    2580 scope.go:117] "RemoveContainer" containerID="91c99e5baa6d57eb5946b6b581dead9287c07d822087c96935bff84ddcefd5e9"
May 25 13:25:19.600099 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:25:19.600081    2580 scope.go:117] "RemoveContainer" containerID="c9cca1805d992e4be768f3d48a5d59173efd6ac11a9318e622bbc078141b4395"
May 25 13:25:19.607801 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:25:19.607785    2580 scope.go:117] "RemoveContainer" containerID="91c99e5baa6d57eb5946b6b581dead9287c07d822087c96935bff84ddcefd5e9"
May 25 13:25:19.608061 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:25:19.608045    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91c99e5baa6d57eb5946b6b581dead9287c07d822087c96935bff84ddcefd5e9\": container with ID starting with 91c99e5baa6d57eb5946b6b581dead9287c07d822087c96935bff84ddcefd5e9 not found: ID does not exist" containerID="91c99e5baa6d57eb5946b6b581dead9287c07d822087c96935bff84ddcefd5e9"
May 25 13:25:19.608103 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:25:19.608068    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91c99e5baa6d57eb5946b6b581dead9287c07d822087c96935bff84ddcefd5e9"} err="failed to get container status \"91c99e5baa6d57eb5946b6b581dead9287c07d822087c96935bff84ddcefd5e9\": rpc error: code = NotFound desc = could not find container \"91c99e5baa6d57eb5946b6b581dead9287c07d822087c96935bff84ddcefd5e9\": container with ID starting with 91c99e5baa6d57eb5946b6b581dead9287c07d822087c96935bff84ddcefd5e9 not found: ID does not exist"
May 25 13:25:19.608103 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:25:19.608083    2580 scope.go:117] "RemoveContainer" containerID="c9cca1805d992e4be768f3d48a5d59173efd6ac11a9318e622bbc078141b4395"
May 25 13:25:19.608301 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:25:19.608280    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9cca1805d992e4be768f3d48a5d59173efd6ac11a9318e622bbc078141b4395\": container with ID starting with c9cca1805d992e4be768f3d48a5d59173efd6ac11a9318e622bbc078141b4395 not found: ID does not exist" containerID="c9cca1805d992e4be768f3d48a5d59173efd6ac11a9318e622bbc078141b4395"
May 25 13:25:19.608358 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:25:19.608311    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9cca1805d992e4be768f3d48a5d59173efd6ac11a9318e622bbc078141b4395"} err="failed to get container status \"c9cca1805d992e4be768f3d48a5d59173efd6ac11a9318e622bbc078141b4395\": rpc error: code = NotFound desc = could not find container \"c9cca1805d992e4be768f3d48a5d59173efd6ac11a9318e622bbc078141b4395\": container with ID starting with c9cca1805d992e4be768f3d48a5d59173efd6ac11a9318e622bbc078141b4395 not found: ID does not exist"
May 25 13:25:19.613636 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:25:19.613616    2580 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-339a4-predictor-58f8586498-9r8dr"]
May 25 13:25:19.618553 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:25:19.618528    2580 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-339a4-predictor-58f8586498-9r8dr"]
May 25 13:25:20.378443 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:25:20.378410    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a3abc8ef-5a38-4ebc-bdbb-1c16bf6e82a0" path="/var/lib/kubelet/pods/a3abc8ef-5a38-4ebc-bdbb-1c16bf6e82a0/volumes"
May 25 13:25:22.587737 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:25:22.587705    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-db114-predictor-586c7dfd-52v5d"
May 25 13:25:22.588273 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:25:22.588236    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-db114-predictor-586c7dfd-52v5d" podUID="830a1fff-253d-45b7-8c19-b896f15c0bc9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
May 25 13:25:32.588501 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:25:32.588459    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-db114-predictor-586c7dfd-52v5d" podUID="830a1fff-253d-45b7-8c19-b896f15c0bc9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
May 25 13:25:42.588399 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:25:42.588344    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-db114-predictor-586c7dfd-52v5d" podUID="830a1fff-253d-45b7-8c19-b896f15c0bc9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
May 25 13:25:45.834464 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:25:45.834429    2580 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-08c99-predictor-cc5d7f476-czxk9"]
May 25 13:25:45.836931 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:25:45.834702    2580 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-08c99-predictor-cc5d7f476-czxk9" podUID="11a87d3c-4909-445a-ab48-fb162c262188" containerName="kserve-container" containerID="cri-o://a2ea57fe73b1bcf01a93b0508130f2f8617c35f48de87203553d255a6c0942a9" gracePeriod=30
May 25 13:25:45.836931 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:25:45.834799    2580 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-08c99-predictor-cc5d7f476-czxk9" podUID="11a87d3c-4909-445a-ab48-fb162c262188" containerName="kube-rbac-proxy" containerID="cri-o://2802e6e9aa0df5e48b1cb24637010f27e6243465f626fe5d1379fcf612ca6aa3" gracePeriod=30
May 25 13:25:45.908058 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:25:45.908027    2580 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-9ee57-predictor-767dfdd744-jlp7p"]
May 25 13:25:45.908395 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:25:45.908380    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3abc8ef-5a38-4ebc-bdbb-1c16bf6e82a0" containerName="kube-rbac-proxy"
May 25 13:25:45.908444 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:25:45.908398    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3abc8ef-5a38-4ebc-bdbb-1c16bf6e82a0" containerName="kube-rbac-proxy"
May 25 13:25:45.908444 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:25:45.908415    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3abc8ef-5a38-4ebc-bdbb-1c16bf6e82a0" containerName="kserve-container"
May 25 13:25:45.908444 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:25:45.908420    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3abc8ef-5a38-4ebc-bdbb-1c16bf6e82a0" containerName="kserve-container"
May 25 13:25:45.908538 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:25:45.908485    2580 memory_manager.go:356] "RemoveStaleState removing state" podUID="a3abc8ef-5a38-4ebc-bdbb-1c16bf6e82a0" containerName="kserve-container"
May 25 13:25:45.908538 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:25:45.908493    2580 memory_manager.go:356] "RemoveStaleState removing state" podUID="a3abc8ef-5a38-4ebc-bdbb-1c16bf6e82a0" containerName="kube-rbac-proxy"
May 25 13:25:45.911521 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:25:45.911495    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-9ee57-predictor-767dfdd744-jlp7p"
May 25 13:25:45.913794 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:25:45.913773    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-9ee57-predictor-serving-cert\""
May 25 13:25:45.913900 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:25:45.913794    2580 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-9ee57-kube-rbac-proxy-sar-config\""
May 25 13:25:45.921938 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:25:45.921891    2580 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-9ee57-predictor-767dfdd744-jlp7p"]
May 25 13:25:46.054966 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:25:46.054928    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"success-200-isvc-9ee57-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/504712a1-9a41-4d39-a504-32f340cfa820-success-200-isvc-9ee57-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-9ee57-predictor-767dfdd744-jlp7p\" (UID: \"504712a1-9a41-4d39-a504-32f340cfa820\") " pod="kserve-ci-e2e-test/success-200-isvc-9ee57-predictor-767dfdd744-jlp7p"
May 25 13:25:46.055137 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:25:46.054972    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/504712a1-9a41-4d39-a504-32f340cfa820-proxy-tls\") pod \"success-200-isvc-9ee57-predictor-767dfdd744-jlp7p\" (UID: \"504712a1-9a41-4d39-a504-32f340cfa820\") " pod="kserve-ci-e2e-test/success-200-isvc-9ee57-predictor-767dfdd744-jlp7p"
May 25 13:25:46.055137 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:25:46.055069    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56h6c\" (UniqueName: \"kubernetes.io/projected/504712a1-9a41-4d39-a504-32f340cfa820-kube-api-access-56h6c\") pod \"success-200-isvc-9ee57-predictor-767dfdd744-jlp7p\" (UID: \"504712a1-9a41-4d39-a504-32f340cfa820\") " pod="kserve-ci-e2e-test/success-200-isvc-9ee57-predictor-767dfdd744-jlp7p"
May 25 13:25:46.156009 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:25:46.155930    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-56h6c\" (UniqueName: \"kubernetes.io/projected/504712a1-9a41-4d39-a504-32f340cfa820-kube-api-access-56h6c\") pod \"success-200-isvc-9ee57-predictor-767dfdd744-jlp7p\" (UID: \"504712a1-9a41-4d39-a504-32f340cfa820\") " pod="kserve-ci-e2e-test/success-200-isvc-9ee57-predictor-767dfdd744-jlp7p"
May 25 13:25:46.156009 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:25:46.155986    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"success-200-isvc-9ee57-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/504712a1-9a41-4d39-a504-32f340cfa820-success-200-isvc-9ee57-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-9ee57-predictor-767dfdd744-jlp7p\" (UID: \"504712a1-9a41-4d39-a504-32f340cfa820\") " pod="kserve-ci-e2e-test/success-200-isvc-9ee57-predictor-767dfdd744-jlp7p"
May 25 13:25:46.156198 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:25:46.156016    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/504712a1-9a41-4d39-a504-32f340cfa820-proxy-tls\") pod \"success-200-isvc-9ee57-predictor-767dfdd744-jlp7p\" (UID: \"504712a1-9a41-4d39-a504-32f340cfa820\") " pod="kserve-ci-e2e-test/success-200-isvc-9ee57-predictor-767dfdd744-jlp7p"
May 25 13:25:46.156737 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:25:46.156678    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"success-200-isvc-9ee57-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/504712a1-9a41-4d39-a504-32f340cfa820-success-200-isvc-9ee57-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-9ee57-predictor-767dfdd744-jlp7p\" (UID: \"504712a1-9a41-4d39-a504-32f340cfa820\") " pod="kserve-ci-e2e-test/success-200-isvc-9ee57-predictor-767dfdd744-jlp7p"
May 25 13:25:46.158560 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:25:46.158538    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/504712a1-9a41-4d39-a504-32f340cfa820-proxy-tls\") pod \"success-200-isvc-9ee57-predictor-767dfdd744-jlp7p\" (UID: \"504712a1-9a41-4d39-a504-32f340cfa820\") " pod="kserve-ci-e2e-test/success-200-isvc-9ee57-predictor-767dfdd744-jlp7p"
May 25 13:25:46.164875 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:25:46.164850    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-56h6c\" (UniqueName: \"kubernetes.io/projected/504712a1-9a41-4d39-a504-32f340cfa820-kube-api-access-56h6c\") pod \"success-200-isvc-9ee57-predictor-767dfdd744-jlp7p\" (UID: \"504712a1-9a41-4d39-a504-32f340cfa820\") " pod="kserve-ci-e2e-test/success-200-isvc-9ee57-predictor-767dfdd744-jlp7p"
May 25 13:25:46.222233 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:25:46.222198    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-9ee57-predictor-767dfdd744-jlp7p"
May 25 13:25:46.353864 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:25:46.353830    2580 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-9ee57-predictor-767dfdd744-jlp7p"]
May 25 13:25:46.356842 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:25:46.356815    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod504712a1_9a41_4d39_a504_32f340cfa820.slice/crio-b3e973b9da6d07f644867a1e4bc04cadeaa89b2e4c827c3519bd64ac553f6c80 WatchSource:0}: Error finding container b3e973b9da6d07f644867a1e4bc04cadeaa89b2e4c827c3519bd64ac553f6c80: Status 404 returned error can't find the container with id b3e973b9da6d07f644867a1e4bc04cadeaa89b2e4c827c3519bd64ac553f6c80
May 25 13:25:46.689733 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:25:46.689636    2580 generic.go:358] "Generic (PLEG): container finished" podID="11a87d3c-4909-445a-ab48-fb162c262188" containerID="2802e6e9aa0df5e48b1cb24637010f27e6243465f626fe5d1379fcf612ca6aa3" exitCode=2
May 25 13:25:46.689733 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:25:46.689706    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-08c99-predictor-cc5d7f476-czxk9" event={"ID":"11a87d3c-4909-445a-ab48-fb162c262188","Type":"ContainerDied","Data":"2802e6e9aa0df5e48b1cb24637010f27e6243465f626fe5d1379fcf612ca6aa3"}
May 25 13:25:46.692925 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:25:46.692887    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-9ee57-predictor-767dfdd744-jlp7p" event={"ID":"504712a1-9a41-4d39-a504-32f340cfa820","Type":"ContainerStarted","Data":"002741c0204601391e10239277d7789ffa3ae5f0eadd871bb36518bb53e91e7d"}
May 25 13:25:46.692925 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:25:46.692932    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-9ee57-predictor-767dfdd744-jlp7p" event={"ID":"504712a1-9a41-4d39-a504-32f340cfa820","Type":"ContainerStarted","Data":"566f3df6b8babe53ae8148a8b0ceb1b64eb142805ad36f5865a73f0eaa01d27a"}
May 25 13:25:46.693188 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:25:46.692947    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-9ee57-predictor-767dfdd744-jlp7p" event={"ID":"504712a1-9a41-4d39-a504-32f340cfa820","Type":"ContainerStarted","Data":"b3e973b9da6d07f644867a1e4bc04cadeaa89b2e4c827c3519bd64ac553f6c80"}
May 25 13:25:46.693188 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:25:46.693075    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-9ee57-predictor-767dfdd744-jlp7p"
May 25 13:25:46.710043 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:25:46.709989    2580 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-9ee57-predictor-767dfdd744-jlp7p" podStartSLOduration=1.709972317 podStartE2EDuration="1.709972317s" podCreationTimestamp="2026-05-25 13:25:45 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 13:25:46.709079687 +0000 UTC m=+1246.901374046" watchObservedRunningTime="2026-05-25 13:25:46.709972317 +0000 UTC m=+1246.902266677"
May 25 13:25:47.696282 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:25:47.696247    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-9ee57-predictor-767dfdd744-jlp7p"
May 25 13:25:47.697607 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:25:47.697576    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-9ee57-predictor-767dfdd744-jlp7p" podUID="504712a1-9a41-4d39-a504-32f340cfa820" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
May 25 13:25:48.699130 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:25:48.699093    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-9ee57-predictor-767dfdd744-jlp7p" podUID="504712a1-9a41-4d39-a504-32f340cfa820" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
May 25 13:25:48.791695 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:25:48.791648    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-08c99-predictor-cc5d7f476-czxk9" podUID="11a87d3c-4909-445a-ab48-fb162c262188" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.36:8643/healthz\": dial tcp 10.134.0.36:8643: connect: connection refused"
May 25 13:25:48.795970 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:25:48.795934    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-08c99-predictor-cc5d7f476-czxk9" podUID="11a87d3c-4909-445a-ab48-fb162c262188" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
May 25 13:25:49.184999 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:25:49.184975    2580 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-08c99-predictor-cc5d7f476-czxk9"
May 25 13:25:49.282767 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:25:49.282685    2580 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/11a87d3c-4909-445a-ab48-fb162c262188-proxy-tls\") pod \"11a87d3c-4909-445a-ab48-fb162c262188\" (UID: \"11a87d3c-4909-445a-ab48-fb162c262188\") "
May 25 13:25:49.282767 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:25:49.282735    2580 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q65s7\" (UniqueName: \"kubernetes.io/projected/11a87d3c-4909-445a-ab48-fb162c262188-kube-api-access-q65s7\") pod \"11a87d3c-4909-445a-ab48-fb162c262188\" (UID: \"11a87d3c-4909-445a-ab48-fb162c262188\") "
May 25 13:25:49.282767 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:25:49.282761    2580 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"success-200-isvc-08c99-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/11a87d3c-4909-445a-ab48-fb162c262188-success-200-isvc-08c99-kube-rbac-proxy-sar-config\") pod \"11a87d3c-4909-445a-ab48-fb162c262188\" (UID: \"11a87d3c-4909-445a-ab48-fb162c262188\") "
May 25 13:25:49.283181 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:25:49.283151    2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11a87d3c-4909-445a-ab48-fb162c262188-success-200-isvc-08c99-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "success-200-isvc-08c99-kube-rbac-proxy-sar-config") pod "11a87d3c-4909-445a-ab48-fb162c262188" (UID: "11a87d3c-4909-445a-ab48-fb162c262188"). InnerVolumeSpecName "success-200-isvc-08c99-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 13:25:49.284990 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:25:49.284960    2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11a87d3c-4909-445a-ab48-fb162c262188-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "11a87d3c-4909-445a-ab48-fb162c262188" (UID: "11a87d3c-4909-445a-ab48-fb162c262188"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 13:25:49.285311 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:25:49.285282    2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11a87d3c-4909-445a-ab48-fb162c262188-kube-api-access-q65s7" (OuterVolumeSpecName: "kube-api-access-q65s7") pod "11a87d3c-4909-445a-ab48-fb162c262188" (UID: "11a87d3c-4909-445a-ab48-fb162c262188"). InnerVolumeSpecName "kube-api-access-q65s7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 13:25:49.383743 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:25:49.383703    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/11a87d3c-4909-445a-ab48-fb162c262188-proxy-tls\") on node \"ip-10-0-135-103.ec2.internal\" DevicePath \"\""
May 25 13:25:49.383743 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:25:49.383736    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q65s7\" (UniqueName: \"kubernetes.io/projected/11a87d3c-4909-445a-ab48-fb162c262188-kube-api-access-q65s7\") on node \"ip-10-0-135-103.ec2.internal\" DevicePath \"\""
May 25 13:25:49.383743 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:25:49.383747    2580 reconciler_common.go:299] "Volume detached for volume \"success-200-isvc-08c99-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/11a87d3c-4909-445a-ab48-fb162c262188-success-200-isvc-08c99-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-103.ec2.internal\" DevicePath \"\""
May 25 13:25:49.704004 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:25:49.703968    2580 generic.go:358] "Generic (PLEG): container finished" podID="11a87d3c-4909-445a-ab48-fb162c262188" containerID="a2ea57fe73b1bcf01a93b0508130f2f8617c35f48de87203553d255a6c0942a9" exitCode=0
May 25 13:25:49.704004 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:25:49.704006    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-08c99-predictor-cc5d7f476-czxk9" event={"ID":"11a87d3c-4909-445a-ab48-fb162c262188","Type":"ContainerDied","Data":"a2ea57fe73b1bcf01a93b0508130f2f8617c35f48de87203553d255a6c0942a9"}
May 25 13:25:49.704472 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:25:49.704028    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-08c99-predictor-cc5d7f476-czxk9" event={"ID":"11a87d3c-4909-445a-ab48-fb162c262188","Type":"ContainerDied","Data":"f233fe73c0486972d4ae1f02dfb6968bfae1a29f531c8d82d2bd5ee8cb88548f"}
May 25 13:25:49.704472 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:25:49.704044    2580 scope.go:117] "RemoveContainer" containerID="2802e6e9aa0df5e48b1cb24637010f27e6243465f626fe5d1379fcf612ca6aa3"
May 25 13:25:49.704472 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:25:49.704054    2580 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-08c99-predictor-cc5d7f476-czxk9"
May 25 13:25:49.712259 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:25:49.712242    2580 scope.go:117] "RemoveContainer" containerID="a2ea57fe73b1bcf01a93b0508130f2f8617c35f48de87203553d255a6c0942a9"
May 25 13:25:49.721787 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:25:49.721766    2580 scope.go:117] "RemoveContainer" containerID="2802e6e9aa0df5e48b1cb24637010f27e6243465f626fe5d1379fcf612ca6aa3"
May 25 13:25:49.722029 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:25:49.722011    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2802e6e9aa0df5e48b1cb24637010f27e6243465f626fe5d1379fcf612ca6aa3\": container with ID starting with 2802e6e9aa0df5e48b1cb24637010f27e6243465f626fe5d1379fcf612ca6aa3 not found: ID does not exist" containerID="2802e6e9aa0df5e48b1cb24637010f27e6243465f626fe5d1379fcf612ca6aa3"
May 25 13:25:49.722074 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:25:49.722038    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2802e6e9aa0df5e48b1cb24637010f27e6243465f626fe5d1379fcf612ca6aa3"} err="failed to get container status \"2802e6e9aa0df5e48b1cb24637010f27e6243465f626fe5d1379fcf612ca6aa3\": rpc error: code = NotFound desc = could not find container \"2802e6e9aa0df5e48b1cb24637010f27e6243465f626fe5d1379fcf612ca6aa3\": container with ID starting with 2802e6e9aa0df5e48b1cb24637010f27e6243465f626fe5d1379fcf612ca6aa3 not found: ID does not exist"
May 25 13:25:49.722074 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:25:49.722057    2580 scope.go:117] "RemoveContainer" containerID="a2ea57fe73b1bcf01a93b0508130f2f8617c35f48de87203553d255a6c0942a9"
May 25 13:25:49.722300 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:25:49.722285    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2ea57fe73b1bcf01a93b0508130f2f8617c35f48de87203553d255a6c0942a9\": container with ID starting with a2ea57fe73b1bcf01a93b0508130f2f8617c35f48de87203553d255a6c0942a9 not found: ID does not exist" containerID="a2ea57fe73b1bcf01a93b0508130f2f8617c35f48de87203553d255a6c0942a9"
May 25 13:25:49.722347 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:25:49.722302    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2ea57fe73b1bcf01a93b0508130f2f8617c35f48de87203553d255a6c0942a9"} err="failed to get container status \"a2ea57fe73b1bcf01a93b0508130f2f8617c35f48de87203553d255a6c0942a9\": rpc error: code = NotFound desc = could not find container \"a2ea57fe73b1bcf01a93b0508130f2f8617c35f48de87203553d255a6c0942a9\": container with ID starting with a2ea57fe73b1bcf01a93b0508130f2f8617c35f48de87203553d255a6c0942a9 not found: ID does not exist"
May 25 13:25:49.725717 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:25:49.725694    2580 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-08c99-predictor-cc5d7f476-czxk9"]
May 25 13:25:49.729275 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:25:49.729254    2580 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-08c99-predictor-cc5d7f476-czxk9"]
May 25 13:25:50.383306 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:25:50.383267    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="11a87d3c-4909-445a-ab48-fb162c262188" path="/var/lib/kubelet/pods/11a87d3c-4909-445a-ab48-fb162c262188/volumes"
May 25 13:25:52.589086 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:25:52.589034    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-db114-predictor-586c7dfd-52v5d" podUID="830a1fff-253d-45b7-8c19-b896f15c0bc9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
May 25 13:25:53.703285 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:25:53.703260    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-9ee57-predictor-767dfdd744-jlp7p"
May 25 13:25:53.703803 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:25:53.703728    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-9ee57-predictor-767dfdd744-jlp7p" podUID="504712a1-9a41-4d39-a504-32f340cfa820" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
May 25 13:26:02.589060 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:26:02.589028    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-db114-predictor-586c7dfd-52v5d"
May 25 13:26:03.704529 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:26:03.704493    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-9ee57-predictor-767dfdd744-jlp7p" podUID="504712a1-9a41-4d39-a504-32f340cfa820" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
May 25 13:26:13.704741 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:26:13.704702    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-9ee57-predictor-767dfdd744-jlp7p" podUID="504712a1-9a41-4d39-a504-32f340cfa820" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
May 25 13:26:23.704000 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:26:23.703956    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-9ee57-predictor-767dfdd744-jlp7p" podUID="504712a1-9a41-4d39-a504-32f340cfa820" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
May 25 13:26:26.144047 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:26:26.144004    2580 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-db114-predictor-586c7dfd-52v5d"]
May 25 13:26:26.144580 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:26:26.144388    2580 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-db114-predictor-586c7dfd-52v5d" podUID="830a1fff-253d-45b7-8c19-b896f15c0bc9" containerName="kserve-container" containerID="cri-o://e6f8e7e839b663edfc3658615af2f6c552c9da5376428fbed4de2366f298f7f4" gracePeriod=30
May 25 13:26:26.144580 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:26:26.144433    2580 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-db114-predictor-586c7dfd-52v5d" podUID="830a1fff-253d-45b7-8c19-b896f15c0bc9" containerName="kube-rbac-proxy" containerID="cri-o://f8e596ff22470a46f681c3094377c4d587ed88bf747c41d5dee74068b837bda5" gracePeriod=30
May 25 13:26:26.182352 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:26:26.182324    2580 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-3e647-predictor-55bfd66d8d-pj8fl"]
May 25 13:26:26.182758 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:26:26.182743    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="11a87d3c-4909-445a-ab48-fb162c262188" containerName="kube-rbac-proxy"
May 25 13:26:26.182804 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:26:26.182759    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="11a87d3c-4909-445a-ab48-fb162c262188" containerName="kube-rbac-proxy"
May 25 13:26:26.182804 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:26:26.182776    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="11a87d3c-4909-445a-ab48-fb162c262188" containerName="kserve-container"
May 25 13:26:26.182804 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:26:26.182781    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="11a87d3c-4909-445a-ab48-fb162c262188" containerName="kserve-container"
May 25 13:26:26.182907 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:26:26.182837    2580 memory_manager.go:356] "RemoveStaleState removing state" podUID="11a87d3c-4909-445a-ab48-fb162c262188" containerName="kube-rbac-proxy"
May 25 13:26:26.182907 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:26:26.182853    2580 memory_manager.go:356] "RemoveStaleState removing state" podUID="11a87d3c-4909-445a-ab48-fb162c262188" containerName="kserve-container"
May 25 13:26:26.186065 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:26:26.186050    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-3e647-predictor-55bfd66d8d-pj8fl"
May 25 13:26:26.188477 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:26:26.188455    2580 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-3e647-kube-rbac-proxy-sar-config\""
May 25 13:26:26.188593 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:26:26.188511    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-3e647-predictor-serving-cert\""
May 25 13:26:26.196636 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:26:26.196607    2580 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-3e647-predictor-55bfd66d8d-pj8fl"]
May 25 13:26:26.289278 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:26:26.289244    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpm89\" (UniqueName: \"kubernetes.io/projected/ade8fe09-bea0-4b3d-a568-e52c3fa9fe0d-kube-api-access-lpm89\") pod \"success-200-isvc-3e647-predictor-55bfd66d8d-pj8fl\" (UID: \"ade8fe09-bea0-4b3d-a568-e52c3fa9fe0d\") " pod="kserve-ci-e2e-test/success-200-isvc-3e647-predictor-55bfd66d8d-pj8fl"
May 25 13:26:26.289460 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:26:26.289315    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"success-200-isvc-3e647-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ade8fe09-bea0-4b3d-a568-e52c3fa9fe0d-success-200-isvc-3e647-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-3e647-predictor-55bfd66d8d-pj8fl\" (UID: \"ade8fe09-bea0-4b3d-a568-e52c3fa9fe0d\") " pod="kserve-ci-e2e-test/success-200-isvc-3e647-predictor-55bfd66d8d-pj8fl"
May 25 13:26:26.289460 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:26:26.289350    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ade8fe09-bea0-4b3d-a568-e52c3fa9fe0d-proxy-tls\") pod \"success-200-isvc-3e647-predictor-55bfd66d8d-pj8fl\" (UID: \"ade8fe09-bea0-4b3d-a568-e52c3fa9fe0d\") " pod="kserve-ci-e2e-test/success-200-isvc-3e647-predictor-55bfd66d8d-pj8fl"
May 25 13:26:26.389756 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:26:26.389712    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ade8fe09-bea0-4b3d-a568-e52c3fa9fe0d-proxy-tls\") pod \"success-200-isvc-3e647-predictor-55bfd66d8d-pj8fl\" (UID: \"ade8fe09-bea0-4b3d-a568-e52c3fa9fe0d\") " pod="kserve-ci-e2e-test/success-200-isvc-3e647-predictor-55bfd66d8d-pj8fl"
May 25 13:26:26.389952 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:26:26.389808    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-lpm89\" (UniqueName: \"kubernetes.io/projected/ade8fe09-bea0-4b3d-a568-e52c3fa9fe0d-kube-api-access-lpm89\") pod \"success-200-isvc-3e647-predictor-55bfd66d8d-pj8fl\" (UID: \"ade8fe09-bea0-4b3d-a568-e52c3fa9fe0d\") " pod="kserve-ci-e2e-test/success-200-isvc-3e647-predictor-55bfd66d8d-pj8fl"
May 25 13:26:26.390059 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:26:26.390041    2580 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-3e647-predictor-serving-cert: secret "success-200-isvc-3e647-predictor-serving-cert" not found
May 25 13:26:26.390142 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:26:26.390128    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ade8fe09-bea0-4b3d-a568-e52c3fa9fe0d-proxy-tls podName:ade8fe09-bea0-4b3d-a568-e52c3fa9fe0d nodeName:}" failed. No retries permitted until 2026-05-25 13:26:26.890108905 +0000 UTC m=+1287.082403245 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/ade8fe09-bea0-4b3d-a568-e52c3fa9fe0d-proxy-tls") pod "success-200-isvc-3e647-predictor-55bfd66d8d-pj8fl" (UID: "ade8fe09-bea0-4b3d-a568-e52c3fa9fe0d") : secret "success-200-isvc-3e647-predictor-serving-cert" not found
May 25 13:26:26.390338 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:26:26.390303    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"success-200-isvc-3e647-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ade8fe09-bea0-4b3d-a568-e52c3fa9fe0d-success-200-isvc-3e647-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-3e647-predictor-55bfd66d8d-pj8fl\" (UID: \"ade8fe09-bea0-4b3d-a568-e52c3fa9fe0d\") " pod="kserve-ci-e2e-test/success-200-isvc-3e647-predictor-55bfd66d8d-pj8fl"
May 25 13:26:26.390958 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:26:26.390934    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"success-200-isvc-3e647-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ade8fe09-bea0-4b3d-a568-e52c3fa9fe0d-success-200-isvc-3e647-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-3e647-predictor-55bfd66d8d-pj8fl\" (UID: \"ade8fe09-bea0-4b3d-a568-e52c3fa9fe0d\") " pod="kserve-ci-e2e-test/success-200-isvc-3e647-predictor-55bfd66d8d-pj8fl"
May 25 13:26:26.398809 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:26:26.398759    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpm89\" (UniqueName: \"kubernetes.io/projected/ade8fe09-bea0-4b3d-a568-e52c3fa9fe0d-kube-api-access-lpm89\") pod \"success-200-isvc-3e647-predictor-55bfd66d8d-pj8fl\" (UID: \"ade8fe09-bea0-4b3d-a568-e52c3fa9fe0d\") " pod="kserve-ci-e2e-test/success-200-isvc-3e647-predictor-55bfd66d8d-pj8fl"
May 25 13:26:26.832544 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:26:26.832504    2580 generic.go:358] "Generic (PLEG): container finished" podID="830a1fff-253d-45b7-8c19-b896f15c0bc9" containerID="f8e596ff22470a46f681c3094377c4d587ed88bf747c41d5dee74068b837bda5" exitCode=2
May 25 13:26:26.832693 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:26:26.832580    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-db114-predictor-586c7dfd-52v5d" event={"ID":"830a1fff-253d-45b7-8c19-b896f15c0bc9","Type":"ContainerDied","Data":"f8e596ff22470a46f681c3094377c4d587ed88bf747c41d5dee74068b837bda5"}
May 25 13:26:26.895106 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:26:26.895074    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ade8fe09-bea0-4b3d-a568-e52c3fa9fe0d-proxy-tls\") pod \"success-200-isvc-3e647-predictor-55bfd66d8d-pj8fl\" (UID: \"ade8fe09-bea0-4b3d-a568-e52c3fa9fe0d\") " pod="kserve-ci-e2e-test/success-200-isvc-3e647-predictor-55bfd66d8d-pj8fl"
May 25 13:26:26.897581 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:26:26.897553    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ade8fe09-bea0-4b3d-a568-e52c3fa9fe0d-proxy-tls\") pod \"success-200-isvc-3e647-predictor-55bfd66d8d-pj8fl\" (UID: \"ade8fe09-bea0-4b3d-a568-e52c3fa9fe0d\") " pod="kserve-ci-e2e-test/success-200-isvc-3e647-predictor-55bfd66d8d-pj8fl"
May 25 13:26:27.098711 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:26:27.098621    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-3e647-predictor-55bfd66d8d-pj8fl"
May 25 13:26:27.227196 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:26:27.227167    2580 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-3e647-predictor-55bfd66d8d-pj8fl"]
May 25 13:26:27.230404 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:26:27.230360    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podade8fe09_bea0_4b3d_a568_e52c3fa9fe0d.slice/crio-9fc87b70ab2ef5fb41e9a6d6ee969adf0a5c6a9157076d71985919a7a72af9da WatchSource:0}: Error finding container 9fc87b70ab2ef5fb41e9a6d6ee969adf0a5c6a9157076d71985919a7a72af9da: Status 404 returned error can't find the container with id 9fc87b70ab2ef5fb41e9a6d6ee969adf0a5c6a9157076d71985919a7a72af9da
May 25 13:26:27.583195 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:26:27.583148    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-db114-predictor-586c7dfd-52v5d" podUID="830a1fff-253d-45b7-8c19-b896f15c0bc9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.37:8643/healthz\": dial tcp 10.134.0.37:8643: connect: connection refused"
May 25 13:26:27.838551 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:26:27.838507    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-3e647-predictor-55bfd66d8d-pj8fl" event={"ID":"ade8fe09-bea0-4b3d-a568-e52c3fa9fe0d","Type":"ContainerStarted","Data":"eafa8b9485f996d825705d71857bba851e4509e7a6a9f487bba089618afd9618"}
May 25 13:26:27.838551 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:26:27.838548    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-3e647-predictor-55bfd66d8d-pj8fl" event={"ID":"ade8fe09-bea0-4b3d-a568-e52c3fa9fe0d","Type":"ContainerStarted","Data":"0fec265aa718d1244f7ebaba9fc08acc343520d17c7b19f95175c361f88f8db1"}
May 25 13:26:27.838799 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:26:27.838557    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-3e647-predictor-55bfd66d8d-pj8fl" event={"ID":"ade8fe09-bea0-4b3d-a568-e52c3fa9fe0d","Type":"ContainerStarted","Data":"9fc87b70ab2ef5fb41e9a6d6ee969adf0a5c6a9157076d71985919a7a72af9da"}
May 25 13:26:27.838799 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:26:27.838595    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-3e647-predictor-55bfd66d8d-pj8fl"
May 25 13:26:27.858532 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:26:27.858492    2580 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-3e647-predictor-55bfd66d8d-pj8fl" podStartSLOduration=1.858480277 podStartE2EDuration="1.858480277s" podCreationTimestamp="2026-05-25 13:26:26 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 13:26:27.85681019 +0000 UTC m=+1288.049104562" watchObservedRunningTime="2026-05-25 13:26:27.858480277 +0000 UTC m=+1288.050774673"
May 25 13:26:28.842170 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:26:28.842141    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-3e647-predictor-55bfd66d8d-pj8fl"
May 25 13:26:28.843410 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:26:28.843381    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-3e647-predictor-55bfd66d8d-pj8fl" podUID="ade8fe09-bea0-4b3d-a568-e52c3fa9fe0d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
May 25 13:26:29.402274 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:26:29.402253    2580 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-db114-predictor-586c7dfd-52v5d"
May 25 13:26:29.518537 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:26:29.518458    2580 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8hcmp\" (UniqueName: \"kubernetes.io/projected/830a1fff-253d-45b7-8c19-b896f15c0bc9-kube-api-access-8hcmp\") pod \"830a1fff-253d-45b7-8c19-b896f15c0bc9\" (UID: \"830a1fff-253d-45b7-8c19-b896f15c0bc9\") "
May 25 13:26:29.518681 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:26:29.518564    2580 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/830a1fff-253d-45b7-8c19-b896f15c0bc9-proxy-tls\") pod \"830a1fff-253d-45b7-8c19-b896f15c0bc9\" (UID: \"830a1fff-253d-45b7-8c19-b896f15c0bc9\") "
May 25 13:26:29.518681 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:26:29.518603    2580 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"success-200-isvc-db114-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/830a1fff-253d-45b7-8c19-b896f15c0bc9-success-200-isvc-db114-kube-rbac-proxy-sar-config\") pod \"830a1fff-253d-45b7-8c19-b896f15c0bc9\" (UID: \"830a1fff-253d-45b7-8c19-b896f15c0bc9\") "
May 25 13:26:29.518975 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:26:29.518950    2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/830a1fff-253d-45b7-8c19-b896f15c0bc9-success-200-isvc-db114-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "success-200-isvc-db114-kube-rbac-proxy-sar-config") pod "830a1fff-253d-45b7-8c19-b896f15c0bc9" (UID: "830a1fff-253d-45b7-8c19-b896f15c0bc9"). InnerVolumeSpecName "success-200-isvc-db114-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 13:26:29.520722 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:26:29.520699    2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/830a1fff-253d-45b7-8c19-b896f15c0bc9-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "830a1fff-253d-45b7-8c19-b896f15c0bc9" (UID: "830a1fff-253d-45b7-8c19-b896f15c0bc9"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 13:26:29.520796 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:26:29.520732    2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/830a1fff-253d-45b7-8c19-b896f15c0bc9-kube-api-access-8hcmp" (OuterVolumeSpecName: "kube-api-access-8hcmp") pod "830a1fff-253d-45b7-8c19-b896f15c0bc9" (UID: "830a1fff-253d-45b7-8c19-b896f15c0bc9"). InnerVolumeSpecName "kube-api-access-8hcmp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 13:26:29.619961 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:26:29.619929    2580 reconciler_common.go:299] "Volume detached for volume \"success-200-isvc-db114-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/830a1fff-253d-45b7-8c19-b896f15c0bc9-success-200-isvc-db114-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-103.ec2.internal\" DevicePath \"\""
May 25 13:26:29.619961 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:26:29.619957    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8hcmp\" (UniqueName: \"kubernetes.io/projected/830a1fff-253d-45b7-8c19-b896f15c0bc9-kube-api-access-8hcmp\") on node \"ip-10-0-135-103.ec2.internal\" DevicePath \"\""
May 25 13:26:29.619961 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:26:29.619967    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/830a1fff-253d-45b7-8c19-b896f15c0bc9-proxy-tls\") on node \"ip-10-0-135-103.ec2.internal\" DevicePath \"\""
May 25 13:26:29.845905 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:26:29.845872    2580 generic.go:358] "Generic (PLEG): container finished" podID="830a1fff-253d-45b7-8c19-b896f15c0bc9" containerID="e6f8e7e839b663edfc3658615af2f6c552c9da5376428fbed4de2366f298f7f4" exitCode=0
May 25 13:26:29.846336 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:26:29.845941    2580 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-db114-predictor-586c7dfd-52v5d"
May 25 13:26:29.846336 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:26:29.845952    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-db114-predictor-586c7dfd-52v5d" event={"ID":"830a1fff-253d-45b7-8c19-b896f15c0bc9","Type":"ContainerDied","Data":"e6f8e7e839b663edfc3658615af2f6c552c9da5376428fbed4de2366f298f7f4"}
May 25 13:26:29.846336 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:26:29.845990    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-db114-predictor-586c7dfd-52v5d" event={"ID":"830a1fff-253d-45b7-8c19-b896f15c0bc9","Type":"ContainerDied","Data":"4ef3bd40ff35b9779ca2de969c841c0e6800eb81315c5982a6d31e25a5d8019b"}
May 25 13:26:29.846336 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:26:29.846007    2580 scope.go:117] "RemoveContainer" containerID="f8e596ff22470a46f681c3094377c4d587ed88bf747c41d5dee74068b837bda5"
May 25 13:26:29.846565 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:26:29.846520    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-3e647-predictor-55bfd66d8d-pj8fl" podUID="ade8fe09-bea0-4b3d-a568-e52c3fa9fe0d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
May 25 13:26:29.854942 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:26:29.854917    2580 scope.go:117] "RemoveContainer" containerID="e6f8e7e839b663edfc3658615af2f6c552c9da5376428fbed4de2366f298f7f4"
May 25 13:26:29.862083 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:26:29.862066    2580 scope.go:117] "RemoveContainer" containerID="f8e596ff22470a46f681c3094377c4d587ed88bf747c41d5dee74068b837bda5"
May 25 13:26:29.862346 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:26:29.862323    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8e596ff22470a46f681c3094377c4d587ed88bf747c41d5dee74068b837bda5\": container with ID starting with f8e596ff22470a46f681c3094377c4d587ed88bf747c41d5dee74068b837bda5 not found: ID does not exist" containerID="f8e596ff22470a46f681c3094377c4d587ed88bf747c41d5dee74068b837bda5"
May 25 13:26:29.862467 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:26:29.862350    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8e596ff22470a46f681c3094377c4d587ed88bf747c41d5dee74068b837bda5"} err="failed to get container status \"f8e596ff22470a46f681c3094377c4d587ed88bf747c41d5dee74068b837bda5\": rpc error: code = NotFound desc = could not find container \"f8e596ff22470a46f681c3094377c4d587ed88bf747c41d5dee74068b837bda5\": container with ID starting with f8e596ff22470a46f681c3094377c4d587ed88bf747c41d5dee74068b837bda5 not found: ID does not exist"
May 25 13:26:29.862467 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:26:29.862391    2580 scope.go:117] "RemoveContainer" containerID="e6f8e7e839b663edfc3658615af2f6c552c9da5376428fbed4de2366f298f7f4"
May 25 13:26:29.862601 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:26:29.862583    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6f8e7e839b663edfc3658615af2f6c552c9da5376428fbed4de2366f298f7f4\": container with ID starting with e6f8e7e839b663edfc3658615af2f6c552c9da5376428fbed4de2366f298f7f4 not found: ID does not exist" containerID="e6f8e7e839b663edfc3658615af2f6c552c9da5376428fbed4de2366f298f7f4"
May 25 13:26:29.862655 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:26:29.862611    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6f8e7e839b663edfc3658615af2f6c552c9da5376428fbed4de2366f298f7f4"} err="failed to get container status \"e6f8e7e839b663edfc3658615af2f6c552c9da5376428fbed4de2366f298f7f4\": rpc error: code = NotFound desc = could not find container \"e6f8e7e839b663edfc3658615af2f6c552c9da5376428fbed4de2366f298f7f4\": container with ID starting with e6f8e7e839b663edfc3658615af2f6c552c9da5376428fbed4de2366f298f7f4 not found: ID does not exist"
May 25 13:26:29.867454 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:26:29.867343    2580 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-db114-predictor-586c7dfd-52v5d"]
May 25 13:26:29.878394 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:26:29.877688    2580 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-db114-predictor-586c7dfd-52v5d"]
May 25 13:26:30.378491 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:26:30.378459    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="830a1fff-253d-45b7-8c19-b896f15c0bc9" path="/var/lib/kubelet/pods/830a1fff-253d-45b7-8c19-b896f15c0bc9/volumes"
May 25 13:26:33.705149 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:26:33.705121    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-9ee57-predictor-767dfdd744-jlp7p"
May 25 13:26:34.850516 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:26:34.850488    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-3e647-predictor-55bfd66d8d-pj8fl"
May 25 13:26:34.851059 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:26:34.851030    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-3e647-predictor-55bfd66d8d-pj8fl" podUID="ade8fe09-bea0-4b3d-a568-e52c3fa9fe0d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
May 25 13:26:44.851107 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:26:44.851067    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-3e647-predictor-55bfd66d8d-pj8fl" podUID="ade8fe09-bea0-4b3d-a568-e52c3fa9fe0d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
May 25 13:26:54.851387 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:26:54.851329    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-3e647-predictor-55bfd66d8d-pj8fl" podUID="ade8fe09-bea0-4b3d-a568-e52c3fa9fe0d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
May 25 13:26:56.135145 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:26:56.135109    2580 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-9ee57-predictor-767dfdd744-jlp7p"]
May 25 13:26:56.135608 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:26:56.135477    2580 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-9ee57-predictor-767dfdd744-jlp7p" podUID="504712a1-9a41-4d39-a504-32f340cfa820" containerName="kserve-container" containerID="cri-o://566f3df6b8babe53ae8148a8b0ceb1b64eb142805ad36f5865a73f0eaa01d27a" gracePeriod=30
May 25 13:26:56.135867 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:26:56.135734    2580 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-9ee57-predictor-767dfdd744-jlp7p" podUID="504712a1-9a41-4d39-a504-32f340cfa820" containerName="kube-rbac-proxy" containerID="cri-o://002741c0204601391e10239277d7789ffa3ae5f0eadd871bb36518bb53e91e7d" gracePeriod=30
May 25 13:26:56.214794 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:26:56.214757    2580 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-72c4b-predictor-6f6fcbb-zg7jb"]
May 25 13:26:56.215209 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:26:56.215192    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="830a1fff-253d-45b7-8c19-b896f15c0bc9" containerName="kserve-container"
May 25 13:26:56.215310 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:26:56.215213    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="830a1fff-253d-45b7-8c19-b896f15c0bc9" containerName="kserve-container"
May 25 13:26:56.215310 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:26:56.215248    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="830a1fff-253d-45b7-8c19-b896f15c0bc9" containerName="kube-rbac-proxy"
May 25 13:26:56.215310 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:26:56.215256    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="830a1fff-253d-45b7-8c19-b896f15c0bc9" containerName="kube-rbac-proxy"
May 25 13:26:56.215514 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:26:56.215334    2580 memory_manager.go:356] "RemoveStaleState removing state" podUID="830a1fff-253d-45b7-8c19-b896f15c0bc9" containerName="kserve-container"
May 25 13:26:56.215514 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:26:56.215348    2580 memory_manager.go:356] "RemoveStaleState removing state" podUID="830a1fff-253d-45b7-8c19-b896f15c0bc9" containerName="kube-rbac-proxy"
May 25 13:26:56.218498 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:26:56.218477    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-72c4b-predictor-6f6fcbb-zg7jb"
May 25 13:26:56.220703 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:26:56.220682    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-72c4b-predictor-serving-cert\""
May 25 13:26:56.220799 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:26:56.220756    2580 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-72c4b-kube-rbac-proxy-sar-config\""
May 25 13:26:56.228318 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:26:56.228297    2580 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-72c4b-predictor-6f6fcbb-zg7jb"]
May 25 13:26:56.347846 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:26:56.347814    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lsvr7\" (UniqueName: \"kubernetes.io/projected/31d66af3-84e0-4725-a704-7d9532b0ffa9-kube-api-access-lsvr7\") pod \"success-200-isvc-72c4b-predictor-6f6fcbb-zg7jb\" (UID: \"31d66af3-84e0-4725-a704-7d9532b0ffa9\") " pod="kserve-ci-e2e-test/success-200-isvc-72c4b-predictor-6f6fcbb-zg7jb"
May 25 13:26:56.348015 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:26:56.347928    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"success-200-isvc-72c4b-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/31d66af3-84e0-4725-a704-7d9532b0ffa9-success-200-isvc-72c4b-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-72c4b-predictor-6f6fcbb-zg7jb\" (UID: \"31d66af3-84e0-4725-a704-7d9532b0ffa9\") " pod="kserve-ci-e2e-test/success-200-isvc-72c4b-predictor-6f6fcbb-zg7jb"
May 25 13:26:56.348015 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:26:56.347955    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/31d66af3-84e0-4725-a704-7d9532b0ffa9-proxy-tls\") pod \"success-200-isvc-72c4b-predictor-6f6fcbb-zg7jb\" (UID: \"31d66af3-84e0-4725-a704-7d9532b0ffa9\") " pod="kserve-ci-e2e-test/success-200-isvc-72c4b-predictor-6f6fcbb-zg7jb"
May 25 13:26:56.449286 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:26:56.449189    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"success-200-isvc-72c4b-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/31d66af3-84e0-4725-a704-7d9532b0ffa9-success-200-isvc-72c4b-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-72c4b-predictor-6f6fcbb-zg7jb\" (UID: \"31d66af3-84e0-4725-a704-7d9532b0ffa9\") " pod="kserve-ci-e2e-test/success-200-isvc-72c4b-predictor-6f6fcbb-zg7jb"
May 25 13:26:56.449286 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:26:56.449231    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/31d66af3-84e0-4725-a704-7d9532b0ffa9-proxy-tls\") pod \"success-200-isvc-72c4b-predictor-6f6fcbb-zg7jb\" (UID: \"31d66af3-84e0-4725-a704-7d9532b0ffa9\") " pod="kserve-ci-e2e-test/success-200-isvc-72c4b-predictor-6f6fcbb-zg7jb"
May 25 13:26:56.449286 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:26:56.449277    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-lsvr7\" (UniqueName: \"kubernetes.io/projected/31d66af3-84e0-4725-a704-7d9532b0ffa9-kube-api-access-lsvr7\") pod \"success-200-isvc-72c4b-predictor-6f6fcbb-zg7jb\" (UID: \"31d66af3-84e0-4725-a704-7d9532b0ffa9\") " pod="kserve-ci-e2e-test/success-200-isvc-72c4b-predictor-6f6fcbb-zg7jb"
May 25 13:26:56.449740 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:26:56.449446    2580 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-72c4b-predictor-serving-cert: secret "success-200-isvc-72c4b-predictor-serving-cert" not found
May 25 13:26:56.449740 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:26:56.449543    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/31d66af3-84e0-4725-a704-7d9532b0ffa9-proxy-tls podName:31d66af3-84e0-4725-a704-7d9532b0ffa9 nodeName:}" failed. No retries permitted until 2026-05-25 13:26:56.949520254 +0000 UTC m=+1317.141814606 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/31d66af3-84e0-4725-a704-7d9532b0ffa9-proxy-tls") pod "success-200-isvc-72c4b-predictor-6f6fcbb-zg7jb" (UID: "31d66af3-84e0-4725-a704-7d9532b0ffa9") : secret "success-200-isvc-72c4b-predictor-serving-cert" not found
May 25 13:26:56.449879 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:26:56.449859    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"success-200-isvc-72c4b-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/31d66af3-84e0-4725-a704-7d9532b0ffa9-success-200-isvc-72c4b-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-72c4b-predictor-6f6fcbb-zg7jb\" (UID: \"31d66af3-84e0-4725-a704-7d9532b0ffa9\") " pod="kserve-ci-e2e-test/success-200-isvc-72c4b-predictor-6f6fcbb-zg7jb"
May 25 13:26:56.458153 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:26:56.458121    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lsvr7\" (UniqueName: \"kubernetes.io/projected/31d66af3-84e0-4725-a704-7d9532b0ffa9-kube-api-access-lsvr7\") pod \"success-200-isvc-72c4b-predictor-6f6fcbb-zg7jb\" (UID: \"31d66af3-84e0-4725-a704-7d9532b0ffa9\") " pod="kserve-ci-e2e-test/success-200-isvc-72c4b-predictor-6f6fcbb-zg7jb"
May 25 13:26:56.942287 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:26:56.942250    2580 generic.go:358] "Generic (PLEG): container finished" podID="504712a1-9a41-4d39-a504-32f340cfa820" containerID="002741c0204601391e10239277d7789ffa3ae5f0eadd871bb36518bb53e91e7d" exitCode=2
May 25 13:26:56.942530 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:26:56.942321    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-9ee57-predictor-767dfdd744-jlp7p" event={"ID":"504712a1-9a41-4d39-a504-32f340cfa820","Type":"ContainerDied","Data":"002741c0204601391e10239277d7789ffa3ae5f0eadd871bb36518bb53e91e7d"}
May 25 13:26:56.952747 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:26:56.952717    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/31d66af3-84e0-4725-a704-7d9532b0ffa9-proxy-tls\") pod \"success-200-isvc-72c4b-predictor-6f6fcbb-zg7jb\" (UID: \"31d66af3-84e0-4725-a704-7d9532b0ffa9\") " pod="kserve-ci-e2e-test/success-200-isvc-72c4b-predictor-6f6fcbb-zg7jb"
May 25 13:26:56.955160 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:26:56.955138    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/31d66af3-84e0-4725-a704-7d9532b0ffa9-proxy-tls\") pod \"success-200-isvc-72c4b-predictor-6f6fcbb-zg7jb\" (UID: \"31d66af3-84e0-4725-a704-7d9532b0ffa9\") " pod="kserve-ci-e2e-test/success-200-isvc-72c4b-predictor-6f6fcbb-zg7jb"
May 25 13:26:57.129243 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:26:57.129205    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-72c4b-predictor-6f6fcbb-zg7jb"
May 25 13:26:57.259190 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:26:57.259168    2580 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-72c4b-predictor-6f6fcbb-zg7jb"]
May 25 13:26:57.261454 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:26:57.261427    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31d66af3_84e0_4725_a704_7d9532b0ffa9.slice/crio-057ed60b6802ca533eb28df34be512143ef19d464735fdcce39b6c5418f20715 WatchSource:0}: Error finding container 057ed60b6802ca533eb28df34be512143ef19d464735fdcce39b6c5418f20715: Status 404 returned error can't find the container with id 057ed60b6802ca533eb28df34be512143ef19d464735fdcce39b6c5418f20715
May 25 13:26:57.950957 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:26:57.950922    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-72c4b-predictor-6f6fcbb-zg7jb" event={"ID":"31d66af3-84e0-4725-a704-7d9532b0ffa9","Type":"ContainerStarted","Data":"b8254ba91edc2b4002421cb29d9c6125ebb03abfc81476c8a487cbb24cb0f354"}
May 25 13:26:57.950957 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:26:57.950965    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-72c4b-predictor-6f6fcbb-zg7jb" event={"ID":"31d66af3-84e0-4725-a704-7d9532b0ffa9","Type":"ContainerStarted","Data":"c122197eeb0b5817e8ceb0a230b7e55bfd22aff6575fa3a28587bf9bebd6d441"}
May 25 13:26:57.951213 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:26:57.950983    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-72c4b-predictor-6f6fcbb-zg7jb" event={"ID":"31d66af3-84e0-4725-a704-7d9532b0ffa9","Type":"ContainerStarted","Data":"057ed60b6802ca533eb28df34be512143ef19d464735fdcce39b6c5418f20715"}
May 25 13:26:57.951213 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:26:57.951091    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-72c4b-predictor-6f6fcbb-zg7jb"
May 25 13:26:57.951213 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:26:57.951131    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-72c4b-predictor-6f6fcbb-zg7jb"
May 25 13:26:57.952642 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:26:57.952611    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-72c4b-predictor-6f6fcbb-zg7jb" podUID="31d66af3-84e0-4725-a704-7d9532b0ffa9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
May 25 13:26:57.970300 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:26:57.970256    2580 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-72c4b-predictor-6f6fcbb-zg7jb" podStartSLOduration=1.970244162 podStartE2EDuration="1.970244162s" podCreationTimestamp="2026-05-25 13:26:56 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 13:26:57.96857769 +0000 UTC m=+1318.160872052" watchObservedRunningTime="2026-05-25 13:26:57.970244162 +0000 UTC m=+1318.162538525"
May 25 13:26:58.700080 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:26:58.700031    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-9ee57-predictor-767dfdd744-jlp7p" podUID="504712a1-9a41-4d39-a504-32f340cfa820" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.38:8643/healthz\": dial tcp 10.134.0.38:8643: connect: connection refused"
May 25 13:26:58.954256 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:26:58.954165    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-72c4b-predictor-6f6fcbb-zg7jb" podUID="31d66af3-84e0-4725-a704-7d9532b0ffa9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
May 25 13:26:59.583725 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:26:59.583702    2580 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-9ee57-predictor-767dfdd744-jlp7p"
May 25 13:26:59.672914 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:26:59.672836    2580 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-56h6c\" (UniqueName: \"kubernetes.io/projected/504712a1-9a41-4d39-a504-32f340cfa820-kube-api-access-56h6c\") pod \"504712a1-9a41-4d39-a504-32f340cfa820\" (UID: \"504712a1-9a41-4d39-a504-32f340cfa820\") "
May 25 13:26:59.672914 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:26:59.672882    2580 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/504712a1-9a41-4d39-a504-32f340cfa820-proxy-tls\") pod \"504712a1-9a41-4d39-a504-32f340cfa820\" (UID: \"504712a1-9a41-4d39-a504-32f340cfa820\") "
May 25 13:26:59.673112 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:26:59.672993    2580 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"success-200-isvc-9ee57-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/504712a1-9a41-4d39-a504-32f340cfa820-success-200-isvc-9ee57-kube-rbac-proxy-sar-config\") pod \"504712a1-9a41-4d39-a504-32f340cfa820\" (UID: \"504712a1-9a41-4d39-a504-32f340cfa820\") "
May 25 13:26:59.673390 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:26:59.673340    2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/504712a1-9a41-4d39-a504-32f340cfa820-success-200-isvc-9ee57-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "success-200-isvc-9ee57-kube-rbac-proxy-sar-config") pod "504712a1-9a41-4d39-a504-32f340cfa820" (UID: "504712a1-9a41-4d39-a504-32f340cfa820"). InnerVolumeSpecName "success-200-isvc-9ee57-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 13:26:59.675137 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:26:59.675109    2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/504712a1-9a41-4d39-a504-32f340cfa820-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "504712a1-9a41-4d39-a504-32f340cfa820" (UID: "504712a1-9a41-4d39-a504-32f340cfa820"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 13:26:59.675236 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:26:59.675134    2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/504712a1-9a41-4d39-a504-32f340cfa820-kube-api-access-56h6c" (OuterVolumeSpecName: "kube-api-access-56h6c") pod "504712a1-9a41-4d39-a504-32f340cfa820" (UID: "504712a1-9a41-4d39-a504-32f340cfa820"). InnerVolumeSpecName "kube-api-access-56h6c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 13:26:59.774024 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:26:59.773978    2580 reconciler_common.go:299] "Volume detached for volume \"success-200-isvc-9ee57-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/504712a1-9a41-4d39-a504-32f340cfa820-success-200-isvc-9ee57-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-103.ec2.internal\" DevicePath \"\""
May 25 13:26:59.774024 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:26:59.774019    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-56h6c\" (UniqueName: \"kubernetes.io/projected/504712a1-9a41-4d39-a504-32f340cfa820-kube-api-access-56h6c\") on node \"ip-10-0-135-103.ec2.internal\" DevicePath \"\""
May 25 13:26:59.774024 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:26:59.774033    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/504712a1-9a41-4d39-a504-32f340cfa820-proxy-tls\") on node \"ip-10-0-135-103.ec2.internal\" DevicePath \"\""
May 25 13:26:59.958466 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:26:59.958379    2580 generic.go:358] "Generic (PLEG): container finished" podID="504712a1-9a41-4d39-a504-32f340cfa820" containerID="566f3df6b8babe53ae8148a8b0ceb1b64eb142805ad36f5865a73f0eaa01d27a" exitCode=0
May 25 13:26:59.958466 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:26:59.958450    2580 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-9ee57-predictor-767dfdd744-jlp7p"
May 25 13:26:59.958706 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:26:59.958468    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-9ee57-predictor-767dfdd744-jlp7p" event={"ID":"504712a1-9a41-4d39-a504-32f340cfa820","Type":"ContainerDied","Data":"566f3df6b8babe53ae8148a8b0ceb1b64eb142805ad36f5865a73f0eaa01d27a"}
May 25 13:26:59.958706 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:26:59.958514    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-9ee57-predictor-767dfdd744-jlp7p" event={"ID":"504712a1-9a41-4d39-a504-32f340cfa820","Type":"ContainerDied","Data":"b3e973b9da6d07f644867a1e4bc04cadeaa89b2e4c827c3519bd64ac553f6c80"}
May 25 13:26:59.958706 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:26:59.958536    2580 scope.go:117] "RemoveContainer" containerID="002741c0204601391e10239277d7789ffa3ae5f0eadd871bb36518bb53e91e7d"
May 25 13:26:59.967146 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:26:59.967129    2580 scope.go:117] "RemoveContainer" containerID="566f3df6b8babe53ae8148a8b0ceb1b64eb142805ad36f5865a73f0eaa01d27a"
May 25 13:26:59.974267 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:26:59.974251    2580 scope.go:117] "RemoveContainer" containerID="002741c0204601391e10239277d7789ffa3ae5f0eadd871bb36518bb53e91e7d"
May 25 13:26:59.974515 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:26:59.974497    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"002741c0204601391e10239277d7789ffa3ae5f0eadd871bb36518bb53e91e7d\": container with ID starting with 002741c0204601391e10239277d7789ffa3ae5f0eadd871bb36518bb53e91e7d not found: ID does not exist" containerID="002741c0204601391e10239277d7789ffa3ae5f0eadd871bb36518bb53e91e7d"
May 25 13:26:59.974578 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:26:59.974524    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"002741c0204601391e10239277d7789ffa3ae5f0eadd871bb36518bb53e91e7d"} err="failed to get container status \"002741c0204601391e10239277d7789ffa3ae5f0eadd871bb36518bb53e91e7d\": rpc error: code = NotFound desc = could not find container \"002741c0204601391e10239277d7789ffa3ae5f0eadd871bb36518bb53e91e7d\": container with ID starting with 002741c0204601391e10239277d7789ffa3ae5f0eadd871bb36518bb53e91e7d not found: ID does not exist"
May 25 13:26:59.974578 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:26:59.974541    2580 scope.go:117] "RemoveContainer" containerID="566f3df6b8babe53ae8148a8b0ceb1b64eb142805ad36f5865a73f0eaa01d27a"
May 25 13:26:59.974796 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:26:59.974775    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"566f3df6b8babe53ae8148a8b0ceb1b64eb142805ad36f5865a73f0eaa01d27a\": container with ID starting with 566f3df6b8babe53ae8148a8b0ceb1b64eb142805ad36f5865a73f0eaa01d27a not found: ID does not exist" containerID="566f3df6b8babe53ae8148a8b0ceb1b64eb142805ad36f5865a73f0eaa01d27a"
May 25 13:26:59.974853 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:26:59.974801    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"566f3df6b8babe53ae8148a8b0ceb1b64eb142805ad36f5865a73f0eaa01d27a"} err="failed to get container status \"566f3df6b8babe53ae8148a8b0ceb1b64eb142805ad36f5865a73f0eaa01d27a\": rpc error: code = NotFound desc = could not find container \"566f3df6b8babe53ae8148a8b0ceb1b64eb142805ad36f5865a73f0eaa01d27a\": container with ID starting with 566f3df6b8babe53ae8148a8b0ceb1b64eb142805ad36f5865a73f0eaa01d27a not found: ID does not exist"
May 25 13:26:59.980449 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:26:59.980428    2580 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-9ee57-predictor-767dfdd744-jlp7p"]
May 25 13:26:59.985496 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:26:59.985477    2580 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-9ee57-predictor-767dfdd744-jlp7p"]
May 25 13:27:00.378444 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:27:00.378414    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="504712a1-9a41-4d39-a504-32f340cfa820" path="/var/lib/kubelet/pods/504712a1-9a41-4d39-a504-32f340cfa820/volumes"
May 25 13:27:03.959297 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:27:03.959268    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-72c4b-predictor-6f6fcbb-zg7jb"
May 25 13:27:03.959800 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:27:03.959772    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-72c4b-predictor-6f6fcbb-zg7jb" podUID="31d66af3-84e0-4725-a704-7d9532b0ffa9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
May 25 13:27:04.851744 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:27:04.851702    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-3e647-predictor-55bfd66d8d-pj8fl" podUID="ade8fe09-bea0-4b3d-a568-e52c3fa9fe0d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
May 25 13:27:13.960777 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:27:13.960696    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-72c4b-predictor-6f6fcbb-zg7jb" podUID="31d66af3-84e0-4725-a704-7d9532b0ffa9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
May 25 13:27:14.851539 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:27:14.851505    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-3e647-predictor-55bfd66d8d-pj8fl"
May 25 13:27:23.960594 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:27:23.960551    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-72c4b-predictor-6f6fcbb-zg7jb" podUID="31d66af3-84e0-4725-a704-7d9532b0ffa9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
May 25 13:27:33.959807 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:27:33.959757    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-72c4b-predictor-6f6fcbb-zg7jb" podUID="31d66af3-84e0-4725-a704-7d9532b0ffa9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
May 25 13:27:43.961180 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:27:43.961147    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-72c4b-predictor-6f6fcbb-zg7jb"
May 25 13:30:00.359313 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:30:00.359283    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-646r6_59b905b8-5ce3-49f6-b47c-b5b3a525fe06/ovn-acl-logging/0.log"
May 25 13:30:00.364424 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:30:00.364404    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-646r6_59b905b8-5ce3-49f6-b47c-b5b3a525fe06/ovn-acl-logging/0.log"
May 25 13:35:00.385585 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:35:00.385557    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-646r6_59b905b8-5ce3-49f6-b47c-b5b3a525fe06/ovn-acl-logging/0.log"
May 25 13:35:00.390667 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:35:00.390647    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-646r6_59b905b8-5ce3-49f6-b47c-b5b3a525fe06/ovn-acl-logging/0.log"
May 25 13:35:41.007018 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:35:41.006978    2580 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-3e647-predictor-55bfd66d8d-pj8fl"]
May 25 13:35:41.007609 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:35:41.007241    2580 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-3e647-predictor-55bfd66d8d-pj8fl" podUID="ade8fe09-bea0-4b3d-a568-e52c3fa9fe0d" containerName="kserve-container" containerID="cri-o://0fec265aa718d1244f7ebaba9fc08acc343520d17c7b19f95175c361f88f8db1" gracePeriod=30
May 25 13:35:41.007609 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:35:41.007292    2580 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-3e647-predictor-55bfd66d8d-pj8fl" podUID="ade8fe09-bea0-4b3d-a568-e52c3fa9fe0d" containerName="kube-rbac-proxy" containerID="cri-o://eafa8b9485f996d825705d71857bba851e4509e7a6a9f487bba089618afd9618" gracePeriod=30
May 25 13:35:41.124944 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:35:41.124910    2580 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-35687-predictor-6d599c8c48-68thq"]
May 25 13:35:41.125495 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:35:41.125474    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="504712a1-9a41-4d39-a504-32f340cfa820" containerName="kserve-container"
May 25 13:35:41.125576 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:35:41.125499    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="504712a1-9a41-4d39-a504-32f340cfa820" containerName="kserve-container"
May 25 13:35:41.125576 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:35:41.125516    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="504712a1-9a41-4d39-a504-32f340cfa820" containerName="kube-rbac-proxy"
May 25 13:35:41.125576 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:35:41.125525    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="504712a1-9a41-4d39-a504-32f340cfa820" containerName="kube-rbac-proxy"
May 25 13:35:41.125744 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:35:41.125638    2580 memory_manager.go:356] "RemoveStaleState removing state" podUID="504712a1-9a41-4d39-a504-32f340cfa820" containerName="kserve-container"
May 25 13:35:41.125744 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:35:41.125650    2580 memory_manager.go:356] "RemoveStaleState removing state" podUID="504712a1-9a41-4d39-a504-32f340cfa820" containerName="kube-rbac-proxy"
May 25 13:35:41.129011 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:35:41.128993    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-35687-predictor-6d599c8c48-68thq"
May 25 13:35:41.131447 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:35:41.131422    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-35687-predictor-serving-cert\""
May 25 13:35:41.131646 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:35:41.131618    2580 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-35687-kube-rbac-proxy-sar-config\""
May 25 13:35:41.146278 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:35:41.146252    2580 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-35687-predictor-6d599c8c48-68thq"]
May 25 13:35:41.164412 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:35:41.164346    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"success-200-isvc-35687-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1fbb8b99-22aa-4a70-87f2-e46d72c808f6-success-200-isvc-35687-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-35687-predictor-6d599c8c48-68thq\" (UID: \"1fbb8b99-22aa-4a70-87f2-e46d72c808f6\") " pod="kserve-ci-e2e-test/success-200-isvc-35687-predictor-6d599c8c48-68thq"
May 25 13:35:41.164573 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:35:41.164424    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1fbb8b99-22aa-4a70-87f2-e46d72c808f6-proxy-tls\") pod \"success-200-isvc-35687-predictor-6d599c8c48-68thq\" (UID: \"1fbb8b99-22aa-4a70-87f2-e46d72c808f6\") " pod="kserve-ci-e2e-test/success-200-isvc-35687-predictor-6d599c8c48-68thq"
May 25 13:35:41.164573 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:35:41.164460    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tmcf\" (UniqueName: \"kubernetes.io/projected/1fbb8b99-22aa-4a70-87f2-e46d72c808f6-kube-api-access-5tmcf\") pod \"success-200-isvc-35687-predictor-6d599c8c48-68thq\" (UID: \"1fbb8b99-22aa-4a70-87f2-e46d72c808f6\") " pod="kserve-ci-e2e-test/success-200-isvc-35687-predictor-6d599c8c48-68thq"
May 25 13:35:41.265318 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:35:41.265236    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"success-200-isvc-35687-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1fbb8b99-22aa-4a70-87f2-e46d72c808f6-success-200-isvc-35687-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-35687-predictor-6d599c8c48-68thq\" (UID: \"1fbb8b99-22aa-4a70-87f2-e46d72c808f6\") " pod="kserve-ci-e2e-test/success-200-isvc-35687-predictor-6d599c8c48-68thq"
May 25 13:35:41.265318 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:35:41.265274    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1fbb8b99-22aa-4a70-87f2-e46d72c808f6-proxy-tls\") pod \"success-200-isvc-35687-predictor-6d599c8c48-68thq\" (UID: \"1fbb8b99-22aa-4a70-87f2-e46d72c808f6\") " pod="kserve-ci-e2e-test/success-200-isvc-35687-predictor-6d599c8c48-68thq"
May 25 13:35:41.265318 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:35:41.265311    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-5tmcf\" (UniqueName: \"kubernetes.io/projected/1fbb8b99-22aa-4a70-87f2-e46d72c808f6-kube-api-access-5tmcf\") pod \"success-200-isvc-35687-predictor-6d599c8c48-68thq\" (UID: \"1fbb8b99-22aa-4a70-87f2-e46d72c808f6\") " pod="kserve-ci-e2e-test/success-200-isvc-35687-predictor-6d599c8c48-68thq"
May 25 13:35:41.265915 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:35:41.265885    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"success-200-isvc-35687-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1fbb8b99-22aa-4a70-87f2-e46d72c808f6-success-200-isvc-35687-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-35687-predictor-6d599c8c48-68thq\" (UID: \"1fbb8b99-22aa-4a70-87f2-e46d72c808f6\") " pod="kserve-ci-e2e-test/success-200-isvc-35687-predictor-6d599c8c48-68thq"
May 25 13:35:41.267856 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:35:41.267833    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1fbb8b99-22aa-4a70-87f2-e46d72c808f6-proxy-tls\") pod \"success-200-isvc-35687-predictor-6d599c8c48-68thq\" (UID: \"1fbb8b99-22aa-4a70-87f2-e46d72c808f6\") " pod="kserve-ci-e2e-test/success-200-isvc-35687-predictor-6d599c8c48-68thq"
May 25 13:35:41.274075 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:35:41.274052    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tmcf\" (UniqueName: \"kubernetes.io/projected/1fbb8b99-22aa-4a70-87f2-e46d72c808f6-kube-api-access-5tmcf\") pod \"success-200-isvc-35687-predictor-6d599c8c48-68thq\" (UID: \"1fbb8b99-22aa-4a70-87f2-e46d72c808f6\") " pod="kserve-ci-e2e-test/success-200-isvc-35687-predictor-6d599c8c48-68thq"
May 25 13:35:41.439889 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:35:41.439852    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-35687-predictor-6d599c8c48-68thq"
May 25 13:35:41.573723 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:35:41.573699    2580 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-35687-predictor-6d599c8c48-68thq"]
May 25 13:35:41.576030 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:35:41.576001    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fbb8b99_22aa_4a70_87f2_e46d72c808f6.slice/crio-2d09f52f3659689915356054063cfced902cfa177dd58b61e0f3202c32db03f5 WatchSource:0}: Error finding container 2d09f52f3659689915356054063cfced902cfa177dd58b61e0f3202c32db03f5: Status 404 returned error can't find the container with id 2d09f52f3659689915356054063cfced902cfa177dd58b61e0f3202c32db03f5
May 25 13:35:41.577696 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:35:41.577681    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 25 13:35:41.711848 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:35:41.711811    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-35687-predictor-6d599c8c48-68thq" event={"ID":"1fbb8b99-22aa-4a70-87f2-e46d72c808f6","Type":"ContainerStarted","Data":"308e4cfa263f957f07be34bb426dd4047a01264f502e15d3910360e6a5737cff"}
May 25 13:35:41.711970 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:35:41.711858    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-35687-predictor-6d599c8c48-68thq" event={"ID":"1fbb8b99-22aa-4a70-87f2-e46d72c808f6","Type":"ContainerStarted","Data":"7accd97455cdbb729594c456760dd86e65f479e278ce01a2c4142c1551c6dd02"}
May 25 13:35:41.711970 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:35:41.711875    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-35687-predictor-6d599c8c48-68thq" event={"ID":"1fbb8b99-22aa-4a70-87f2-e46d72c808f6","Type":"ContainerStarted","Data":"2d09f52f3659689915356054063cfced902cfa177dd58b61e0f3202c32db03f5"}
May 25 13:35:41.711970 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:35:41.711892    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-35687-predictor-6d599c8c48-68thq"
May 25 13:35:41.713780 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:35:41.713755    2580 generic.go:358] "Generic (PLEG): container finished" podID="ade8fe09-bea0-4b3d-a568-e52c3fa9fe0d" containerID="eafa8b9485f996d825705d71857bba851e4509e7a6a9f487bba089618afd9618" exitCode=2
May 25 13:35:41.713896 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:35:41.713784    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-3e647-predictor-55bfd66d8d-pj8fl" event={"ID":"ade8fe09-bea0-4b3d-a568-e52c3fa9fe0d","Type":"ContainerDied","Data":"eafa8b9485f996d825705d71857bba851e4509e7a6a9f487bba089618afd9618"}
May 25 13:35:41.730479 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:35:41.730434    2580 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-35687-predictor-6d599c8c48-68thq" podStartSLOduration=0.730419936 podStartE2EDuration="730.419936ms" podCreationTimestamp="2026-05-25 13:35:41 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 13:35:41.728739504 +0000 UTC m=+1841.921033864" watchObservedRunningTime="2026-05-25 13:35:41.730419936 +0000 UTC m=+1841.922714369"
May 25 13:35:42.719891 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:35:42.719855    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-35687-predictor-6d599c8c48-68thq"
May 25 13:35:42.720948 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:35:42.720919    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-35687-predictor-6d599c8c48-68thq" podUID="1fbb8b99-22aa-4a70-87f2-e46d72c808f6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
May 25 13:35:43.723664 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:35:43.723622    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-35687-predictor-6d599c8c48-68thq" podUID="1fbb8b99-22aa-4a70-87f2-e46d72c808f6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
May 25 13:35:44.058786 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:35:44.058763    2580 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-3e647-predictor-55bfd66d8d-pj8fl"
May 25 13:35:44.090305 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:35:44.090273    2580 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"success-200-isvc-3e647-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ade8fe09-bea0-4b3d-a568-e52c3fa9fe0d-success-200-isvc-3e647-kube-rbac-proxy-sar-config\") pod \"ade8fe09-bea0-4b3d-a568-e52c3fa9fe0d\" (UID: \"ade8fe09-bea0-4b3d-a568-e52c3fa9fe0d\") "
May 25 13:35:44.090490 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:35:44.090330    2580 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ade8fe09-bea0-4b3d-a568-e52c3fa9fe0d-proxy-tls\") pod \"ade8fe09-bea0-4b3d-a568-e52c3fa9fe0d\" (UID: \"ade8fe09-bea0-4b3d-a568-e52c3fa9fe0d\") "
May 25 13:35:44.090490 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:35:44.090361    2580 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lpm89\" (UniqueName: \"kubernetes.io/projected/ade8fe09-bea0-4b3d-a568-e52c3fa9fe0d-kube-api-access-lpm89\") pod \"ade8fe09-bea0-4b3d-a568-e52c3fa9fe0d\" (UID: \"ade8fe09-bea0-4b3d-a568-e52c3fa9fe0d\") "
May 25 13:35:44.090713 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:35:44.090680    2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ade8fe09-bea0-4b3d-a568-e52c3fa9fe0d-success-200-isvc-3e647-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "success-200-isvc-3e647-kube-rbac-proxy-sar-config") pod "ade8fe09-bea0-4b3d-a568-e52c3fa9fe0d" (UID: "ade8fe09-bea0-4b3d-a568-e52c3fa9fe0d"). InnerVolumeSpecName "success-200-isvc-3e647-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 13:35:44.092560 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:35:44.092526    2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ade8fe09-bea0-4b3d-a568-e52c3fa9fe0d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ade8fe09-bea0-4b3d-a568-e52c3fa9fe0d" (UID: "ade8fe09-bea0-4b3d-a568-e52c3fa9fe0d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 13:35:44.092744 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:35:44.092722    2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ade8fe09-bea0-4b3d-a568-e52c3fa9fe0d-kube-api-access-lpm89" (OuterVolumeSpecName: "kube-api-access-lpm89") pod "ade8fe09-bea0-4b3d-a568-e52c3fa9fe0d" (UID: "ade8fe09-bea0-4b3d-a568-e52c3fa9fe0d"). InnerVolumeSpecName "kube-api-access-lpm89". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 13:35:44.191802 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:35:44.191760    2580 reconciler_common.go:299] "Volume detached for volume \"success-200-isvc-3e647-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ade8fe09-bea0-4b3d-a568-e52c3fa9fe0d-success-200-isvc-3e647-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-103.ec2.internal\" DevicePath \"\""
May 25 13:35:44.191802 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:35:44.191804    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ade8fe09-bea0-4b3d-a568-e52c3fa9fe0d-proxy-tls\") on node \"ip-10-0-135-103.ec2.internal\" DevicePath \"\""
May 25 13:35:44.192001 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:35:44.191819    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lpm89\" (UniqueName: \"kubernetes.io/projected/ade8fe09-bea0-4b3d-a568-e52c3fa9fe0d-kube-api-access-lpm89\") on node \"ip-10-0-135-103.ec2.internal\" DevicePath \"\""
May 25 13:35:44.728231 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:35:44.728140    2580 generic.go:358] "Generic (PLEG): container finished" podID="ade8fe09-bea0-4b3d-a568-e52c3fa9fe0d" containerID="0fec265aa718d1244f7ebaba9fc08acc343520d17c7b19f95175c361f88f8db1" exitCode=0
May 25 13:35:44.728708 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:35:44.728227    2580 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-3e647-predictor-55bfd66d8d-pj8fl"
May 25 13:35:44.728708 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:35:44.728226    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-3e647-predictor-55bfd66d8d-pj8fl" event={"ID":"ade8fe09-bea0-4b3d-a568-e52c3fa9fe0d","Type":"ContainerDied","Data":"0fec265aa718d1244f7ebaba9fc08acc343520d17c7b19f95175c361f88f8db1"}
May 25 13:35:44.728708 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:35:44.728274    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-3e647-predictor-55bfd66d8d-pj8fl" event={"ID":"ade8fe09-bea0-4b3d-a568-e52c3fa9fe0d","Type":"ContainerDied","Data":"9fc87b70ab2ef5fb41e9a6d6ee969adf0a5c6a9157076d71985919a7a72af9da"}
May 25 13:35:44.728708 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:35:44.728290    2580 scope.go:117] "RemoveContainer" containerID="eafa8b9485f996d825705d71857bba851e4509e7a6a9f487bba089618afd9618"
May 25 13:35:44.736558 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:35:44.736538    2580 scope.go:117] "RemoveContainer" containerID="0fec265aa718d1244f7ebaba9fc08acc343520d17c7b19f95175c361f88f8db1"
May 25 13:35:44.743599 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:35:44.743561    2580 scope.go:117] "RemoveContainer" containerID="eafa8b9485f996d825705d71857bba851e4509e7a6a9f487bba089618afd9618"
May 25 13:35:44.743894 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:35:44.743870    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eafa8b9485f996d825705d71857bba851e4509e7a6a9f487bba089618afd9618\": container with ID starting with eafa8b9485f996d825705d71857bba851e4509e7a6a9f487bba089618afd9618 not found: ID does not exist" containerID="eafa8b9485f996d825705d71857bba851e4509e7a6a9f487bba089618afd9618"
May 25 13:35:44.743985 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:35:44.743906    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eafa8b9485f996d825705d71857bba851e4509e7a6a9f487bba089618afd9618"} err="failed to get container status \"eafa8b9485f996d825705d71857bba851e4509e7a6a9f487bba089618afd9618\": rpc error: code = NotFound desc = could not find container \"eafa8b9485f996d825705d71857bba851e4509e7a6a9f487bba089618afd9618\": container with ID starting with eafa8b9485f996d825705d71857bba851e4509e7a6a9f487bba089618afd9618 not found: ID does not exist"
May 25 13:35:44.743985 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:35:44.743931    2580 scope.go:117] "RemoveContainer" containerID="0fec265aa718d1244f7ebaba9fc08acc343520d17c7b19f95175c361f88f8db1"
May 25 13:35:44.744329 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:35:44.744211    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0fec265aa718d1244f7ebaba9fc08acc343520d17c7b19f95175c361f88f8db1\": container with ID starting with 0fec265aa718d1244f7ebaba9fc08acc343520d17c7b19f95175c361f88f8db1 not found: ID does not exist" containerID="0fec265aa718d1244f7ebaba9fc08acc343520d17c7b19f95175c361f88f8db1"
May 25 13:35:44.744329 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:35:44.744264    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0fec265aa718d1244f7ebaba9fc08acc343520d17c7b19f95175c361f88f8db1"} err="failed to get container status \"0fec265aa718d1244f7ebaba9fc08acc343520d17c7b19f95175c361f88f8db1\": rpc error: code = NotFound desc = could not find container \"0fec265aa718d1244f7ebaba9fc08acc343520d17c7b19f95175c361f88f8db1\": container with ID starting with 0fec265aa718d1244f7ebaba9fc08acc343520d17c7b19f95175c361f88f8db1 not found: ID does not exist"
May 25 13:35:44.745636 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:35:44.745616    2580 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-3e647-predictor-55bfd66d8d-pj8fl"]
May 25 13:35:44.749086 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:35:44.749063    2580 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-3e647-predictor-55bfd66d8d-pj8fl"]
May 25 13:35:46.377454 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:35:46.377415    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ade8fe09-bea0-4b3d-a568-e52c3fa9fe0d" path="/var/lib/kubelet/pods/ade8fe09-bea0-4b3d-a568-e52c3fa9fe0d/volumes"
May 25 13:35:48.728108 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:35:48.728077    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-35687-predictor-6d599c8c48-68thq"
May 25 13:35:48.728717 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:35:48.728677    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-35687-predictor-6d599c8c48-68thq" podUID="1fbb8b99-22aa-4a70-87f2-e46d72c808f6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
May 25 13:35:58.729050 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:35:58.729011    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-35687-predictor-6d599c8c48-68thq" podUID="1fbb8b99-22aa-4a70-87f2-e46d72c808f6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
May 25 13:36:08.729224 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:36:08.729181    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-35687-predictor-6d599c8c48-68thq" podUID="1fbb8b99-22aa-4a70-87f2-e46d72c808f6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
May 25 13:36:11.060683 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:36:11.060642    2580 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-72c4b-predictor-6f6fcbb-zg7jb"]
May 25 13:36:11.061116 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:36:11.060888    2580 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-72c4b-predictor-6f6fcbb-zg7jb" podUID="31d66af3-84e0-4725-a704-7d9532b0ffa9" containerName="kserve-container" containerID="cri-o://c122197eeb0b5817e8ceb0a230b7e55bfd22aff6575fa3a28587bf9bebd6d441" gracePeriod=30
May 25 13:36:11.061116 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:36:11.060922    2580 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-72c4b-predictor-6f6fcbb-zg7jb" podUID="31d66af3-84e0-4725-a704-7d9532b0ffa9" containerName="kube-rbac-proxy" containerID="cri-o://b8254ba91edc2b4002421cb29d9c6125ebb03abfc81476c8a487cbb24cb0f354" gracePeriod=30
May 25 13:36:11.109807 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:36:11.109772    2580 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-12313-predictor-857dcf8cc4-46rl6"]
May 25 13:36:11.110116 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:36:11.110104    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ade8fe09-bea0-4b3d-a568-e52c3fa9fe0d" containerName="kube-rbac-proxy"
May 25 13:36:11.110174 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:36:11.110118    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="ade8fe09-bea0-4b3d-a568-e52c3fa9fe0d" containerName="kube-rbac-proxy"
May 25 13:36:11.110174 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:36:11.110129    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ade8fe09-bea0-4b3d-a568-e52c3fa9fe0d" containerName="kserve-container"
May 25 13:36:11.110174 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:36:11.110134    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="ade8fe09-bea0-4b3d-a568-e52c3fa9fe0d" containerName="kserve-container"
May 25 13:36:11.110268 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:36:11.110202    2580 memory_manager.go:356] "RemoveStaleState removing state" podUID="ade8fe09-bea0-4b3d-a568-e52c3fa9fe0d" containerName="kube-rbac-proxy"
May 25 13:36:11.110268 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:36:11.110211    2580 memory_manager.go:356] "RemoveStaleState removing state" podUID="ade8fe09-bea0-4b3d-a568-e52c3fa9fe0d" containerName="kserve-container"
May 25 13:36:11.114450 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:36:11.114433    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-12313-predictor-857dcf8cc4-46rl6"
May 25 13:36:11.118167 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:36:11.118143    2580 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-12313-kube-rbac-proxy-sar-config\""
May 25 13:36:11.118316 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:36:11.118294    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-12313-predictor-serving-cert\""
May 25 13:36:11.124654 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:36:11.124625    2580 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-12313-predictor-857dcf8cc4-46rl6"]
May 25 13:36:11.220722 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:36:11.220679    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/51711b09-0032-45d2-99eb-00e0eff727e0-proxy-tls\") pod \"success-200-isvc-12313-predictor-857dcf8cc4-46rl6\" (UID: \"51711b09-0032-45d2-99eb-00e0eff727e0\") " pod="kserve-ci-e2e-test/success-200-isvc-12313-predictor-857dcf8cc4-46rl6"
May 25 13:36:11.220909 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:36:11.220744    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jx86g\" (UniqueName: \"kubernetes.io/projected/51711b09-0032-45d2-99eb-00e0eff727e0-kube-api-access-jx86g\") pod \"success-200-isvc-12313-predictor-857dcf8cc4-46rl6\" (UID: \"51711b09-0032-45d2-99eb-00e0eff727e0\") " pod="kserve-ci-e2e-test/success-200-isvc-12313-predictor-857dcf8cc4-46rl6"
May 25 13:36:11.220909 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:36:11.220785    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"success-200-isvc-12313-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/51711b09-0032-45d2-99eb-00e0eff727e0-success-200-isvc-12313-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-12313-predictor-857dcf8cc4-46rl6\" (UID: \"51711b09-0032-45d2-99eb-00e0eff727e0\") " pod="kserve-ci-e2e-test/success-200-isvc-12313-predictor-857dcf8cc4-46rl6"
May 25 13:36:11.321444 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:36:11.321337    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-jx86g\" (UniqueName: \"kubernetes.io/projected/51711b09-0032-45d2-99eb-00e0eff727e0-kube-api-access-jx86g\") pod \"success-200-isvc-12313-predictor-857dcf8cc4-46rl6\" (UID: \"51711b09-0032-45d2-99eb-00e0eff727e0\") " pod="kserve-ci-e2e-test/success-200-isvc-12313-predictor-857dcf8cc4-46rl6"
May 25 13:36:11.321444 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:36:11.321409    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"success-200-isvc-12313-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/51711b09-0032-45d2-99eb-00e0eff727e0-success-200-isvc-12313-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-12313-predictor-857dcf8cc4-46rl6\" (UID: \"51711b09-0032-45d2-99eb-00e0eff727e0\") " pod="kserve-ci-e2e-test/success-200-isvc-12313-predictor-857dcf8cc4-46rl6"
May 25 13:36:11.321670 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:36:11.321470    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/51711b09-0032-45d2-99eb-00e0eff727e0-proxy-tls\") pod \"success-200-isvc-12313-predictor-857dcf8cc4-46rl6\" (UID: \"51711b09-0032-45d2-99eb-00e0eff727e0\") " pod="kserve-ci-e2e-test/success-200-isvc-12313-predictor-857dcf8cc4-46rl6"
May 25 13:36:11.322068 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:36:11.322047    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"success-200-isvc-12313-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/51711b09-0032-45d2-99eb-00e0eff727e0-success-200-isvc-12313-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-12313-predictor-857dcf8cc4-46rl6\" (UID: \"51711b09-0032-45d2-99eb-00e0eff727e0\") " pod="kserve-ci-e2e-test/success-200-isvc-12313-predictor-857dcf8cc4-46rl6"
May 25 13:36:11.323985 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:36:11.323960    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/51711b09-0032-45d2-99eb-00e0eff727e0-proxy-tls\") pod \"success-200-isvc-12313-predictor-857dcf8cc4-46rl6\" (UID: \"51711b09-0032-45d2-99eb-00e0eff727e0\") " pod="kserve-ci-e2e-test/success-200-isvc-12313-predictor-857dcf8cc4-46rl6"
May 25 13:36:11.329657 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:36:11.329633    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jx86g\" (UniqueName: \"kubernetes.io/projected/51711b09-0032-45d2-99eb-00e0eff727e0-kube-api-access-jx86g\") pod \"success-200-isvc-12313-predictor-857dcf8cc4-46rl6\" (UID: \"51711b09-0032-45d2-99eb-00e0eff727e0\") " pod="kserve-ci-e2e-test/success-200-isvc-12313-predictor-857dcf8cc4-46rl6"
May 25 13:36:11.425678 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:36:11.425633    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-12313-predictor-857dcf8cc4-46rl6"
May 25 13:36:11.561274 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:36:11.561250    2580 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-12313-predictor-857dcf8cc4-46rl6"]
May 25 13:36:11.563405 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:36:11.563355    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51711b09_0032_45d2_99eb_00e0eff727e0.slice/crio-14cd739922656c225a280b0c1d93111e8d3d7159e45bb06955b618554c3ec584 WatchSource:0}: Error finding container 14cd739922656c225a280b0c1d93111e8d3d7159e45bb06955b618554c3ec584: Status 404 returned error can't find the container with id 14cd739922656c225a280b0c1d93111e8d3d7159e45bb06955b618554c3ec584
May 25 13:36:11.824935 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:36:11.824891    2580 generic.go:358] "Generic (PLEG): container finished" podID="31d66af3-84e0-4725-a704-7d9532b0ffa9" containerID="b8254ba91edc2b4002421cb29d9c6125ebb03abfc81476c8a487cbb24cb0f354" exitCode=2
May 25 13:36:11.825102 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:36:11.824973    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-72c4b-predictor-6f6fcbb-zg7jb" event={"ID":"31d66af3-84e0-4725-a704-7d9532b0ffa9","Type":"ContainerDied","Data":"b8254ba91edc2b4002421cb29d9c6125ebb03abfc81476c8a487cbb24cb0f354"}
May 25 13:36:11.826529 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:36:11.826506    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-12313-predictor-857dcf8cc4-46rl6" event={"ID":"51711b09-0032-45d2-99eb-00e0eff727e0","Type":"ContainerStarted","Data":"5ff3ffddb10a8dd1b122bb5f13d0847ec320d03f94fdc5e1a5ae8e49fad8027c"}
May 25 13:36:11.826529 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:36:11.826536    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-12313-predictor-857dcf8cc4-46rl6" event={"ID":"51711b09-0032-45d2-99eb-00e0eff727e0","Type":"ContainerStarted","Data":"514be88f1d3d8daa239e43fb4acdfc0f6bf79f4508c61bb8fd03345a1ced237a"}
May 25 13:36:11.826707 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:36:11.826550    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-12313-predictor-857dcf8cc4-46rl6" event={"ID":"51711b09-0032-45d2-99eb-00e0eff727e0","Type":"ContainerStarted","Data":"14cd739922656c225a280b0c1d93111e8d3d7159e45bb06955b618554c3ec584"}
May 25 13:36:11.826707 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:36:11.826692    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-12313-predictor-857dcf8cc4-46rl6"
May 25 13:36:11.846398 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:36:11.846304    2580 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-12313-predictor-857dcf8cc4-46rl6" podStartSLOduration=0.846293756 podStartE2EDuration="846.293756ms" podCreationTimestamp="2026-05-25 13:36:11 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 13:36:11.845389955 +0000 UTC m=+1872.037684316" watchObservedRunningTime="2026-05-25 13:36:11.846293756 +0000 UTC m=+1872.038588116"
May 25 13:36:12.830464 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:36:12.830385    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-12313-predictor-857dcf8cc4-46rl6"
May 25 13:36:12.831768 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:36:12.831736    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-12313-predictor-857dcf8cc4-46rl6" podUID="51711b09-0032-45d2-99eb-00e0eff727e0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
May 25 13:36:13.833063 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:36:13.833022    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-12313-predictor-857dcf8cc4-46rl6" podUID="51711b09-0032-45d2-99eb-00e0eff727e0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
May 25 13:36:13.954733 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:36:13.954688    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-72c4b-predictor-6f6fcbb-zg7jb" podUID="31d66af3-84e0-4725-a704-7d9532b0ffa9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.40:8643/healthz\": dial tcp 10.134.0.40:8643: connect: connection refused"
May 25 13:36:13.960107 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:36:13.960081    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-72c4b-predictor-6f6fcbb-zg7jb" podUID="31d66af3-84e0-4725-a704-7d9532b0ffa9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
May 25 13:36:14.308491 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:36:14.308467    2580 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-72c4b-predictor-6f6fcbb-zg7jb"
May 25 13:36:14.449923 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:36:14.449820    2580 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/31d66af3-84e0-4725-a704-7d9532b0ffa9-proxy-tls\") pod \"31d66af3-84e0-4725-a704-7d9532b0ffa9\" (UID: \"31d66af3-84e0-4725-a704-7d9532b0ffa9\") "
May 25 13:36:14.449923 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:36:14.449884    2580 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"success-200-isvc-72c4b-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/31d66af3-84e0-4725-a704-7d9532b0ffa9-success-200-isvc-72c4b-kube-rbac-proxy-sar-config\") pod \"31d66af3-84e0-4725-a704-7d9532b0ffa9\" (UID: \"31d66af3-84e0-4725-a704-7d9532b0ffa9\") "
May 25 13:36:14.450171 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:36:14.449932    2580 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lsvr7\" (UniqueName: \"kubernetes.io/projected/31d66af3-84e0-4725-a704-7d9532b0ffa9-kube-api-access-lsvr7\") pod \"31d66af3-84e0-4725-a704-7d9532b0ffa9\" (UID: \"31d66af3-84e0-4725-a704-7d9532b0ffa9\") "
May 25 13:36:14.450237 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:36:14.450214    2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31d66af3-84e0-4725-a704-7d9532b0ffa9-success-200-isvc-72c4b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "success-200-isvc-72c4b-kube-rbac-proxy-sar-config") pod "31d66af3-84e0-4725-a704-7d9532b0ffa9" (UID: "31d66af3-84e0-4725-a704-7d9532b0ffa9"). InnerVolumeSpecName "success-200-isvc-72c4b-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 13:36:14.452189 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:36:14.452162    2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31d66af3-84e0-4725-a704-7d9532b0ffa9-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "31d66af3-84e0-4725-a704-7d9532b0ffa9" (UID: "31d66af3-84e0-4725-a704-7d9532b0ffa9"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 13:36:14.452279 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:36:14.452215    2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31d66af3-84e0-4725-a704-7d9532b0ffa9-kube-api-access-lsvr7" (OuterVolumeSpecName: "kube-api-access-lsvr7") pod "31d66af3-84e0-4725-a704-7d9532b0ffa9" (UID: "31d66af3-84e0-4725-a704-7d9532b0ffa9"). InnerVolumeSpecName "kube-api-access-lsvr7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 13:36:14.550911 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:36:14.550870    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/31d66af3-84e0-4725-a704-7d9532b0ffa9-proxy-tls\") on node \"ip-10-0-135-103.ec2.internal\" DevicePath \"\""
May 25 13:36:14.550911 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:36:14.550909    2580 reconciler_common.go:299] "Volume detached for volume \"success-200-isvc-72c4b-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/31d66af3-84e0-4725-a704-7d9532b0ffa9-success-200-isvc-72c4b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-103.ec2.internal\" DevicePath \"\""
May 25 13:36:14.551125 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:36:14.550927    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lsvr7\" (UniqueName: \"kubernetes.io/projected/31d66af3-84e0-4725-a704-7d9532b0ffa9-kube-api-access-lsvr7\") on node \"ip-10-0-135-103.ec2.internal\" DevicePath \"\""
May 25 13:36:14.837603 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:36:14.837569    2580 generic.go:358] "Generic (PLEG): container finished" podID="31d66af3-84e0-4725-a704-7d9532b0ffa9" containerID="c122197eeb0b5817e8ceb0a230b7e55bfd22aff6575fa3a28587bf9bebd6d441" exitCode=0
May 25 13:36:14.837985 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:36:14.837647    2580 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-72c4b-predictor-6f6fcbb-zg7jb"
May 25 13:36:14.837985 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:36:14.837654    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-72c4b-predictor-6f6fcbb-zg7jb" event={"ID":"31d66af3-84e0-4725-a704-7d9532b0ffa9","Type":"ContainerDied","Data":"c122197eeb0b5817e8ceb0a230b7e55bfd22aff6575fa3a28587bf9bebd6d441"}
May 25 13:36:14.837985 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:36:14.837700    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-72c4b-predictor-6f6fcbb-zg7jb" event={"ID":"31d66af3-84e0-4725-a704-7d9532b0ffa9","Type":"ContainerDied","Data":"057ed60b6802ca533eb28df34be512143ef19d464735fdcce39b6c5418f20715"}
May 25 13:36:14.837985 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:36:14.837715    2580 scope.go:117] "RemoveContainer" containerID="b8254ba91edc2b4002421cb29d9c6125ebb03abfc81476c8a487cbb24cb0f354"
May 25 13:36:14.846289 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:36:14.846268    2580 scope.go:117] "RemoveContainer" containerID="c122197eeb0b5817e8ceb0a230b7e55bfd22aff6575fa3a28587bf9bebd6d441"
May 25 13:36:14.853893 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:36:14.853875    2580 scope.go:117] "RemoveContainer" containerID="b8254ba91edc2b4002421cb29d9c6125ebb03abfc81476c8a487cbb24cb0f354"
May 25 13:36:14.854143 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:36:14.854127    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8254ba91edc2b4002421cb29d9c6125ebb03abfc81476c8a487cbb24cb0f354\": container with ID starting with b8254ba91edc2b4002421cb29d9c6125ebb03abfc81476c8a487cbb24cb0f354 not found: ID does not exist" containerID="b8254ba91edc2b4002421cb29d9c6125ebb03abfc81476c8a487cbb24cb0f354"
May 25 13:36:14.854209 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:36:14.854150    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8254ba91edc2b4002421cb29d9c6125ebb03abfc81476c8a487cbb24cb0f354"} err="failed to get container status \"b8254ba91edc2b4002421cb29d9c6125ebb03abfc81476c8a487cbb24cb0f354\": rpc error: code = NotFound desc = could not find container \"b8254ba91edc2b4002421cb29d9c6125ebb03abfc81476c8a487cbb24cb0f354\": container with ID starting with b8254ba91edc2b4002421cb29d9c6125ebb03abfc81476c8a487cbb24cb0f354 not found: ID does not exist"
May 25 13:36:14.854209 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:36:14.854168    2580 scope.go:117] "RemoveContainer" containerID="c122197eeb0b5817e8ceb0a230b7e55bfd22aff6575fa3a28587bf9bebd6d441"
May 25 13:36:14.854406 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:36:14.854387    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c122197eeb0b5817e8ceb0a230b7e55bfd22aff6575fa3a28587bf9bebd6d441\": container with ID starting with c122197eeb0b5817e8ceb0a230b7e55bfd22aff6575fa3a28587bf9bebd6d441 not found: ID does not exist" containerID="c122197eeb0b5817e8ceb0a230b7e55bfd22aff6575fa3a28587bf9bebd6d441"
May 25 13:36:14.854451 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:36:14.854410    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c122197eeb0b5817e8ceb0a230b7e55bfd22aff6575fa3a28587bf9bebd6d441"} err="failed to get container status \"c122197eeb0b5817e8ceb0a230b7e55bfd22aff6575fa3a28587bf9bebd6d441\": rpc error: code = NotFound desc = could not find container \"c122197eeb0b5817e8ceb0a230b7e55bfd22aff6575fa3a28587bf9bebd6d441\": container with ID starting with c122197eeb0b5817e8ceb0a230b7e55bfd22aff6575fa3a28587bf9bebd6d441 not found: ID does not exist"
May 25 13:36:14.858823 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:36:14.858799    2580 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-72c4b-predictor-6f6fcbb-zg7jb"]
May 25 13:36:14.860915 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:36:14.860893    2580 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-72c4b-predictor-6f6fcbb-zg7jb"]
May 25 13:36:16.377424 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:36:16.377389    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="31d66af3-84e0-4725-a704-7d9532b0ffa9" path="/var/lib/kubelet/pods/31d66af3-84e0-4725-a704-7d9532b0ffa9/volumes"
May 25 13:36:18.729423 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:36:18.729349    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-35687-predictor-6d599c8c48-68thq" podUID="1fbb8b99-22aa-4a70-87f2-e46d72c808f6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
May 25 13:36:18.837174 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:36:18.837140    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-12313-predictor-857dcf8cc4-46rl6"
May 25 13:36:18.837653 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:36:18.837630    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-12313-predictor-857dcf8cc4-46rl6" podUID="51711b09-0032-45d2-99eb-00e0eff727e0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
May 25 13:36:28.729147 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:36:28.729116    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-35687-predictor-6d599c8c48-68thq"
May 25 13:36:28.838557 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:36:28.838511    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-12313-predictor-857dcf8cc4-46rl6" podUID="51711b09-0032-45d2-99eb-00e0eff727e0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
May 25 13:36:38.838174 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:36:38.838132    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-12313-predictor-857dcf8cc4-46rl6" podUID="51711b09-0032-45d2-99eb-00e0eff727e0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
May 25 13:36:48.837585 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:36:48.837548    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-12313-predictor-857dcf8cc4-46rl6" podUID="51711b09-0032-45d2-99eb-00e0eff727e0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
May 25 13:36:51.330449 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:36:51.330415    2580 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-35687-predictor-6d599c8c48-68thq"]
May 25 13:36:51.330881 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:36:51.330802    2580 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-35687-predictor-6d599c8c48-68thq" podUID="1fbb8b99-22aa-4a70-87f2-e46d72c808f6" containerName="kserve-container" containerID="cri-o://7accd97455cdbb729594c456760dd86e65f479e278ce01a2c4142c1551c6dd02" gracePeriod=30
May 25 13:36:51.330959 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:36:51.330846    2580 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-35687-predictor-6d599c8c48-68thq" podUID="1fbb8b99-22aa-4a70-87f2-e46d72c808f6" containerName="kube-rbac-proxy" containerID="cri-o://308e4cfa263f957f07be34bb426dd4047a01264f502e15d3910360e6a5737cff" gracePeriod=30
May 25 13:36:51.349560 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:36:51.349535    2580 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-766ab-predictor-69d568958b-r85bq"]
May 25 13:36:51.349921 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:36:51.349909    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31d66af3-84e0-4725-a704-7d9532b0ffa9" containerName="kube-rbac-proxy"
May 25 13:36:51.349967 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:36:51.349923    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="31d66af3-84e0-4725-a704-7d9532b0ffa9" containerName="kube-rbac-proxy"
May 25 13:36:51.349967 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:36:51.349946    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31d66af3-84e0-4725-a704-7d9532b0ffa9" containerName="kserve-container"
May 25 13:36:51.349967 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:36:51.349952    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="31d66af3-84e0-4725-a704-7d9532b0ffa9" containerName="kserve-container"
May 25 13:36:51.350062 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:36:51.350000    2580 memory_manager.go:356] "RemoveStaleState removing state" podUID="31d66af3-84e0-4725-a704-7d9532b0ffa9" containerName="kube-rbac-proxy"
May 25 13:36:51.350062 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:36:51.350010    2580 memory_manager.go:356] "RemoveStaleState removing state" podUID="31d66af3-84e0-4725-a704-7d9532b0ffa9" containerName="kserve-container"
May 25 13:36:51.353353 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:36:51.353336    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-766ab-predictor-69d568958b-r85bq"
May 25 13:36:51.355746 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:36:51.355724    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-766ab-predictor-serving-cert\""
May 25 13:36:51.355877 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:36:51.355858    2580 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-766ab-kube-rbac-proxy-sar-config\""
May 25 13:36:51.365896 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:36:51.365874    2580 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-766ab-predictor-69d568958b-r85bq"]
May 25 13:36:51.461537 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:36:51.461502    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/62f285ff-1f15-4a53-992e-f071b91c6c0a-proxy-tls\") pod \"success-200-isvc-766ab-predictor-69d568958b-r85bq\" (UID: \"62f285ff-1f15-4a53-992e-f071b91c6c0a\") " pod="kserve-ci-e2e-test/success-200-isvc-766ab-predictor-69d568958b-r85bq"
May 25 13:36:51.461715 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:36:51.461673    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"success-200-isvc-766ab-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/62f285ff-1f15-4a53-992e-f071b91c6c0a-success-200-isvc-766ab-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-766ab-predictor-69d568958b-r85bq\" (UID: \"62f285ff-1f15-4a53-992e-f071b91c6c0a\") " pod="kserve-ci-e2e-test/success-200-isvc-766ab-predictor-69d568958b-r85bq"
May 25 13:36:51.461801 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:36:51.461742    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbctw\" (UniqueName: \"kubernetes.io/projected/62f285ff-1f15-4a53-992e-f071b91c6c0a-kube-api-access-lbctw\") pod \"success-200-isvc-766ab-predictor-69d568958b-r85bq\" (UID: \"62f285ff-1f15-4a53-992e-f071b91c6c0a\") " pod="kserve-ci-e2e-test/success-200-isvc-766ab-predictor-69d568958b-r85bq"
May 25 13:36:51.563271 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:36:51.563232    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/62f285ff-1f15-4a53-992e-f071b91c6c0a-proxy-tls\") pod \"success-200-isvc-766ab-predictor-69d568958b-r85bq\" (UID: \"62f285ff-1f15-4a53-992e-f071b91c6c0a\") " pod="kserve-ci-e2e-test/success-200-isvc-766ab-predictor-69d568958b-r85bq"
May 25 13:36:51.563479 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:36:51.563309    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"success-200-isvc-766ab-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/62f285ff-1f15-4a53-992e-f071b91c6c0a-success-200-isvc-766ab-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-766ab-predictor-69d568958b-r85bq\" (UID: \"62f285ff-1f15-4a53-992e-f071b91c6c0a\") " pod="kserve-ci-e2e-test/success-200-isvc-766ab-predictor-69d568958b-r85bq"
May 25 13:36:51.563479 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:36:51.563341    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-lbctw\" (UniqueName: \"kubernetes.io/projected/62f285ff-1f15-4a53-992e-f071b91c6c0a-kube-api-access-lbctw\") pod \"success-200-isvc-766ab-predictor-69d568958b-r85bq\" (UID: \"62f285ff-1f15-4a53-992e-f071b91c6c0a\") " pod="kserve-ci-e2e-test/success-200-isvc-766ab-predictor-69d568958b-r85bq"
May 25 13:36:51.563479 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:36:51.563414    2580 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-766ab-predictor-serving-cert: secret "success-200-isvc-766ab-predictor-serving-cert" not found
May 25 13:36:51.563610 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:36:51.563498    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62f285ff-1f15-4a53-992e-f071b91c6c0a-proxy-tls podName:62f285ff-1f15-4a53-992e-f071b91c6c0a nodeName:}" failed. No retries permitted until 2026-05-25 13:36:52.063477218 +0000 UTC m=+1912.255771572 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/62f285ff-1f15-4a53-992e-f071b91c6c0a-proxy-tls") pod "success-200-isvc-766ab-predictor-69d568958b-r85bq" (UID: "62f285ff-1f15-4a53-992e-f071b91c6c0a") : secret "success-200-isvc-766ab-predictor-serving-cert" not found
May 25 13:36:51.563976 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:36:51.563956    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"success-200-isvc-766ab-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/62f285ff-1f15-4a53-992e-f071b91c6c0a-success-200-isvc-766ab-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-766ab-predictor-69d568958b-r85bq\" (UID: \"62f285ff-1f15-4a53-992e-f071b91c6c0a\") " pod="kserve-ci-e2e-test/success-200-isvc-766ab-predictor-69d568958b-r85bq"
May 25 13:36:51.572227 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:36:51.572196    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbctw\" (UniqueName: \"kubernetes.io/projected/62f285ff-1f15-4a53-992e-f071b91c6c0a-kube-api-access-lbctw\") pod \"success-200-isvc-766ab-predictor-69d568958b-r85bq\" (UID: \"62f285ff-1f15-4a53-992e-f071b91c6c0a\") " pod="kserve-ci-e2e-test/success-200-isvc-766ab-predictor-69d568958b-r85bq"
May 25 13:36:51.964395 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:36:51.964336    2580 generic.go:358] "Generic (PLEG): container finished" podID="1fbb8b99-22aa-4a70-87f2-e46d72c808f6" containerID="308e4cfa263f957f07be34bb426dd4047a01264f502e15d3910360e6a5737cff" exitCode=2
May 25 13:36:51.964579 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:36:51.964411    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-35687-predictor-6d599c8c48-68thq" event={"ID":"1fbb8b99-22aa-4a70-87f2-e46d72c808f6","Type":"ContainerDied","Data":"308e4cfa263f957f07be34bb426dd4047a01264f502e15d3910360e6a5737cff"}
May 25 13:36:52.066825 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:36:52.066758    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/62f285ff-1f15-4a53-992e-f071b91c6c0a-proxy-tls\") pod \"success-200-isvc-766ab-predictor-69d568958b-r85bq\" (UID: \"62f285ff-1f15-4a53-992e-f071b91c6c0a\") " pod="kserve-ci-e2e-test/success-200-isvc-766ab-predictor-69d568958b-r85bq"
May 25 13:36:52.069380 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:36:52.069340    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/62f285ff-1f15-4a53-992e-f071b91c6c0a-proxy-tls\") pod \"success-200-isvc-766ab-predictor-69d568958b-r85bq\" (UID: \"62f285ff-1f15-4a53-992e-f071b91c6c0a\") " pod="kserve-ci-e2e-test/success-200-isvc-766ab-predictor-69d568958b-r85bq"
May 25 13:36:52.262911 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:36:52.262805    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-766ab-predictor-69d568958b-r85bq"
May 25 13:36:52.392693 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:36:52.392635    2580 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-766ab-predictor-69d568958b-r85bq"]
May 25 13:36:52.394951 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:36:52.394921    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62f285ff_1f15_4a53_992e_f071b91c6c0a.slice/crio-6c81c7da49bd2bf649f8ffa3603c07aaf6a4f27ce261a56027b7c85aef80dc48 WatchSource:0}: Error finding container 6c81c7da49bd2bf649f8ffa3603c07aaf6a4f27ce261a56027b7c85aef80dc48: Status 404 returned error can't find the container with id 6c81c7da49bd2bf649f8ffa3603c07aaf6a4f27ce261a56027b7c85aef80dc48
May 25 13:36:52.969054 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:36:52.969017    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-766ab-predictor-69d568958b-r85bq" event={"ID":"62f285ff-1f15-4a53-992e-f071b91c6c0a","Type":"ContainerStarted","Data":"c57265350870fd608752ef68ff3c86dc2355b75d369ad266ce92bfaaefa59b66"}
May 25 13:36:52.969258 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:36:52.969062    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-766ab-predictor-69d568958b-r85bq" event={"ID":"62f285ff-1f15-4a53-992e-f071b91c6c0a","Type":"ContainerStarted","Data":"38e2e314d960f222c39949f6d9162ae9f2b462ff257b9f303f1f8e94af6c9371"}
May 25 13:36:52.969258 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:36:52.969075    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-766ab-predictor-69d568958b-r85bq" event={"ID":"62f285ff-1f15-4a53-992e-f071b91c6c0a","Type":"ContainerStarted","Data":"6c81c7da49bd2bf649f8ffa3603c07aaf6a4f27ce261a56027b7c85aef80dc48"}
May 25 13:36:52.969258 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:36:52.969106    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-766ab-predictor-69d568958b-r85bq"
May 25 13:36:52.988240 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:36:52.988191    2580 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-766ab-predictor-69d568958b-r85bq" podStartSLOduration=1.988176763 podStartE2EDuration="1.988176763s" podCreationTimestamp="2026-05-25 13:36:51 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 13:36:52.986694149 +0000 UTC m=+1913.178988510" watchObservedRunningTime="2026-05-25 13:36:52.988176763 +0000 UTC m=+1913.180471123"
May 25 13:36:53.723807 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:36:53.723768    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-35687-predictor-6d599c8c48-68thq" podUID="1fbb8b99-22aa-4a70-87f2-e46d72c808f6" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.41:8643/healthz\": dial tcp 10.134.0.41:8643: connect: connection refused"
May 25 13:36:53.972622 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:36:53.972590    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-766ab-predictor-69d568958b-r85bq"
May 25 13:36:53.973912 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:36:53.973839    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-766ab-predictor-69d568958b-r85bq" podUID="62f285ff-1f15-4a53-992e-f071b91c6c0a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
May 25 13:36:54.786628 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:36:54.786604    2580 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-35687-predictor-6d599c8c48-68thq"
May 25 13:36:54.892632 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:36:54.892537    2580 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"success-200-isvc-35687-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1fbb8b99-22aa-4a70-87f2-e46d72c808f6-success-200-isvc-35687-kube-rbac-proxy-sar-config\") pod \"1fbb8b99-22aa-4a70-87f2-e46d72c808f6\" (UID: \"1fbb8b99-22aa-4a70-87f2-e46d72c808f6\") "
May 25 13:36:54.892632 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:36:54.892580    2580 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1fbb8b99-22aa-4a70-87f2-e46d72c808f6-proxy-tls\") pod \"1fbb8b99-22aa-4a70-87f2-e46d72c808f6\" (UID: \"1fbb8b99-22aa-4a70-87f2-e46d72c808f6\") "
May 25 13:36:54.892632 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:36:54.892608    2580 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5tmcf\" (UniqueName: \"kubernetes.io/projected/1fbb8b99-22aa-4a70-87f2-e46d72c808f6-kube-api-access-5tmcf\") pod \"1fbb8b99-22aa-4a70-87f2-e46d72c808f6\" (UID: \"1fbb8b99-22aa-4a70-87f2-e46d72c808f6\") "
May 25 13:36:54.892924 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:36:54.892885    2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1fbb8b99-22aa-4a70-87f2-e46d72c808f6-success-200-isvc-35687-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "success-200-isvc-35687-kube-rbac-proxy-sar-config") pod "1fbb8b99-22aa-4a70-87f2-e46d72c808f6" (UID: "1fbb8b99-22aa-4a70-87f2-e46d72c808f6"). InnerVolumeSpecName "success-200-isvc-35687-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 13:36:54.894796 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:36:54.894766    2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fbb8b99-22aa-4a70-87f2-e46d72c808f6-kube-api-access-5tmcf" (OuterVolumeSpecName: "kube-api-access-5tmcf") pod "1fbb8b99-22aa-4a70-87f2-e46d72c808f6" (UID: "1fbb8b99-22aa-4a70-87f2-e46d72c808f6"). InnerVolumeSpecName "kube-api-access-5tmcf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 13:36:54.894918 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:36:54.894851    2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1fbb8b99-22aa-4a70-87f2-e46d72c808f6-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "1fbb8b99-22aa-4a70-87f2-e46d72c808f6" (UID: "1fbb8b99-22aa-4a70-87f2-e46d72c808f6"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 13:36:54.980151 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:36:54.980117    2580 generic.go:358] "Generic (PLEG): container finished" podID="1fbb8b99-22aa-4a70-87f2-e46d72c808f6" containerID="7accd97455cdbb729594c456760dd86e65f479e278ce01a2c4142c1551c6dd02" exitCode=0
May 25 13:36:54.980317 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:36:54.980191    2580 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-35687-predictor-6d599c8c48-68thq"
May 25 13:36:54.980317 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:36:54.980195    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-35687-predictor-6d599c8c48-68thq" event={"ID":"1fbb8b99-22aa-4a70-87f2-e46d72c808f6","Type":"ContainerDied","Data":"7accd97455cdbb729594c456760dd86e65f479e278ce01a2c4142c1551c6dd02"}
May 25 13:36:54.980317 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:36:54.980233    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-35687-predictor-6d599c8c48-68thq" event={"ID":"1fbb8b99-22aa-4a70-87f2-e46d72c808f6","Type":"ContainerDied","Data":"2d09f52f3659689915356054063cfced902cfa177dd58b61e0f3202c32db03f5"}
May 25 13:36:54.980317 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:36:54.980254    2580 scope.go:117] "RemoveContainer" containerID="308e4cfa263f957f07be34bb426dd4047a01264f502e15d3910360e6a5737cff"
May 25 13:36:54.980880 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:36:54.980851    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-766ab-predictor-69d568958b-r85bq" podUID="62f285ff-1f15-4a53-992e-f071b91c6c0a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
May 25 13:36:54.991138 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:36:54.991117    2580 scope.go:117] "RemoveContainer" containerID="7accd97455cdbb729594c456760dd86e65f479e278ce01a2c4142c1551c6dd02"
May 25 13:36:54.993574 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:36:54.993554    2580 reconciler_common.go:299] "Volume detached for volume \"success-200-isvc-35687-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1fbb8b99-22aa-4a70-87f2-e46d72c808f6-success-200-isvc-35687-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-103.ec2.internal\" DevicePath \"\""
May 25 13:36:54.993649 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:36:54.993577    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1fbb8b99-22aa-4a70-87f2-e46d72c808f6-proxy-tls\") on node \"ip-10-0-135-103.ec2.internal\" DevicePath \"\""
May 25 13:36:54.993649 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:36:54.993589    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5tmcf\" (UniqueName: \"kubernetes.io/projected/1fbb8b99-22aa-4a70-87f2-e46d72c808f6-kube-api-access-5tmcf\") on node \"ip-10-0-135-103.ec2.internal\" DevicePath \"\""
May 25 13:36:54.998777 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:36:54.998733    2580 scope.go:117] "RemoveContainer" containerID="308e4cfa263f957f07be34bb426dd4047a01264f502e15d3910360e6a5737cff"
May 25 13:36:54.999104 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:36:54.999077    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"308e4cfa263f957f07be34bb426dd4047a01264f502e15d3910360e6a5737cff\": container with ID starting with 308e4cfa263f957f07be34bb426dd4047a01264f502e15d3910360e6a5737cff not found: ID does not exist" containerID="308e4cfa263f957f07be34bb426dd4047a01264f502e15d3910360e6a5737cff"
May 25 13:36:54.999213 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:36:54.999115    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"308e4cfa263f957f07be34bb426dd4047a01264f502e15d3910360e6a5737cff"} err="failed to get container status \"308e4cfa263f957f07be34bb426dd4047a01264f502e15d3910360e6a5737cff\": rpc error: code = NotFound desc = could not find container \"308e4cfa263f957f07be34bb426dd4047a01264f502e15d3910360e6a5737cff\": container with ID starting with 308e4cfa263f957f07be34bb426dd4047a01264f502e15d3910360e6a5737cff not found: ID does not exist"
May 25 13:36:54.999213 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:36:54.999154    2580 scope.go:117] "RemoveContainer" containerID="7accd97455cdbb729594c456760dd86e65f479e278ce01a2c4142c1551c6dd02"
May 25 13:36:54.999513 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:36:54.999448    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7accd97455cdbb729594c456760dd86e65f479e278ce01a2c4142c1551c6dd02\": container with ID starting with 7accd97455cdbb729594c456760dd86e65f479e278ce01a2c4142c1551c6dd02 not found: ID does not exist" containerID="7accd97455cdbb729594c456760dd86e65f479e278ce01a2c4142c1551c6dd02"
May 25 13:36:54.999513 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:36:54.999491    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7accd97455cdbb729594c456760dd86e65f479e278ce01a2c4142c1551c6dd02"} err="failed to get container status \"7accd97455cdbb729594c456760dd86e65f479e278ce01a2c4142c1551c6dd02\": rpc error: code = NotFound desc = could not find container \"7accd97455cdbb729594c456760dd86e65f479e278ce01a2c4142c1551c6dd02\": container with ID starting with 7accd97455cdbb729594c456760dd86e65f479e278ce01a2c4142c1551c6dd02 not found: ID does not exist"
May 25 13:36:55.000987 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:36:55.000966    2580 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-35687-predictor-6d599c8c48-68thq"]
May 25 13:36:55.006843 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:36:55.006822    2580 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-35687-predictor-6d599c8c48-68thq"]
May 25 13:36:56.377651 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:36:56.377615    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1fbb8b99-22aa-4a70-87f2-e46d72c808f6" path="/var/lib/kubelet/pods/1fbb8b99-22aa-4a70-87f2-e46d72c808f6/volumes"
May 25 13:36:58.838269 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:36:58.838241    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-12313-predictor-857dcf8cc4-46rl6"
May 25 13:36:59.985013 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:36:59.984977    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-766ab-predictor-69d568958b-r85bq"
May 25 13:36:59.985595 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:36:59.985569    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-766ab-predictor-69d568958b-r85bq" podUID="62f285ff-1f15-4a53-992e-f071b91c6c0a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
May 25 13:37:09.985998 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:37:09.985951    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-766ab-predictor-69d568958b-r85bq" podUID="62f285ff-1f15-4a53-992e-f071b91c6c0a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
May 25 13:37:19.986323 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:37:19.986283    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-766ab-predictor-69d568958b-r85bq" podUID="62f285ff-1f15-4a53-992e-f071b91c6c0a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
May 25 13:37:29.986409 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:37:29.986341    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-766ab-predictor-69d568958b-r85bq" podUID="62f285ff-1f15-4a53-992e-f071b91c6c0a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
May 25 13:37:39.986395 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:37:39.986336    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-766ab-predictor-69d568958b-r85bq"
May 25 13:40:00.409352 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:40:00.409322    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-646r6_59b905b8-5ce3-49f6-b47c-b5b3a525fe06/ovn-acl-logging/0.log"
May 25 13:40:00.421124 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:40:00.421094    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-646r6_59b905b8-5ce3-49f6-b47c-b5b3a525fe06/ovn-acl-logging/0.log"
May 25 13:45:00.445964 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:45:00.445856    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-646r6_59b905b8-5ce3-49f6-b47c-b5b3a525fe06/ovn-acl-logging/0.log"
May 25 13:45:00.450025 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:45:00.448212    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-646r6_59b905b8-5ce3-49f6-b47c-b5b3a525fe06/ovn-acl-logging/0.log"
May 25 13:46:06.269535 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:46:06.269494    2580 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-766ab-predictor-69d568958b-r85bq"]
May 25 13:46:06.270069 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:46:06.269858    2580 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-766ab-predictor-69d568958b-r85bq" podUID="62f285ff-1f15-4a53-992e-f071b91c6c0a" containerName="kserve-container" containerID="cri-o://38e2e314d960f222c39949f6d9162ae9f2b462ff257b9f303f1f8e94af6c9371" gracePeriod=30
May 25 13:46:06.270069 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:46:06.269891    2580 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-766ab-predictor-69d568958b-r85bq" podUID="62f285ff-1f15-4a53-992e-f071b91c6c0a" containerName="kube-rbac-proxy" containerID="cri-o://c57265350870fd608752ef68ff3c86dc2355b75d369ad266ce92bfaaefa59b66" gracePeriod=30
May 25 13:46:06.828987 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:46:06.828945    2580 generic.go:358] "Generic (PLEG): container finished" podID="62f285ff-1f15-4a53-992e-f071b91c6c0a" containerID="c57265350870fd608752ef68ff3c86dc2355b75d369ad266ce92bfaaefa59b66" exitCode=2
May 25 13:46:06.829164 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:46:06.829019    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-766ab-predictor-69d568958b-r85bq" event={"ID":"62f285ff-1f15-4a53-992e-f071b91c6c0a","Type":"ContainerDied","Data":"c57265350870fd608752ef68ff3c86dc2355b75d369ad266ce92bfaaefa59b66"}
May 25 13:46:09.414415 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:46:09.414390    2580 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-766ab-predictor-69d568958b-r85bq"
May 25 13:46:09.514223 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:46:09.514135    2580 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/62f285ff-1f15-4a53-992e-f071b91c6c0a-proxy-tls\") pod \"62f285ff-1f15-4a53-992e-f071b91c6c0a\" (UID: \"62f285ff-1f15-4a53-992e-f071b91c6c0a\") "
May 25 13:46:09.514409 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:46:09.514269    2580 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lbctw\" (UniqueName: \"kubernetes.io/projected/62f285ff-1f15-4a53-992e-f071b91c6c0a-kube-api-access-lbctw\") pod \"62f285ff-1f15-4a53-992e-f071b91c6c0a\" (UID: \"62f285ff-1f15-4a53-992e-f071b91c6c0a\") "
May 25 13:46:09.514409 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:46:09.514305    2580 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"success-200-isvc-766ab-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/62f285ff-1f15-4a53-992e-f071b91c6c0a-success-200-isvc-766ab-kube-rbac-proxy-sar-config\") pod \"62f285ff-1f15-4a53-992e-f071b91c6c0a\" (UID: \"62f285ff-1f15-4a53-992e-f071b91c6c0a\") "
May 25 13:46:09.514727 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:46:09.514702    2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62f285ff-1f15-4a53-992e-f071b91c6c0a-success-200-isvc-766ab-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "success-200-isvc-766ab-kube-rbac-proxy-sar-config") pod "62f285ff-1f15-4a53-992e-f071b91c6c0a" (UID: "62f285ff-1f15-4a53-992e-f071b91c6c0a"). InnerVolumeSpecName "success-200-isvc-766ab-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 13:46:09.516400 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:46:09.516360    2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62f285ff-1f15-4a53-992e-f071b91c6c0a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "62f285ff-1f15-4a53-992e-f071b91c6c0a" (UID: "62f285ff-1f15-4a53-992e-f071b91c6c0a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 13:46:09.516518 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:46:09.516497    2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62f285ff-1f15-4a53-992e-f071b91c6c0a-kube-api-access-lbctw" (OuterVolumeSpecName: "kube-api-access-lbctw") pod "62f285ff-1f15-4a53-992e-f071b91c6c0a" (UID: "62f285ff-1f15-4a53-992e-f071b91c6c0a"). InnerVolumeSpecName "kube-api-access-lbctw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 13:46:09.615352 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:46:09.615314    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lbctw\" (UniqueName: \"kubernetes.io/projected/62f285ff-1f15-4a53-992e-f071b91c6c0a-kube-api-access-lbctw\") on node \"ip-10-0-135-103.ec2.internal\" DevicePath \"\""
May 25 13:46:09.615352 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:46:09.615347    2580 reconciler_common.go:299] "Volume detached for volume \"success-200-isvc-766ab-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/62f285ff-1f15-4a53-992e-f071b91c6c0a-success-200-isvc-766ab-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-103.ec2.internal\" DevicePath \"\""
May 25 13:46:09.615352 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:46:09.615359    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/62f285ff-1f15-4a53-992e-f071b91c6c0a-proxy-tls\") on node \"ip-10-0-135-103.ec2.internal\" DevicePath \"\""
May 25 13:46:09.840283 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:46:09.840239    2580 generic.go:358] "Generic (PLEG): container finished" podID="62f285ff-1f15-4a53-992e-f071b91c6c0a" containerID="38e2e314d960f222c39949f6d9162ae9f2b462ff257b9f303f1f8e94af6c9371" exitCode=0
May 25 13:46:09.840463 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:46:09.840312    2580 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-766ab-predictor-69d568958b-r85bq"
May 25 13:46:09.840463 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:46:09.840308    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-766ab-predictor-69d568958b-r85bq" event={"ID":"62f285ff-1f15-4a53-992e-f071b91c6c0a","Type":"ContainerDied","Data":"38e2e314d960f222c39949f6d9162ae9f2b462ff257b9f303f1f8e94af6c9371"}
May 25 13:46:09.840463 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:46:09.840357    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-766ab-predictor-69d568958b-r85bq" event={"ID":"62f285ff-1f15-4a53-992e-f071b91c6c0a","Type":"ContainerDied","Data":"6c81c7da49bd2bf649f8ffa3603c07aaf6a4f27ce261a56027b7c85aef80dc48"}
May 25 13:46:09.840463 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:46:09.840393    2580 scope.go:117] "RemoveContainer" containerID="c57265350870fd608752ef68ff3c86dc2355b75d369ad266ce92bfaaefa59b66"
May 25 13:46:09.849098 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:46:09.849078    2580 scope.go:117] "RemoveContainer" containerID="38e2e314d960f222c39949f6d9162ae9f2b462ff257b9f303f1f8e94af6c9371"
May 25 13:46:09.856306 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:46:09.856285    2580 scope.go:117] "RemoveContainer" containerID="c57265350870fd608752ef68ff3c86dc2355b75d369ad266ce92bfaaefa59b66"
May 25 13:46:09.856601 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:46:09.856582    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c57265350870fd608752ef68ff3c86dc2355b75d369ad266ce92bfaaefa59b66\": container with ID starting with c57265350870fd608752ef68ff3c86dc2355b75d369ad266ce92bfaaefa59b66 not found: ID does not exist" containerID="c57265350870fd608752ef68ff3c86dc2355b75d369ad266ce92bfaaefa59b66"
May 25 13:46:09.856653 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:46:09.856608    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c57265350870fd608752ef68ff3c86dc2355b75d369ad266ce92bfaaefa59b66"} err="failed to get container status \"c57265350870fd608752ef68ff3c86dc2355b75d369ad266ce92bfaaefa59b66\": rpc error: code = NotFound desc = could not find container \"c57265350870fd608752ef68ff3c86dc2355b75d369ad266ce92bfaaefa59b66\": container with ID starting with c57265350870fd608752ef68ff3c86dc2355b75d369ad266ce92bfaaefa59b66 not found: ID does not exist"
May 25 13:46:09.856653 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:46:09.856623    2580 scope.go:117] "RemoveContainer" containerID="38e2e314d960f222c39949f6d9162ae9f2b462ff257b9f303f1f8e94af6c9371"
May 25 13:46:09.856837 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:46:09.856815    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38e2e314d960f222c39949f6d9162ae9f2b462ff257b9f303f1f8e94af6c9371\": container with ID starting with 38e2e314d960f222c39949f6d9162ae9f2b462ff257b9f303f1f8e94af6c9371 not found: ID does not exist" containerID="38e2e314d960f222c39949f6d9162ae9f2b462ff257b9f303f1f8e94af6c9371"
May 25 13:46:09.856896 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:46:09.856847    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38e2e314d960f222c39949f6d9162ae9f2b462ff257b9f303f1f8e94af6c9371"} err="failed to get container status \"38e2e314d960f222c39949f6d9162ae9f2b462ff257b9f303f1f8e94af6c9371\": rpc error: code = NotFound desc = could not find container \"38e2e314d960f222c39949f6d9162ae9f2b462ff257b9f303f1f8e94af6c9371\": container with ID starting with 38e2e314d960f222c39949f6d9162ae9f2b462ff257b9f303f1f8e94af6c9371 not found: ID does not exist"
May 25 13:46:09.862618 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:46:09.862597    2580 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-766ab-predictor-69d568958b-r85bq"]
May 25 13:46:09.867735 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:46:09.867713    2580 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-766ab-predictor-69d568958b-r85bq"]
May 25 13:46:10.377993 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:46:10.377957    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="62f285ff-1f15-4a53-992e-f071b91c6c0a" path="/var/lib/kubelet/pods/62f285ff-1f15-4a53-992e-f071b91c6c0a/volumes"
May 25 13:50:00.475827 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:50:00.475700    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-646r6_59b905b8-5ce3-49f6-b47c-b5b3a525fe06/ovn-acl-logging/0.log"
May 25 13:50:00.479781 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:50:00.477962    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-646r6_59b905b8-5ce3-49f6-b47c-b5b3a525fe06/ovn-acl-logging/0.log"
May 25 13:53:30.697310 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:53:30.697276    2580 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-12313-predictor-857dcf8cc4-46rl6"]
May 25 13:53:30.697840 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:53:30.697593    2580 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-12313-predictor-857dcf8cc4-46rl6" podUID="51711b09-0032-45d2-99eb-00e0eff727e0" containerName="kserve-container" containerID="cri-o://514be88f1d3d8daa239e43fb4acdfc0f6bf79f4508c61bb8fd03345a1ced237a" gracePeriod=30
May 25 13:53:30.697840 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:53:30.697632    2580 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-12313-predictor-857dcf8cc4-46rl6" podUID="51711b09-0032-45d2-99eb-00e0eff727e0" containerName="kube-rbac-proxy" containerID="cri-o://5ff3ffddb10a8dd1b122bb5f13d0847ec320d03f94fdc5e1a5ae8e49fad8027c" gracePeriod=30
May 25 13:53:31.324071 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:53:31.324039    2580 generic.go:358] "Generic (PLEG): container finished" podID="51711b09-0032-45d2-99eb-00e0eff727e0" containerID="5ff3ffddb10a8dd1b122bb5f13d0847ec320d03f94fdc5e1a5ae8e49fad8027c" exitCode=2
May 25 13:53:31.324248 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:53:31.324099    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-12313-predictor-857dcf8cc4-46rl6" event={"ID":"51711b09-0032-45d2-99eb-00e0eff727e0","Type":"ContainerDied","Data":"5ff3ffddb10a8dd1b122bb5f13d0847ec320d03f94fdc5e1a5ae8e49fad8027c"}
May 25 13:53:33.645799 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:53:33.645778    2580 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-12313-predictor-857dcf8cc4-46rl6"
May 25 13:53:33.726646 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:53:33.726614    2580 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jx86g\" (UniqueName: \"kubernetes.io/projected/51711b09-0032-45d2-99eb-00e0eff727e0-kube-api-access-jx86g\") pod \"51711b09-0032-45d2-99eb-00e0eff727e0\" (UID: \"51711b09-0032-45d2-99eb-00e0eff727e0\") "
May 25 13:53:33.726843 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:53:33.726683    2580 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"success-200-isvc-12313-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/51711b09-0032-45d2-99eb-00e0eff727e0-success-200-isvc-12313-kube-rbac-proxy-sar-config\") pod \"51711b09-0032-45d2-99eb-00e0eff727e0\" (UID: \"51711b09-0032-45d2-99eb-00e0eff727e0\") "
May 25 13:53:33.726843 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:53:33.726755    2580 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/51711b09-0032-45d2-99eb-00e0eff727e0-proxy-tls\") pod \"51711b09-0032-45d2-99eb-00e0eff727e0\" (UID: \"51711b09-0032-45d2-99eb-00e0eff727e0\") "
May 25 13:53:33.727029 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:53:33.727002    2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51711b09-0032-45d2-99eb-00e0eff727e0-success-200-isvc-12313-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "success-200-isvc-12313-kube-rbac-proxy-sar-config") pod "51711b09-0032-45d2-99eb-00e0eff727e0" (UID: "51711b09-0032-45d2-99eb-00e0eff727e0"). InnerVolumeSpecName "success-200-isvc-12313-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 13:53:33.729022 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:53:33.728992    2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51711b09-0032-45d2-99eb-00e0eff727e0-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "51711b09-0032-45d2-99eb-00e0eff727e0" (UID: "51711b09-0032-45d2-99eb-00e0eff727e0"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 13:53:33.729022 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:53:33.729002    2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51711b09-0032-45d2-99eb-00e0eff727e0-kube-api-access-jx86g" (OuterVolumeSpecName: "kube-api-access-jx86g") pod "51711b09-0032-45d2-99eb-00e0eff727e0" (UID: "51711b09-0032-45d2-99eb-00e0eff727e0"). InnerVolumeSpecName "kube-api-access-jx86g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 13:53:33.827723 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:53:33.827699    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/51711b09-0032-45d2-99eb-00e0eff727e0-proxy-tls\") on node \"ip-10-0-135-103.ec2.internal\" DevicePath \"\""
May 25 13:53:33.827861 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:53:33.827726    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jx86g\" (UniqueName: \"kubernetes.io/projected/51711b09-0032-45d2-99eb-00e0eff727e0-kube-api-access-jx86g\") on node \"ip-10-0-135-103.ec2.internal\" DevicePath \"\""
May 25 13:53:33.827861 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:53:33.827739    2580 reconciler_common.go:299] "Volume detached for volume \"success-200-isvc-12313-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/51711b09-0032-45d2-99eb-00e0eff727e0-success-200-isvc-12313-kube-rbac-proxy-sar-config\") on node \"ip-10-0-135-103.ec2.internal\" DevicePath \"\""
May 25 13:53:34.335703 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:53:34.335667    2580 generic.go:358] "Generic (PLEG): container finished" podID="51711b09-0032-45d2-99eb-00e0eff727e0" containerID="514be88f1d3d8daa239e43fb4acdfc0f6bf79f4508c61bb8fd03345a1ced237a" exitCode=0
May 25 13:53:34.335889 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:53:34.335740    2580 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-12313-predictor-857dcf8cc4-46rl6"
May 25 13:53:34.335889 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:53:34.335741    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-12313-predictor-857dcf8cc4-46rl6" event={"ID":"51711b09-0032-45d2-99eb-00e0eff727e0","Type":"ContainerDied","Data":"514be88f1d3d8daa239e43fb4acdfc0f6bf79f4508c61bb8fd03345a1ced237a"}
May 25 13:53:34.335889 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:53:34.335853    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-12313-predictor-857dcf8cc4-46rl6" event={"ID":"51711b09-0032-45d2-99eb-00e0eff727e0","Type":"ContainerDied","Data":"14cd739922656c225a280b0c1d93111e8d3d7159e45bb06955b618554c3ec584"}
May 25 13:53:34.335889 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:53:34.335874    2580 scope.go:117] "RemoveContainer" containerID="5ff3ffddb10a8dd1b122bb5f13d0847ec320d03f94fdc5e1a5ae8e49fad8027c"
May 25 13:53:34.344182 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:53:34.344166    2580 scope.go:117] "RemoveContainer" containerID="514be88f1d3d8daa239e43fb4acdfc0f6bf79f4508c61bb8fd03345a1ced237a"
May 25 13:53:34.351354 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:53:34.351336    2580 scope.go:117] "RemoveContainer" containerID="5ff3ffddb10a8dd1b122bb5f13d0847ec320d03f94fdc5e1a5ae8e49fad8027c"
May 25 13:53:34.351630 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:53:34.351609    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ff3ffddb10a8dd1b122bb5f13d0847ec320d03f94fdc5e1a5ae8e49fad8027c\": container with ID starting with 5ff3ffddb10a8dd1b122bb5f13d0847ec320d03f94fdc5e1a5ae8e49fad8027c not found: ID does not exist" containerID="5ff3ffddb10a8dd1b122bb5f13d0847ec320d03f94fdc5e1a5ae8e49fad8027c"
May 25 13:53:34.351674 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:53:34.351638    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ff3ffddb10a8dd1b122bb5f13d0847ec320d03f94fdc5e1a5ae8e49fad8027c"} err="failed to get container status \"5ff3ffddb10a8dd1b122bb5f13d0847ec320d03f94fdc5e1a5ae8e49fad8027c\": rpc error: code = NotFound desc = could not find container \"5ff3ffddb10a8dd1b122bb5f13d0847ec320d03f94fdc5e1a5ae8e49fad8027c\": container with ID starting with 5ff3ffddb10a8dd1b122bb5f13d0847ec320d03f94fdc5e1a5ae8e49fad8027c not found: ID does not exist"
May 25 13:53:34.351674 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:53:34.351656    2580 scope.go:117] "RemoveContainer" containerID="514be88f1d3d8daa239e43fb4acdfc0f6bf79f4508c61bb8fd03345a1ced237a"
May 25 13:53:34.351884 ip-10-0-135-103 kubenswrapper[2580]: E0525 13:53:34.351868    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"514be88f1d3d8daa239e43fb4acdfc0f6bf79f4508c61bb8fd03345a1ced237a\": container with ID starting with 514be88f1d3d8daa239e43fb4acdfc0f6bf79f4508c61bb8fd03345a1ced237a not found: ID does not exist" containerID="514be88f1d3d8daa239e43fb4acdfc0f6bf79f4508c61bb8fd03345a1ced237a"
May 25 13:53:34.351936 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:53:34.351892    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"514be88f1d3d8daa239e43fb4acdfc0f6bf79f4508c61bb8fd03345a1ced237a"} err="failed to get container status \"514be88f1d3d8daa239e43fb4acdfc0f6bf79f4508c61bb8fd03345a1ced237a\": rpc error: code = NotFound desc = could not find container \"514be88f1d3d8daa239e43fb4acdfc0f6bf79f4508c61bb8fd03345a1ced237a\": container with ID starting with 514be88f1d3d8daa239e43fb4acdfc0f6bf79f4508c61bb8fd03345a1ced237a not found: ID does not exist"
May 25 13:53:34.356461 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:53:34.356440    2580 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-12313-predictor-857dcf8cc4-46rl6"]
May 25 13:53:34.359878 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:53:34.359860    2580 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-12313-predictor-857dcf8cc4-46rl6"]
May 25 13:53:34.376590 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:53:34.376571    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="51711b09-0032-45d2-99eb-00e0eff727e0" path="/var/lib/kubelet/pods/51711b09-0032-45d2-99eb-00e0eff727e0/volumes"
May 25 13:54:04.711609 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:54:04.711576    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-rg7x8_1cdfba04-46b6-40d5-9207-cce82288ca53/global-pull-secret-syncer/0.log"
May 25 13:54:04.799230 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:54:04.799194    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-rqkvc_5a8fc898-888f-420d-a433-27e831c96721/konnectivity-agent/0.log"
May 25 13:54:04.878587 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:54:04.878544    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-135-103.ec2.internal_06181dfa378829b92cc5ea772451a452/haproxy/0.log"
May 25 13:54:08.390876 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:54:08.390825    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_7734bf32-5077-42ae-ad48-28b644b05756/alertmanager/0.log"
May 25 13:54:08.422155 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:54:08.422126    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_7734bf32-5077-42ae-ad48-28b644b05756/config-reloader/0.log"
May 25 13:54:08.445565 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:54:08.445542    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_7734bf32-5077-42ae-ad48-28b644b05756/kube-rbac-proxy-web/0.log"
May 25 13:54:08.471472 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:54:08.471448    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_7734bf32-5077-42ae-ad48-28b644b05756/kube-rbac-proxy/0.log"
May 25 13:54:08.498282 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:54:08.498255    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_7734bf32-5077-42ae-ad48-28b644b05756/kube-rbac-proxy-metric/0.log"
May 25 13:54:08.524055 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:54:08.524023    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_7734bf32-5077-42ae-ad48-28b644b05756/prom-label-proxy/0.log"
May 25 13:54:08.549907 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:54:08.549880    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_7734bf32-5077-42ae-ad48-28b644b05756/init-config-reloader/0.log"
May 25 13:54:08.745254 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:54:08.745171    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-665c9c8584-jlbqb_a094f284-76a0-46e2-97ad-f6ee7d1910d5/monitoring-plugin/0.log"
May 25 13:54:08.780354 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:54:08.780330    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-j2jrt_7355e93e-5dee-448a-a74a-2f54c9d5e943/node-exporter/0.log"
May 25 13:54:08.803738 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:54:08.803712    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-j2jrt_7355e93e-5dee-448a-a74a-2f54c9d5e943/kube-rbac-proxy/0.log"
May 25 13:54:08.827510 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:54:08.827486    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-j2jrt_7355e93e-5dee-448a-a74a-2f54c9d5e943/init-textfile/0.log"
May 25 13:54:09.329805 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:54:09.329772    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-7779bbdd78-x4h7v_2eceea46-f0be-4ff7-9b90-45761f6431e0/prometheus-operator/0.log"
May 25 13:54:09.354420 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:54:09.354399    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-7779bbdd78-x4h7v_2eceea46-f0be-4ff7-9b90-45761f6431e0/kube-rbac-proxy/0.log"
May 25 13:54:09.417089 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:54:09.417058    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5dcdc9c674-ml8pd_e6b50ea7-d401-4f36-8fd3-08c48aad8efc/telemeter-client/0.log"
May 25 13:54:09.445489 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:54:09.445466    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5dcdc9c674-ml8pd_e6b50ea7-d401-4f36-8fd3-08c48aad8efc/reload/0.log"
May 25 13:54:09.470098 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:54:09.470072    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5dcdc9c674-ml8pd_e6b50ea7-d401-4f36-8fd3-08c48aad8efc/kube-rbac-proxy/0.log"
May 25 13:54:10.785460 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:54:10.785433    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-d4d8f94f9-dlbn5_dc041e36-d4ec-4dac-b8e7-d0092d61edf5/networking-console-plugin/0.log"
May 25 13:54:11.572870 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:54:11.572842    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-6c7dc66496-6dvf5_9fffd9b4-fef3-4dd5-ad3a-3fa68d4c5fad/console/0.log"
May 25 13:54:12.093619 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:54:12.093584    2580 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-must-gather-q2qb9/perf-node-gather-daemonset-74qth"]
May 25 13:54:12.093977 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:54:12.093921    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1fbb8b99-22aa-4a70-87f2-e46d72c808f6" containerName="kserve-container"
May 25 13:54:12.093977 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:54:12.093933    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="1fbb8b99-22aa-4a70-87f2-e46d72c808f6" containerName="kserve-container"
May 25 13:54:12.093977 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:54:12.093941    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62f285ff-1f15-4a53-992e-f071b91c6c0a" containerName="kube-rbac-proxy"
May 25 13:54:12.093977 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:54:12.093947    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="62f285ff-1f15-4a53-992e-f071b91c6c0a" containerName="kube-rbac-proxy"
May 25 13:54:12.093977 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:54:12.093972    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51711b09-0032-45d2-99eb-00e0eff727e0" containerName="kube-rbac-proxy"
May 25 13:54:12.093977 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:54:12.093977    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="51711b09-0032-45d2-99eb-00e0eff727e0" containerName="kube-rbac-proxy"
May 25 13:54:12.094182 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:54:12.093986    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1fbb8b99-22aa-4a70-87f2-e46d72c808f6" containerName="kube-rbac-proxy"
May 25 13:54:12.094182 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:54:12.093991    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="1fbb8b99-22aa-4a70-87f2-e46d72c808f6" containerName="kube-rbac-proxy"
May 25 13:54:12.094182 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:54:12.093998    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51711b09-0032-45d2-99eb-00e0eff727e0" containerName="kserve-container"
May 25 13:54:12.094182 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:54:12.094003    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="51711b09-0032-45d2-99eb-00e0eff727e0" containerName="kserve-container"
May 25 13:54:12.094182 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:54:12.094013    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62f285ff-1f15-4a53-992e-f071b91c6c0a" containerName="kserve-container"
May 25 13:54:12.094182 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:54:12.094018    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="62f285ff-1f15-4a53-992e-f071b91c6c0a" containerName="kserve-container"
May 25 13:54:12.094182 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:54:12.094075    2580 memory_manager.go:356] "RemoveStaleState removing state" podUID="1fbb8b99-22aa-4a70-87f2-e46d72c808f6" containerName="kube-rbac-proxy"
May 25 13:54:12.094182 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:54:12.094086    2580 memory_manager.go:356] "RemoveStaleState removing state" podUID="1fbb8b99-22aa-4a70-87f2-e46d72c808f6" containerName="kserve-container"
May 25 13:54:12.094182 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:54:12.094092    2580 memory_manager.go:356] "RemoveStaleState removing state" podUID="51711b09-0032-45d2-99eb-00e0eff727e0" containerName="kserve-container"
May 25 13:54:12.094182 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:54:12.094099    2580 memory_manager.go:356] "RemoveStaleState removing state" podUID="62f285ff-1f15-4a53-992e-f071b91c6c0a" containerName="kube-rbac-proxy"
May 25 13:54:12.094182 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:54:12.094105    2580 memory_manager.go:356] "RemoveStaleState removing state" podUID="51711b09-0032-45d2-99eb-00e0eff727e0" containerName="kube-rbac-proxy"
May 25 13:54:12.094182 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:54:12.094111    2580 memory_manager.go:356] "RemoveStaleState removing state" podUID="62f285ff-1f15-4a53-992e-f071b91c6c0a" containerName="kserve-container"
May 25 13:54:12.098240 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:54:12.098223    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-q2qb9/perf-node-gather-daemonset-74qth"
May 25 13:54:12.100822 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:54:12.100799    2580 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-q2qb9\"/\"default-dockercfg-pvqsp\""
May 25 13:54:12.101620 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:54:12.101602    2580 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-q2qb9\"/\"openshift-service-ca.crt\""
May 25 13:54:12.101718 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:54:12.101609    2580 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-q2qb9\"/\"kube-root-ca.crt\""
May 25 13:54:12.106698 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:54:12.106673    2580 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-q2qb9/perf-node-gather-daemonset-74qth"]
May 25 13:54:12.143304 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:54:12.143272    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7l5p2\" (UniqueName: \"kubernetes.io/projected/35bc6c04-a60d-43e6-95de-8420d1602278-kube-api-access-7l5p2\") pod \"perf-node-gather-daemonset-74qth\" (UID: \"35bc6c04-a60d-43e6-95de-8420d1602278\") " pod="openshift-must-gather-q2qb9/perf-node-gather-daemonset-74qth"
May 25 13:54:12.143468 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:54:12.143321    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/35bc6c04-a60d-43e6-95de-8420d1602278-proc\") pod \"perf-node-gather-daemonset-74qth\" (UID: \"35bc6c04-a60d-43e6-95de-8420d1602278\") " pod="openshift-must-gather-q2qb9/perf-node-gather-daemonset-74qth"
May 25 13:54:12.143468 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:54:12.143348    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/35bc6c04-a60d-43e6-95de-8420d1602278-podres\") pod \"perf-node-gather-daemonset-74qth\" (UID: \"35bc6c04-a60d-43e6-95de-8420d1602278\") " pod="openshift-must-gather-q2qb9/perf-node-gather-daemonset-74qth"
May 25 13:54:12.143468 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:54:12.143431    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/35bc6c04-a60d-43e6-95de-8420d1602278-sys\") pod \"perf-node-gather-daemonset-74qth\" (UID: \"35bc6c04-a60d-43e6-95de-8420d1602278\") " pod="openshift-must-gather-q2qb9/perf-node-gather-daemonset-74qth"
May 25 13:54:12.143468 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:54:12.143464    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/35bc6c04-a60d-43e6-95de-8420d1602278-lib-modules\") pod \"perf-node-gather-daemonset-74qth\" (UID: \"35bc6c04-a60d-43e6-95de-8420d1602278\") " pod="openshift-must-gather-q2qb9/perf-node-gather-daemonset-74qth"
May 25 13:54:12.244032 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:54:12.243998    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-7l5p2\" (UniqueName: \"kubernetes.io/projected/35bc6c04-a60d-43e6-95de-8420d1602278-kube-api-access-7l5p2\") pod \"perf-node-gather-daemonset-74qth\" (UID: \"35bc6c04-a60d-43e6-95de-8420d1602278\") " pod="openshift-must-gather-q2qb9/perf-node-gather-daemonset-74qth"
May 25 13:54:12.244205 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:54:12.244050    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/35bc6c04-a60d-43e6-95de-8420d1602278-proc\") pod \"perf-node-gather-daemonset-74qth\" (UID: \"35bc6c04-a60d-43e6-95de-8420d1602278\") " pod="openshift-must-gather-q2qb9/perf-node-gather-daemonset-74qth"
May 25 13:54:12.244205 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:54:12.244073    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/35bc6c04-a60d-43e6-95de-8420d1602278-podres\") pod \"perf-node-gather-daemonset-74qth\" (UID: \"35bc6c04-a60d-43e6-95de-8420d1602278\") " pod="openshift-must-gather-q2qb9/perf-node-gather-daemonset-74qth"
May 25 13:54:12.244205 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:54:12.244099    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/35bc6c04-a60d-43e6-95de-8420d1602278-sys\") pod \"perf-node-gather-daemonset-74qth\" (UID: \"35bc6c04-a60d-43e6-95de-8420d1602278\") " pod="openshift-must-gather-q2qb9/perf-node-gather-daemonset-74qth"
May 25 13:54:12.244205 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:54:12.244124    2580 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/35bc6c04-a60d-43e6-95de-8420d1602278-lib-modules\") pod \"perf-node-gather-daemonset-74qth\" (UID: \"35bc6c04-a60d-43e6-95de-8420d1602278\") " pod="openshift-must-gather-q2qb9/perf-node-gather-daemonset-74qth"
May 25 13:54:12.244413 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:54:12.244220    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/35bc6c04-a60d-43e6-95de-8420d1602278-proc\") pod \"perf-node-gather-daemonset-74qth\" (UID: \"35bc6c04-a60d-43e6-95de-8420d1602278\") " pod="openshift-must-gather-q2qb9/perf-node-gather-daemonset-74qth"
May 25 13:54:12.244413 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:54:12.244237    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/35bc6c04-a60d-43e6-95de-8420d1602278-podres\") pod \"perf-node-gather-daemonset-74qth\" (UID: \"35bc6c04-a60d-43e6-95de-8420d1602278\") " pod="openshift-must-gather-q2qb9/perf-node-gather-daemonset-74qth"
May 25 13:54:12.244413 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:54:12.244251    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/35bc6c04-a60d-43e6-95de-8420d1602278-lib-modules\") pod \"perf-node-gather-daemonset-74qth\" (UID: \"35bc6c04-a60d-43e6-95de-8420d1602278\") " pod="openshift-must-gather-q2qb9/perf-node-gather-daemonset-74qth"
May 25 13:54:12.244413 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:54:12.244216    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/35bc6c04-a60d-43e6-95de-8420d1602278-sys\") pod \"perf-node-gather-daemonset-74qth\" (UID: \"35bc6c04-a60d-43e6-95de-8420d1602278\") " pod="openshift-must-gather-q2qb9/perf-node-gather-daemonset-74qth"
May 25 13:54:12.251772 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:54:12.251750    2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7l5p2\" (UniqueName: \"kubernetes.io/projected/35bc6c04-a60d-43e6-95de-8420d1602278-kube-api-access-7l5p2\") pod \"perf-node-gather-daemonset-74qth\" (UID: \"35bc6c04-a60d-43e6-95de-8420d1602278\") " pod="openshift-must-gather-q2qb9/perf-node-gather-daemonset-74qth"
May 25 13:54:12.408419 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:54:12.408310    2580 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-q2qb9/perf-node-gather-daemonset-74qth"
May 25 13:54:12.543844 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:54:12.543815    2580 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-q2qb9/perf-node-gather-daemonset-74qth"]
May 25 13:54:12.546040 ip-10-0-135-103 kubenswrapper[2580]: W0525 13:54:12.546009    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod35bc6c04_a60d_43e6_95de_8420d1602278.slice/crio-1311ce01eef1ea8653cb5a681225c5d5fc6e591c22f4b7c4d76e1d6032ef78be WatchSource:0}: Error finding container 1311ce01eef1ea8653cb5a681225c5d5fc6e591c22f4b7c4d76e1d6032ef78be: Status 404 returned error can't find the container with id 1311ce01eef1ea8653cb5a681225c5d5fc6e591c22f4b7c4d76e1d6032ef78be
May 25 13:54:12.547638 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:54:12.547618    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 25 13:54:12.727773 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:54:12.727681    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-ltl6z_c04c0bd2-e6c2-42a0-949b-8f818987dd6b/dns/0.log"
May 25 13:54:12.748980 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:54:12.748957    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-ltl6z_c04c0bd2-e6c2-42a0-949b-8f818987dd6b/kube-rbac-proxy/0.log"
May 25 13:54:12.866992 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:54:12.866965    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-vlk4j_37b3e162-412d-4e21-8f2f-bb8398f30435/dns-node-resolver/0.log"
May 25 13:54:13.337589 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:54:13.337562    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-wlq4q_6ef8bb8d-98b5-4fe9-94a7-8e06d34ab971/node-ca/0.log"
May 25 13:54:13.464201 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:54:13.464167    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-q2qb9/perf-node-gather-daemonset-74qth" event={"ID":"35bc6c04-a60d-43e6-95de-8420d1602278","Type":"ContainerStarted","Data":"b7fc1d85074f3ca23ea722541996332af77d226f40ed503230b5b5e6ded61320"}
May 25 13:54:13.464201 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:54:13.464206    2580 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-q2qb9/perf-node-gather-daemonset-74qth" event={"ID":"35bc6c04-a60d-43e6-95de-8420d1602278","Type":"ContainerStarted","Data":"1311ce01eef1ea8653cb5a681225c5d5fc6e591c22f4b7c4d76e1d6032ef78be"}
May 25 13:54:13.464430 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:54:13.464236    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-q2qb9/perf-node-gather-daemonset-74qth"
May 25 13:54:13.482559 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:54:13.482509    2580 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-must-gather-q2qb9/perf-node-gather-daemonset-74qth" podStartSLOduration=1.4824957140000001 podStartE2EDuration="1.482495714s" podCreationTimestamp="2026-05-25 13:54:12 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 13:54:13.480634971 +0000 UTC m=+2953.672929334" watchObservedRunningTime="2026-05-25 13:54:13.482495714 +0000 UTC m=+2953.674790073"
May 25 13:54:14.033264 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:54:14.033234    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-549b58d588-dnld2_264a8628-6305-4840-9925-88aeafe69834/router/0.log"
May 25 13:54:14.376043 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:54:14.376008    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-8gtmh_2e9a6623-9ccb-484c-b669-38caefdf2976/serve-healthcheck-canary/0.log"
May 25 13:54:14.744746 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:54:14.744677    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-78bdfb888c-tnkbv_cf32b96d-4371-4b92-89db-ff71111c977a/insights-operator/0.log"
May 25 13:54:14.745095 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:54:14.745079    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-78bdfb888c-tnkbv_cf32b96d-4371-4b92-89db-ff71111c977a/insights-operator/1.log"
May 25 13:54:14.893087 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:54:14.893057    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-ql787_1f1fc207-9a8d-4a4c-beb7-bb4405007c8e/kube-rbac-proxy/0.log"
May 25 13:54:14.912463 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:54:14.912435    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-ql787_1f1fc207-9a8d-4a4c-beb7-bb4405007c8e/exporter/0.log"
May 25 13:54:14.933226 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:54:14.933202    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-ql787_1f1fc207-9a8d-4a4c-beb7-bb4405007c8e/extractor/0.log"
May 25 13:54:16.983469 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:54:16.983441    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-86f7b4b499-vxhdm_eaf98502-5fa4-474f-8567-ffa3d161e84e/server/0.log"
May 25 13:54:17.292241 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:54:17.292154    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-79f457656b-p928f_1a80a0f5-44b8-43b6-a873-f7943feed70f/seaweedfs/0.log"
May 25 13:54:19.477624 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:54:19.477591    2580 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-q2qb9/perf-node-gather-daemonset-74qth"
May 25 13:54:22.611928 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:54:22.611894    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-8xdhg_c749c4da-8f53-4bb5-98c2-280f3cd403af/kube-multus/0.log"
May 25 13:54:22.633794 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:54:22.633773    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-hj74k_33d153fd-31f6-4481-855e-728e06ce70f5/kube-multus-additional-cni-plugins/0.log"
May 25 13:54:22.659678 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:54:22.659657    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-hj74k_33d153fd-31f6-4481-855e-728e06ce70f5/egress-router-binary-copy/0.log"
May 25 13:54:22.681634 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:54:22.681618    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-hj74k_33d153fd-31f6-4481-855e-728e06ce70f5/cni-plugins/0.log"
May 25 13:54:22.702787 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:54:22.702765    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-hj74k_33d153fd-31f6-4481-855e-728e06ce70f5/bond-cni-plugin/0.log"
May 25 13:54:22.723254 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:54:22.723231    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-hj74k_33d153fd-31f6-4481-855e-728e06ce70f5/routeoverride-cni/0.log"
May 25 13:54:22.750735 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:54:22.750716    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-hj74k_33d153fd-31f6-4481-855e-728e06ce70f5/whereabouts-cni-bincopy/0.log"
May 25 13:54:22.772682 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:54:22.772663    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-hj74k_33d153fd-31f6-4481-855e-728e06ce70f5/whereabouts-cni/0.log"
May 25 13:54:23.526286 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:54:23.526256    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-svlzs_e445efe1-4ac4-4df5-881f-748bcf8c5e0f/network-metrics-daemon/0.log"
May 25 13:54:23.545099 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:54:23.545077    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-svlzs_e445efe1-4ac4-4df5-881f-748bcf8c5e0f/kube-rbac-proxy/0.log"
May 25 13:54:24.293241 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:54:24.293214    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-646r6_59b905b8-5ce3-49f6-b47c-b5b3a525fe06/ovn-controller/0.log"
May 25 13:54:24.313227 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:54:24.313204    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-646r6_59b905b8-5ce3-49f6-b47c-b5b3a525fe06/ovn-acl-logging/0.log"
May 25 13:54:24.325672 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:54:24.325650    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-646r6_59b905b8-5ce3-49f6-b47c-b5b3a525fe06/ovn-acl-logging/1.log"
May 25 13:54:24.344523 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:54:24.344503    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-646r6_59b905b8-5ce3-49f6-b47c-b5b3a525fe06/kube-rbac-proxy-node/0.log"
May 25 13:54:24.373246 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:54:24.373229    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-646r6_59b905b8-5ce3-49f6-b47c-b5b3a525fe06/kube-rbac-proxy-ovn-metrics/0.log"
May 25 13:54:24.397156 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:54:24.397138    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-646r6_59b905b8-5ce3-49f6-b47c-b5b3a525fe06/northd/0.log"
May 25 13:54:24.417351 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:54:24.417334    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-646r6_59b905b8-5ce3-49f6-b47c-b5b3a525fe06/nbdb/0.log"
May 25 13:54:24.439262 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:54:24.439242    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-646r6_59b905b8-5ce3-49f6-b47c-b5b3a525fe06/sbdb/0.log"
May 25 13:54:24.541656 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:54:24.541628    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-646r6_59b905b8-5ce3-49f6-b47c-b5b3a525fe06/ovnkube-controller/0.log"
May 25 13:54:26.167732 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:54:26.167701    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-source-7f44b6dd87-tmx95_2334e399-4548-4098-b14a-0e4b8a82e640/check-endpoints/0.log"
May 25 13:54:26.193876 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:54:26.193845    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-25spc_fdd5eb93-deb6-441f-b1fe-3511c4eee77e/network-check-target-container/0.log"
May 25 13:54:27.102572 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:54:27.102544    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-nd955_47afaed9-536d-4fc6-ab45-e56c19bb2ea6/iptables-alerter/0.log"
May 25 13:54:27.711528 ip-10-0-135-103 kubenswrapper[2580]: I0525 13:54:27.711499    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-lpfd5_38b712ff-d8de-43d6-9262-4e460ecf7b32/tuned/0.log"