May 25 12:13:16.160582 ip-10-0-142-53 systemd[1]: Starting Kubernetes Kubelet...
May 25 12:13:16.607858 ip-10-0-142-53 kubenswrapper[2549]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
May 25 12:13:16.607858 ip-10-0-142-53 kubenswrapper[2549]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
May 25 12:13:16.607858 ip-10-0-142-53 kubenswrapper[2549]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
May 25 12:13:16.607858 ip-10-0-142-53 kubenswrapper[2549]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
May 25 12:13:16.607858 ip-10-0-142-53 kubenswrapper[2549]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
May 25 12:13:16.608520 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.607915    2549 server.go:212] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
May 25 12:13:16.610593 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.610578    2549 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 25 12:13:16.610593 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.610593    2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 25 12:13:16.610651 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.610596    2549 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 25 12:13:16.610651 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.610599    2549 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 25 12:13:16.610651 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.610602    2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 25 12:13:16.610651 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.610605    2549 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 25 12:13:16.610651 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.610608    2549 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 25 12:13:16.610651 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.610611    2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 25 12:13:16.610651 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.610613    2549 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 25 12:13:16.610651 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.610616    2549 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 25 12:13:16.610651 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.610618    2549 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 25 12:13:16.610651 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.610621    2549 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 25 12:13:16.610651 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.610624    2549 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 25 12:13:16.610651 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.610627    2549 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 25 12:13:16.610651 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.610629    2549 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 25 12:13:16.610651 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.610632    2549 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 25 12:13:16.610651 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.610635    2549 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 25 12:13:16.610651 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.610637    2549 feature_gate.go:328] unrecognized feature gate: Example2
May 25 12:13:16.610651 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.610641    2549 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 25 12:13:16.610651 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.610644    2549 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 25 12:13:16.610651 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.610646    2549 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 25 12:13:16.611125 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.610649    2549 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 25 12:13:16.611125 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.610652    2549 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 25 12:13:16.611125 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.610654    2549 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 25 12:13:16.611125 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.610657    2549 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 25 12:13:16.611125 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.610660    2549 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 25 12:13:16.611125 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.610663    2549 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 25 12:13:16.611125 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.610665    2549 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 25 12:13:16.611125 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.610668    2549 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 25 12:13:16.611125 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.610671    2549 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 25 12:13:16.611125 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.610673    2549 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 25 12:13:16.611125 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.610676    2549 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 25 12:13:16.611125 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.610678    2549 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 25 12:13:16.611125 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.610681    2549 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 25 12:13:16.611125 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.610683    2549 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 25 12:13:16.611125 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.610686    2549 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 25 12:13:16.611125 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.610689    2549 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 25 12:13:16.611125 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.610691    2549 feature_gate.go:328] unrecognized feature gate: Example
May 25 12:13:16.611125 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.610693    2549 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 25 12:13:16.611125 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.610696    2549 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 25 12:13:16.611125 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.610698    2549 feature_gate.go:328] unrecognized feature gate: DualReplica
May 25 12:13:16.611622 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.610701    2549 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 25 12:13:16.611622 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.610704    2549 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 25 12:13:16.611622 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.610712    2549 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 25 12:13:16.611622 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.610715    2549 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 25 12:13:16.611622 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.610718    2549 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 25 12:13:16.611622 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.610720    2549 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 25 12:13:16.611622 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.610723    2549 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 25 12:13:16.611622 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.610727    2549 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 25 12:13:16.611622 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.610730    2549 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 25 12:13:16.611622 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.610733    2549 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 25 12:13:16.611622 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.610735    2549 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 25 12:13:16.611622 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.610738    2549 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 25 12:13:16.611622 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.610741    2549 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 25 12:13:16.611622 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.610745    2549 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 25 12:13:16.611622 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.610747    2549 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 25 12:13:16.611622 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.610750    2549 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 25 12:13:16.611622 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.610753    2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 25 12:13:16.611622 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.610756    2549 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 25 12:13:16.611622 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.610759    2549 feature_gate.go:328] unrecognized feature gate: NewOLM
May 25 12:13:16.612093 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.610761    2549 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 25 12:13:16.612093 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.610763    2549 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 25 12:13:16.612093 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.610766    2549 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 25 12:13:16.612093 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.610768    2549 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 25 12:13:16.612093 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.610771    2549 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 25 12:13:16.612093 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.610773    2549 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 25 12:13:16.612093 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.610776    2549 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 25 12:13:16.612093 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.610778    2549 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 25 12:13:16.612093 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.610782    2549 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 25 12:13:16.612093 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.610784    2549 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 25 12:13:16.612093 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.610787    2549 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 25 12:13:16.612093 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.610789    2549 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 25 12:13:16.612093 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.610792    2549 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 25 12:13:16.612093 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.610794    2549 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 25 12:13:16.612093 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.610798    2549 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 25 12:13:16.612093 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.610800    2549 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 25 12:13:16.612093 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.610803    2549 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 25 12:13:16.612093 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.610805    2549 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 25 12:13:16.612093 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.610809    2549 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 25 12:13:16.612093 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.610812    2549 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 25 12:13:16.612568 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.610815    2549 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 25 12:13:16.612568 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.610818    2549 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 25 12:13:16.612568 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.610821    2549 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 25 12:13:16.612568 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.610823    2549 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 25 12:13:16.612568 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.610826    2549 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 25 12:13:16.612568 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.610828    2549 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 25 12:13:16.612568 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.611247    2549 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 25 12:13:16.612568 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.611252    2549 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 25 12:13:16.612568 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.611257    2549 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 25 12:13:16.612568 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.611261    2549 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 25 12:13:16.612568 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.611264    2549 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 25 12:13:16.612568 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.611268    2549 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 25 12:13:16.612568 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.611270    2549 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 25 12:13:16.612568 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.611274    2549 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 25 12:13:16.612568 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.611277    2549 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 25 12:13:16.612568 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.611279    2549 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 25 12:13:16.612568 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.611282    2549 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 25 12:13:16.612568 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.611285    2549 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 25 12:13:16.612568 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.611288    2549 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 25 12:13:16.612568 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.611291    2549 feature_gate.go:328] unrecognized feature gate: Example
May 25 12:13:16.613039 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.611293    2549 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 25 12:13:16.613039 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.611296    2549 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 25 12:13:16.613039 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.611299    2549 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 25 12:13:16.613039 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.611302    2549 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 25 12:13:16.613039 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.611304    2549 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 25 12:13:16.613039 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.611307    2549 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 25 12:13:16.613039 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.611310    2549 feature_gate.go:328] unrecognized feature gate: DualReplica
May 25 12:13:16.613039 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.611312    2549 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 25 12:13:16.613039 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.611315    2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 25 12:13:16.613039 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.611318    2549 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 25 12:13:16.613039 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.611320    2549 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 25 12:13:16.613039 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.611323    2549 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 25 12:13:16.613039 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.611325    2549 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 25 12:13:16.613039 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.611328    2549 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 25 12:13:16.613039 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.611330    2549 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 25 12:13:16.613039 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.611333    2549 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 25 12:13:16.613039 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.611335    2549 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 25 12:13:16.613039 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.611339    2549 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 25 12:13:16.613039 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.611342    2549 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 25 12:13:16.613522 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.611344    2549 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 25 12:13:16.613522 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.611347    2549 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 25 12:13:16.613522 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.611350    2549 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 25 12:13:16.613522 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.611352    2549 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 25 12:13:16.613522 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.611355    2549 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 25 12:13:16.613522 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.611357    2549 feature_gate.go:328] unrecognized feature gate: NewOLM
May 25 12:13:16.613522 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.611360    2549 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 25 12:13:16.613522 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.611362    2549 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 25 12:13:16.613522 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.611364    2549 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 25 12:13:16.613522 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.611367    2549 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 25 12:13:16.613522 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.611369    2549 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 25 12:13:16.613522 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.611372    2549 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 25 12:13:16.613522 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.611374    2549 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 25 12:13:16.613522 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.611377    2549 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 25 12:13:16.613522 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.611379    2549 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 25 12:13:16.613522 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.611382    2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 25 12:13:16.613522 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.611384    2549 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 25 12:13:16.613522 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.611387    2549 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 25 12:13:16.613522 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.611389    2549 feature_gate.go:328] unrecognized feature gate: Example2
May 25 12:13:16.613522 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.611392    2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 25 12:13:16.613522 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.611395    2549 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 25 12:13:16.614033 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.611397    2549 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 25 12:13:16.614033 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.611400    2549 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 25 12:13:16.614033 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.611402    2549 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 25 12:13:16.614033 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.611405    2549 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 25 12:13:16.614033 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.611408    2549 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 25 12:13:16.614033 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.611410    2549 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 25 12:13:16.614033 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.611420    2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 25 12:13:16.614033 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.611423    2549 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 25 12:13:16.614033 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.611426    2549 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 25 12:13:16.614033 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.611428    2549 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 25 12:13:16.614033 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.611431    2549 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 25 12:13:16.614033 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.611433    2549 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 25 12:13:16.614033 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.611437    2549 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 25 12:13:16.614033 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.611441    2549 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 25 12:13:16.614033 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.611444    2549 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 25 12:13:16.614033 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.611447    2549 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 25 12:13:16.614033 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.611450    2549 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 25 12:13:16.614033 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.611452    2549 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 25 12:13:16.614033 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.611454    2549 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 25 12:13:16.614498 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.611457    2549 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 25 12:13:16.614498 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.611462    2549 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 25 12:13:16.614498 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.611465    2549 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 25 12:13:16.614498 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.611467    2549 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 25 12:13:16.614498 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.611470    2549 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 25 12:13:16.614498 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.611472    2549 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 25 12:13:16.614498 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.611475    2549 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 25 12:13:16.614498 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.611478    2549 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 25 12:13:16.614498 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.611480    2549 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 25 12:13:16.614498 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.611483    2549 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 25 12:13:16.614498 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.611485    2549 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 25 12:13:16.614498 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.611487    2549 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 25 12:13:16.614498 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.611490    2549 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 25 12:13:16.614498 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.612164    2549 flags.go:64] FLAG: --address="0.0.0.0"
May 25 12:13:16.614498 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.612173    2549 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
May 25 12:13:16.614498 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.612180    2549 flags.go:64] FLAG: --anonymous-auth="true"
May 25 12:13:16.614498 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.612184    2549 flags.go:64] FLAG: --application-metrics-count-limit="100"
May 25 12:13:16.614498 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.612188    2549 flags.go:64] FLAG: --authentication-token-webhook="false"
May 25 12:13:16.614498 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.612191    2549 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
May 25 12:13:16.614498 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.612208    2549 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
May 25 12:13:16.614498 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.612212    2549 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
May 25 12:13:16.615045 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.612215    2549 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
May 25 12:13:16.615045 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.612218    2549 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
May 25 12:13:16.615045 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.612222    2549 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
May 25 12:13:16.615045 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.612225    2549 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
May 25 12:13:16.615045 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.612228    2549 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
May 25 12:13:16.615045 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.612231    2549 flags.go:64] FLAG: --cgroup-root=""
May 25 12:13:16.615045 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.612234    2549 flags.go:64] FLAG: --cgroups-per-qos="true"
May 25 12:13:16.615045 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.612237    2549 flags.go:64] FLAG: --client-ca-file=""
May 25 12:13:16.615045 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.612239    2549 flags.go:64] FLAG: --cloud-config=""
May 25 12:13:16.615045 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.612242    2549 flags.go:64] FLAG: --cloud-provider="external"
May 25 12:13:16.615045 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.612245    2549 flags.go:64] FLAG: --cluster-dns="[]"
May 25 12:13:16.615045 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.612249    2549 flags.go:64] FLAG: --cluster-domain=""
May 25 12:13:16.615045 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.612252    2549 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
May 25 12:13:16.615045 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.612255    2549 flags.go:64] FLAG: --config-dir=""
May 25 12:13:16.615045 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.612258    2549 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
May 25 12:13:16.615045 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.612261    2549 flags.go:64] FLAG: --container-log-max-files="5"
May 25 12:13:16.615045 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.612264    2549 flags.go:64] FLAG: --container-log-max-size="10Mi"
May 25 12:13:16.615045 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.612267    2549 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
May 25 12:13:16.615045 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.612270    2549 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
May 25 12:13:16.615045 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.612273    2549 flags.go:64] FLAG: --containerd-namespace="k8s.io"
May 25 12:13:16.615045 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.612276    2549 flags.go:64] FLAG: --contention-profiling="false"
May 25 12:13:16.615045 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.612279    2549 flags.go:64] FLAG: --cpu-cfs-quota="true"
May 25 12:13:16.615045 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.612282    2549 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
May 25 12:13:16.615045 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.612285    2549 flags.go:64] FLAG: --cpu-manager-policy="none"
May 25 12:13:16.615045 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.612288    2549 flags.go:64] FLAG: --cpu-manager-policy-options=""
May 25 12:13:16.615663 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.612293    2549 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
May 25 12:13:16.615663 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.612296    2549 flags.go:64] FLAG: --enable-controller-attach-detach="true"
May 25 12:13:16.615663 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.612298    2549 flags.go:64] FLAG: --enable-debugging-handlers="true"
May 25 12:13:16.615663 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.612301    2549 flags.go:64] FLAG: --enable-load-reader="false"
May 25 12:13:16.615663 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.612304    2549 flags.go:64] FLAG: --enable-server="true"
May 25 12:13:16.615663 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.612307    2549 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
May 25 12:13:16.615663 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.612311    2549 flags.go:64] FLAG: --event-burst="100"
May 25 12:13:16.615663 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.612314    2549 flags.go:64] FLAG: --event-qps="50"
May 25 12:13:16.615663 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.612317    2549 flags.go:64] FLAG: --event-storage-age-limit="default=0"
May 25 12:13:16.615663 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.612320    2549 flags.go:64] FLAG: --event-storage-event-limit="default=0"
May 25 12:13:16.615663 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.612322    2549 flags.go:64] FLAG: --eviction-hard=""
May 25 12:13:16.615663 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.612326    2549 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
May 25 12:13:16.615663 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.612329    2549 flags.go:64] FLAG: --eviction-minimum-reclaim=""
May 25 12:13:16.615663 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.612331    2549 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
May 25 12:13:16.615663 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.612334    2549 flags.go:64] FLAG: --eviction-soft=""
May 25 12:13:16.615663 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.612337    2549 flags.go:64] FLAG: --eviction-soft-grace-period=""
May 25 12:13:16.615663 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.612339    2549 flags.go:64] FLAG: --exit-on-lock-contention="false"
May 25 12:13:16.615663 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.612342    2549 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
May 25 12:13:16.615663 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.612345    2549 flags.go:64] FLAG: --experimental-mounter-path=""
May 25 12:13:16.615663 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.612348    2549 flags.go:64] FLAG: --fail-cgroupv1="false"
May 25 12:13:16.615663 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.612351    2549 flags.go:64] FLAG: --fail-swap-on="true"
May 25 12:13:16.615663 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.612354    2549 flags.go:64] FLAG: --feature-gates=""
May 25 12:13:16.615663 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.612358    2549 flags.go:64] FLAG: --file-check-frequency="20s"
May 25 12:13:16.615663 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.612361    2549 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
May 25 12:13:16.615663 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.612364    2549 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
May 25 12:13:16.616306 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.612367    2549 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
May 25 12:13:16.616306 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.612370    2549 flags.go:64] FLAG: --healthz-port="10248"
May 25 12:13:16.616306 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.612373    2549 flags.go:64] FLAG: --help="false"
May 25 12:13:16.616306 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.612375    2549 flags.go:64] FLAG: --hostname-override="ip-10-0-142-53.ec2.internal"
May 25 12:13:16.616306 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.612379    2549 flags.go:64] FLAG: --housekeeping-interval="10s"
May 25 12:13:16.616306 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.612382    2549 flags.go:64] FLAG: --http-check-frequency="20s"
May 25 12:13:16.616306 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.612385    2549 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
May 25 12:13:16.616306 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.612388    2549 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
May 25 12:13:16.616306 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.612392    2549 flags.go:64] FLAG: --image-gc-high-threshold="85"
May 25 12:13:16.616306 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.612395    2549 flags.go:64] FLAG: --image-gc-low-threshold="80"
May 25 12:13:16.616306 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.612397    2549 flags.go:64] FLAG: --image-service-endpoint=""
May 25 12:13:16.616306 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.612400    2549 flags.go:64] FLAG: --kernel-memcg-notification="false"
May 25 12:13:16.616306 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.612403    2549 flags.go:64] FLAG: --kube-api-burst="100"
May 25 12:13:16.616306 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.612405    2549 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
May 25 12:13:16.616306 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.612409    2549 flags.go:64] FLAG: --kube-api-qps="50"
May 25 12:13:16.616306 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.612411    2549 flags.go:64] FLAG: --kube-reserved=""
May 25 12:13:16.616306 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.612414    2549 flags.go:64] FLAG: --kube-reserved-cgroup=""
May 25 12:13:16.616306 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.612417    2549 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
May 25 12:13:16.616306 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.612420    2549 flags.go:64] FLAG: --kubelet-cgroups=""
May 25 12:13:16.616306 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.612423    2549 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
May 25 12:13:16.616306 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.612425    2549 flags.go:64] FLAG: --lock-file=""
May 25 12:13:16.616306 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.612428    2549 flags.go:64] FLAG: --log-cadvisor-usage="false"
May 25 12:13:16.616306 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.612431    2549 flags.go:64] FLAG: --log-flush-frequency="5s"
May 25 12:13:16.616306 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.612433    2549 flags.go:64] FLAG: --log-json-info-buffer-size="0"
May 25 12:13:16.616860 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.612439    2549 flags.go:64] FLAG: --log-json-split-stream="false"
May 25 12:13:16.616860 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.612442    2549 flags.go:64] FLAG: --log-text-info-buffer-size="0"
May 25 12:13:16.616860 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.612445    2549 flags.go:64] FLAG: --log-text-split-stream="false"
May 25 12:13:16.616860 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.612448    2549 flags.go:64] FLAG: --logging-format="text"
May 25 12:13:16.616860 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.612451    2549 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
May 25 12:13:16.616860 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.612454    2549 flags.go:64] FLAG: --make-iptables-util-chains="true"
May 25 12:13:16.616860 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.612457    2549 flags.go:64] FLAG: --manifest-url=""
May 25 12:13:16.616860 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.612460    2549 flags.go:64] FLAG: --manifest-url-header=""
May 25 12:13:16.616860 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.612464    2549 flags.go:64] FLAG: --max-housekeeping-interval="15s"
May 25 12:13:16.616860 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.612466    2549 flags.go:64] FLAG: --max-open-files="1000000"
May 25 12:13:16.616860 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.612470    2549 flags.go:64] FLAG: --max-pods="110"
May 25 12:13:16.616860 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.612473    2549 flags.go:64] FLAG: --maximum-dead-containers="-1"
May 25 12:13:16.616860 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.612476    2549 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
May 25 12:13:16.616860 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.612479    2549 flags.go:64] FLAG: --memory-manager-policy="None"
May 25 12:13:16.616860 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.612482    2549 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
May 25 12:13:16.616860 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.612484    2549 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
May 25 12:13:16.616860 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.612488    2549 flags.go:64] FLAG: --node-ip="0.0.0.0"
May 25 12:13:16.616860 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.612491    2549 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
May 25 12:13:16.616860 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.612499    2549 flags.go:64] FLAG: --node-status-max-images="50"
May 25 12:13:16.616860 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.612502    2549 flags.go:64] FLAG: --node-status-update-frequency="10s"
May 25 12:13:16.616860 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.612504    2549 flags.go:64] FLAG: --oom-score-adj="-999"
May 25 12:13:16.616860 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.612507    2549 flags.go:64] FLAG: --pod-cidr=""
May 25 12:13:16.616860 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.612510    2549 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:afe88671bf163c42b467d159dc6f0b784dbe28b1ed44e171edc9c06fd99cc6fc"
May 25 12:13:16.617454 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.612516    2549 flags.go:64] FLAG: --pod-manifest-path=""
May 25 12:13:16.617454 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.612518    2549 flags.go:64] FLAG: --pod-max-pids="-1"
May 25 12:13:16.617454 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.612521    2549 flags.go:64] FLAG: --pods-per-core="0"
May 25 12:13:16.617454 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.612524    2549 flags.go:64] FLAG: --port="10250"
May 25 12:13:16.617454 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.612527    2549 flags.go:64] FLAG: --protect-kernel-defaults="false"
May 25 12:13:16.617454 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.612530    2549 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-04a894c610d692d5d"
May 25 12:13:16.617454 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.612533    2549 flags.go:64] FLAG: --qos-reserved=""
May 25 12:13:16.617454 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.612536    2549 flags.go:64] FLAG: --read-only-port="10255"
May 25 12:13:16.617454 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.612539    2549 flags.go:64] FLAG: --register-node="true"
May 25 12:13:16.617454 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.612541    2549 flags.go:64] FLAG: --register-schedulable="true"
May 25 12:13:16.617454 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.612544    2549 flags.go:64] FLAG: --register-with-taints=""
May 25 12:13:16.617454 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.612548    2549 flags.go:64] FLAG: --registry-burst="10"
May 25 12:13:16.617454 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.612551    2549 flags.go:64] FLAG: --registry-qps="5"
May 25 12:13:16.617454 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.612553    2549 flags.go:64] FLAG: --reserved-cpus=""
May 25 12:13:16.617454 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.612556    2549 flags.go:64] FLAG: --reserved-memory=""
May 25 12:13:16.617454 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.612560    2549 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
May 25 12:13:16.617454 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.612563    2549 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
May 25 12:13:16.617454 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.612566    2549 flags.go:64] FLAG: --rotate-certificates="false"
May 25 12:13:16.617454 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.612568    2549 flags.go:64] FLAG: --rotate-server-certificates="false"
May 25 12:13:16.617454 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.612571    2549 flags.go:64] FLAG: --runonce="false"
May 25 12:13:16.617454 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.612574    2549 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
May 25 12:13:16.617454 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.612578    2549 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
May 25 12:13:16.617454 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.612581    2549 flags.go:64] FLAG: --seccomp-default="false"
May 25 12:13:16.617454 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.612584    2549 flags.go:64] FLAG: --serialize-image-pulls="true"
May 25 12:13:16.617454 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.612587    2549 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
May 25 12:13:16.617454 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.612590    2549 flags.go:64] FLAG: --storage-driver-db="cadvisor"
May 25 12:13:16.618449 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.612593    2549 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
May 25 12:13:16.618449 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.612597    2549 flags.go:64] FLAG: --storage-driver-password="root"
May 25 12:13:16.618449 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.612600    2549 flags.go:64] FLAG: --storage-driver-secure="false"
May 25 12:13:16.618449 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.612602    2549 flags.go:64] FLAG: --storage-driver-table="stats"
May 25 12:13:16.618449 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.612605    2549 flags.go:64] FLAG: --storage-driver-user="root"
May 25 12:13:16.618449 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.612608    2549 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
May 25 12:13:16.618449 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.612611    2549 flags.go:64] FLAG: --sync-frequency="1m0s"
May 25 12:13:16.618449 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.612614    2549 flags.go:64] FLAG: --system-cgroups=""
May 25 12:13:16.618449 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.612617    2549 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
May 25 12:13:16.618449 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.612622    2549 flags.go:64] FLAG: --system-reserved-cgroup=""
May 25 12:13:16.618449 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.612625    2549 flags.go:64] FLAG: --tls-cert-file=""
May 25 12:13:16.618449 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.612627    2549 flags.go:64] FLAG: --tls-cipher-suites="[]"
May 25 12:13:16.618449 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.612653    2549 flags.go:64] FLAG: --tls-min-version=""
May 25 12:13:16.618449 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.612656    2549 flags.go:64] FLAG: --tls-private-key-file=""
May 25 12:13:16.618449 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.612659    2549 flags.go:64] FLAG: --topology-manager-policy="none"
May 25 12:13:16.618449 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.612662    2549 flags.go:64] FLAG: --topology-manager-policy-options=""
May 25 12:13:16.618449 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.612665    2549 flags.go:64] FLAG: --topology-manager-scope="container"
May 25 12:13:16.618449 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.612668    2549 flags.go:64] FLAG: --v="2"
May 25 12:13:16.618449 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.612674    2549 flags.go:64] FLAG: --version="false"
May 25 12:13:16.618449 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.612678    2549 flags.go:64] FLAG: --vmodule=""
May 25 12:13:16.618449 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.612682    2549 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
May 25 12:13:16.618449 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.612685    2549 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
May 25 12:13:16.618449 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.612772    2549 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 25 12:13:16.618449 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.612776    2549 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 25 12:13:16.619465 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.612779    2549 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 25 12:13:16.619465 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.612782    2549 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 25 12:13:16.619465 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.612785    2549 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 25 12:13:16.619465 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.612789    2549 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 25 12:13:16.619465 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.612792    2549 feature_gate.go:328] unrecognized feature gate: Example2
May 25 12:13:16.619465 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.612795    2549 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 25 12:13:16.619465 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.612797    2549 feature_gate.go:328] unrecognized feature gate: Example
May 25 12:13:16.619465 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.612801    2549 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 25 12:13:16.619465 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.612803    2549 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 25 12:13:16.619465 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.612806    2549 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 25 12:13:16.619465 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.612809    2549 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 25 12:13:16.619465 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.612811    2549 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 25 12:13:16.619465 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.612814    2549 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 25 12:13:16.619465 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.612816    2549 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 25 12:13:16.619465 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.612819    2549 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 25 12:13:16.619465 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.612822    2549 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 25 12:13:16.619465 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.612824    2549 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 25 12:13:16.619465 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.612827    2549 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 25 12:13:16.619465 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.612829    2549 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 25 12:13:16.619465 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.612832    2549 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 25 12:13:16.620416 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.612835    2549 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 25 12:13:16.620416 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.612839    2549 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 25 12:13:16.620416 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.612842    2549 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 25 12:13:16.620416 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.612844    2549 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 25 12:13:16.620416 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.612847    2549 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 25 12:13:16.620416 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.612849    2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 25 12:13:16.620416 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.612854    2549 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 25 12:13:16.620416 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.612857    2549 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 25 12:13:16.620416 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.612859    2549 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 25 12:13:16.620416 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.612862    2549 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 25 12:13:16.620416 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.612864    2549 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 25 12:13:16.620416 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.612867    2549 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 25 12:13:16.620416 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.612869    2549 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 25 12:13:16.620416 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.612872    2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 25 12:13:16.620416 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.612874    2549 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 25 12:13:16.620416 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.612878    2549 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 25 12:13:16.620416 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.612881    2549 feature_gate.go:328] unrecognized feature gate: DualReplica
May 25 12:13:16.620416 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.612884    2549 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 25 12:13:16.620416 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.612886    2549 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 25 12:13:16.620416 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.612889    2549 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 25 12:13:16.621116 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.612891    2549 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 25 12:13:16.621116 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.612894    2549 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 25 12:13:16.621116 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.612896    2549 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 25 12:13:16.621116 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.612899    2549 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 25 12:13:16.621116 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.612902    2549 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 25 12:13:16.621116 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.612904    2549 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 25 12:13:16.621116 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.612906    2549 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 25 12:13:16.621116 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.612909    2549 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 25 12:13:16.621116 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.612911    2549 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 25 12:13:16.621116 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.612914    2549 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 25 12:13:16.621116 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.612917    2549 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 25 12:13:16.621116 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.612919    2549 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 25 12:13:16.621116 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.612922    2549 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 25 12:13:16.621116 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.612924    2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 25 12:13:16.621116 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.612927    2549 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 25 12:13:16.621116 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.612929    2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 25 12:13:16.621116 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.612932    2549 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 25 12:13:16.621116 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.612935    2549 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 25 12:13:16.621116 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.612939    2549 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 25 12:13:16.621624 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.612941    2549 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 25 12:13:16.621624 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.612944    2549 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 25 12:13:16.621624 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.612947    2549 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 25 12:13:16.621624 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.612949    2549 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 25 12:13:16.621624 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.612952    2549 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 25 12:13:16.621624 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.612954    2549 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 25 12:13:16.621624 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.612957    2549 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 25 12:13:16.621624 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.612959    2549 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 25 12:13:16.621624 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.612963    2549 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 25 12:13:16.621624 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.612965    2549 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 25 12:13:16.621624 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.612968    2549 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 25 12:13:16.621624 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.612970    2549 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 25 12:13:16.621624 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.612973    2549 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 25 12:13:16.621624 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.612975    2549 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 25 12:13:16.621624 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.612978    2549 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 25 12:13:16.621624 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.612980    2549 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 25 12:13:16.621624 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.612982    2549 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 25 12:13:16.621624 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.612985    2549 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 25 12:13:16.621624 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.612988    2549 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 25 12:13:16.621624 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.612990    2549 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 25 12:13:16.622135 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.612993    2549 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 25 12:13:16.622135 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.612995    2549 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 25 12:13:16.622135 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.612997    2549 feature_gate.go:328] unrecognized feature gate: NewOLM
May 25 12:13:16.622135 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.613000    2549 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 25 12:13:16.622135 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.613004    2549 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 25 12:13:16.622135 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.613009    2549 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]}
May 25 12:13:16.622135 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.619719    2549 server.go:530] "Kubelet version" kubeletVersion="v1.33.11"
May 25 12:13:16.622135 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.619737    2549 server.go:532] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
May 25 12:13:16.622135 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.619832    2549 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 25 12:13:16.622135 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.619841    2549 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 25 12:13:16.622135 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.619846    2549 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 25 12:13:16.622135 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.619851    2549 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 25 12:13:16.622135 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.619855    2549 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 25 12:13:16.622135 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.619860    2549 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 25 12:13:16.622135 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.619864    2549 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 25 12:13:16.622552 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.619868    2549 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 25 12:13:16.622552 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.619873    2549 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 25 12:13:16.622552 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.619877    2549 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 25 12:13:16.622552 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.619884    2549 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 25 12:13:16.622552 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.619890    2549 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 25 12:13:16.622552 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.619895    2549 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 25 12:13:16.622552 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.619899    2549 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 25 12:13:16.622552 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.619904    2549 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 25 12:13:16.622552 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.619908    2549 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 25 12:13:16.622552 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.619912    2549 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 25 12:13:16.622552 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.619917    2549 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 25 12:13:16.622552 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.619921    2549 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 25 12:13:16.622552 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.619925    2549 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 25 12:13:16.622552 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.619930    2549 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 25 12:13:16.622552 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.619935    2549 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 25 12:13:16.622552 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.619939    2549 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 25 12:13:16.622552 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.619944    2549 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 25 12:13:16.622552 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.619948    2549 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 25 12:13:16.622552 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.619952    2549 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 25 12:13:16.623370 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.619956    2549 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 25 12:13:16.623370 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.619961    2549 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 25 12:13:16.623370 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.619965    2549 feature_gate.go:328] unrecognized feature gate: NewOLM
May 25 12:13:16.623370 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.619969    2549 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 25 12:13:16.623370 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.619974    2549 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 25 12:13:16.623370 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.619978    2549 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 25 12:13:16.623370 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.619983    2549 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 25 12:13:16.623370 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.619987    2549 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 25 12:13:16.623370 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.619992    2549 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 25 12:13:16.623370 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.619997    2549 feature_gate.go:328] unrecognized feature gate: Example
May 25 12:13:16.623370 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.620001    2549 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 25 12:13:16.623370 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.620005    2549 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 25 12:13:16.623370 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.620009    2549 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 25 12:13:16.623370 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.620013    2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 25 12:13:16.623370 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.620018    2549 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 25 12:13:16.623370 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.620022    2549 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 25 12:13:16.623370 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.620026    2549 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 25 12:13:16.623370 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.620030    2549 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 25 12:13:16.623370 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.620035    2549 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 25 12:13:16.623370 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.620039    2549 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 25 12:13:16.624169 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.620043    2549 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 25 12:13:16.624169 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.620048    2549 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 25 12:13:16.624169 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.620052    2549 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 25 12:13:16.624169 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.620056    2549 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 25 12:13:16.624169 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.620061    2549 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 25 12:13:16.624169 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.620065    2549 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 25 12:13:16.624169 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.620081    2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 25 12:13:16.624169 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.620085    2549 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 25 12:13:16.624169 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.620090    2549 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 25 12:13:16.624169 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.620095    2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 25 12:13:16.624169 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.620099    2549 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 25 12:13:16.624169 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.620103    2549 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 25 12:13:16.624169 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.620107    2549 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 25 12:13:16.624169 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.620111    2549 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 25 12:13:16.624169 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.620115    2549 feature_gate.go:328] unrecognized feature gate: DualReplica
May 25 12:13:16.624169 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.620119    2549 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 25 12:13:16.624169 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.620123    2549 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 25 12:13:16.624169 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.620128    2549 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 25 12:13:16.624169 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.620132    2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 25 12:13:16.624169 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.620138    2549 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 25 12:13:16.624680 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.620145    2549 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 25 12:13:16.624680 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.620149    2549 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 25 12:13:16.624680 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.620153    2549 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 25 12:13:16.624680 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.620157    2549 feature_gate.go:328] unrecognized feature gate: Example2
May 25 12:13:16.624680 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.620161    2549 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 25 12:13:16.624680 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.620166    2549 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 25 12:13:16.624680 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.620170    2549 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 25 12:13:16.624680 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.620175    2549 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 25 12:13:16.624680 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.620179    2549 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 25 12:13:16.624680 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.620184    2549 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 25 12:13:16.624680 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.620188    2549 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 25 12:13:16.624680 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.620208    2549 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 25 12:13:16.624680 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.620213    2549 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 25 12:13:16.624680 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.620217    2549 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 25 12:13:16.624680 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.620220    2549 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 25 12:13:16.624680 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.620224    2549 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 25 12:13:16.624680 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.620228    2549 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 25 12:13:16.624680 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.620232    2549 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 25 12:13:16.624680 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.620236    2549 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 25 12:13:16.624680 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.620240    2549 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 25 12:13:16.625371 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.620247    2549 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]}
May 25 12:13:16.625371 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.620397    2549 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 25 12:13:16.625371 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.620405    2549 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 25 12:13:16.625371 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.620410    2549 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 25 12:13:16.625371 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.620414    2549 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 25 12:13:16.625371 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.620418    2549 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 25 12:13:16.625371 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.620422    2549 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 25 12:13:16.625371 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.620426    2549 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 25 12:13:16.625371 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.620431    2549 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 25 12:13:16.625371 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.620438    2549 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 25 12:13:16.625371 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.620442    2549 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 25 12:13:16.625371 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.620446    2549 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 25 12:13:16.625371 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.620450    2549 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 25 12:13:16.625371 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.620454    2549 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 25 12:13:16.625722 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.620458    2549 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 25 12:13:16.625722 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.620462    2549 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 25 12:13:16.625722 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.620466    2549 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 25 12:13:16.625722 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.620470    2549 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 25 12:13:16.625722 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.620474    2549 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 25 12:13:16.625722 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.620479    2549 feature_gate.go:328] unrecognized feature gate: Example
May 25 12:13:16.625722 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.620483    2549 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 25 12:13:16.625722 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.620487    2549 feature_gate.go:328] unrecognized feature gate: NewOLM
May 25 12:13:16.625722 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.620491    2549 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 25 12:13:16.625722 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.620495    2549 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 25 12:13:16.625722 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.620499    2549 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 25 12:13:16.625722 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.620503    2549 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 25 12:13:16.625722 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.620507    2549 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 25 12:13:16.625722 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.620511    2549 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 25 12:13:16.625722 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.620515    2549 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 25 12:13:16.625722 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.620519    2549 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 25 12:13:16.625722 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.620523    2549 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 25 12:13:16.625722 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.620527    2549 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 25 12:13:16.625722 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.620531    2549 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 25 12:13:16.625722 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.620536    2549 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 25 12:13:16.626255 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.620541    2549 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 25 12:13:16.626255 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.620545    2549 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 25 12:13:16.626255 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.620550    2549 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 25 12:13:16.626255 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.620553    2549 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 25 12:13:16.626255 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.620558    2549 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 25 12:13:16.626255 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.620561    2549 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 25 12:13:16.626255 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.620565    2549 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 25 12:13:16.626255 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.620569    2549 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 25 12:13:16.626255 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.620573    2549 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 25 12:13:16.626255 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.620577    2549 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 25 12:13:16.626255 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.620581    2549 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 25 12:13:16.626255 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.620585    2549 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 25 12:13:16.626255 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.620589    2549 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 25 12:13:16.626255 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.620592    2549 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 25 12:13:16.626255 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.620596    2549 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 25 12:13:16.626255 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.620600    2549 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 25 12:13:16.626255 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.620604    2549 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 25 12:13:16.626255 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.620608    2549 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 25 12:13:16.626255 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.620612    2549 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 25 12:13:16.626255 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.620616    2549 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 25 12:13:16.626767 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.620620    2549 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 25 12:13:16.626767 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.620624    2549 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 25 12:13:16.626767 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.620628    2549 feature_gate.go:328] unrecognized feature gate: DualReplica
May 25 12:13:16.626767 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.620632    2549 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 25 12:13:16.626767 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.620637    2549 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 25 12:13:16.626767 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.620641    2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 25 12:13:16.626767 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.620645    2549 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 25 12:13:16.626767 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.620649    2549 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 25 12:13:16.626767 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.620652    2549 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 25 12:13:16.626767 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.620656    2549 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 25 12:13:16.626767 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.620660    2549 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 25 12:13:16.626767 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.620664    2549 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 25 12:13:16.626767 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.620668    2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 25 12:13:16.626767 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.620672    2549 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 25 12:13:16.626767 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.620676    2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 25 12:13:16.626767 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.620680    2549 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 25 12:13:16.626767 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.620685    2549 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 25 12:13:16.626767 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.620688    2549 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 25 12:13:16.626767 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.620692    2549 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 25 12:13:16.626767 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.620696    2549 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 25 12:13:16.627307 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.620700    2549 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 25 12:13:16.627307 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.620704    2549 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 25 12:13:16.627307 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.620708    2549 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 25 12:13:16.627307 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.620714    2549 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 25 12:13:16.627307 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.620719    2549 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 25 12:13:16.627307 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.620723    2549 feature_gate.go:328] unrecognized feature gate: Example2
May 25 12:13:16.627307 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.620727    2549 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 25 12:13:16.627307 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.620731    2549 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 25 12:13:16.627307 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.620734    2549 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 25 12:13:16.627307 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.620739    2549 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 25 12:13:16.627307 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.620743    2549 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 25 12:13:16.627307 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.620747    2549 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 25 12:13:16.627307 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:16.620750    2549 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 25 12:13:16.627307 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.620758    2549 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]}
May 25 12:13:16.627307 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.622065    2549 server.go:962] "Client rotation is on, will bootstrap in background"
May 25 12:13:16.627669 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.624824    2549 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
May 25 12:13:16.627669 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.625725    2549 server.go:1019] "Starting client certificate rotation"
May 25 12:13:16.627669 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.625844    2549 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
May 25 12:13:16.627669 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.626512    2549 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
May 25 12:13:16.648936 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.648909    2549 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
May 25 12:13:16.653599 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.653581    2549 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
May 25 12:13:16.667678 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.667660    2549 log.go:25] "Validated CRI v1 runtime API"
May 25 12:13:16.673344 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.673330    2549 log.go:25] "Validated CRI v1 image API"
May 25 12:13:16.674529 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.674508    2549 server.go:1452] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
May 25 12:13:16.679425 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.679405    2549 fs.go:135] Filesystem UUIDs: map[1ed5017a-5a3e-4b31-8fc9-a22a3785552b:/dev/nvme0n1p3 5ed374b2-4509-4566-a832-4a744deb4314:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2]
May 25 12:13:16.679478 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.679425    2549 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
May 25 12:13:16.683631 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.683527    2549 manager.go:217] Machine: {Timestamp:2026-05-25 12:13:16.683030422 +0000 UTC m=+0.398845699 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3160704 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2c2481aad0f77b3a6d2d468446f45c SystemUUID:ec2c2481-aad0-f77b-3a6d-2d468446f45c BootID:56f6ade7-b7b0-4b8d-8cab-b133f02b889b Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6103040 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:d0:83:48:99:09 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:d0:83:48:99:09 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:1a:82:73:94:ff:bd Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
May 25 12:13:16.684331 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.684320    2549 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
May 25 12:13:16.684408 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.684397    2549 manager.go:233] Version: {KernelVersion:5.14.0-570.114.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260512-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
May 25 12:13:16.685907 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.685874    2549 container_manager_linux.go:270] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
May 25 12:13:16.685907 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.685899    2549 reflector.go:430] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
May 25 12:13:16.686046 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.685910    2549 container_manager_linux.go:275] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-142-53.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
May 25 12:13:16.686095 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.686055    2549 topology_manager.go:138] "Creating topology manager with none policy"
May 25 12:13:16.686095 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.686063    2549 container_manager_linux.go:306] "Creating device plugin manager"
May 25 12:13:16.686095 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.686070    2549 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
May 25 12:13:16.686095 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.686083    2549 server.go:72] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
May 25 12:13:16.687257 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.687246    2549 state_mem.go:36] "Initialized new in-memory state store"
May 25 12:13:16.687358 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.687350    2549 server.go:1267] "Using root directory" path="/var/lib/kubelet"
May 25 12:13:16.690068 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.690058    2549 kubelet.go:491] "Attempting to sync node with API server"
May 25 12:13:16.690106 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.690077    2549 kubelet.go:386] "Adding static pod path" path="/etc/kubernetes/manifests"
May 25 12:13:16.690106 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.690089    2549 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
May 25 12:13:16.690106 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.690100    2549 kubelet.go:397] "Adding apiserver pod source"
May 25 12:13:16.690225 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.690108    2549 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
May 25 12:13:16.691137 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.691125    2549 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
May 25 12:13:16.691173 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.691145    2549 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
May 25 12:13:16.694342 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.694325    2549 kuberuntime_manager.go:279] "Container runtime initialized" containerRuntime="cri-o" version="1.33.11-2.rhaos4.20.gitb2a8320.el9" apiVersion="v1"
May 25 12:13:16.696144 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.696130    2549 kubelet.go:953] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
May 25 12:13:16.696855 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.696844    2549 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
May 25 12:13:16.696895 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.696861    2549 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
May 25 12:13:16.696895 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.696867    2549 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
May 25 12:13:16.696895 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.696873    2549 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
May 25 12:13:16.696895 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.696881    2549 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
May 25 12:13:16.696895 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.696887    2549 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/secret"
May 25 12:13:16.696895 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.696893    2549 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
May 25 12:13:16.697041 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.696898    2549 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
May 25 12:13:16.697041 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.696905    2549 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/fc"
May 25 12:13:16.697041 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.696911    2549 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
May 25 12:13:16.697041 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.696924    2549 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/projected"
May 25 12:13:16.697138 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.697123    2549 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
May 25 12:13:16.699101 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.699089    2549 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/csi"
May 25 12:13:16.699139 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.699103    2549 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/image"
May 25 12:13:16.702812 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.702791    2549 watchdog_linux.go:99] "Systemd watchdog is not enabled"
May 25 12:13:16.702935 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.702814    2549 csi_plugin.go:988] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-142-53.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
May 25 12:13:16.702935 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.702833    2549 server.go:1295] "Started kubelet"
May 25 12:13:16.703056 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.702960    2549 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
May 25 12:13:16.703056 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.702965    2549 ratelimit.go:55] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
May 25 12:13:16.703056 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.703022    2549 server_v1.go:47] "podresources" method="list" useActivePods=true
May 25 12:13:16.703723 ip-10-0-142-53 systemd[1]: Started Kubernetes Kubelet.
May 25 12:13:16.704353 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.704340    2549 server.go:255] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
May 25 12:13:16.705647 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.705631    2549 server.go:317] "Adding debug handlers to kubelet server"
May 25 12:13:16.710070 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.710054    2549 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
May 25 12:13:16.710627 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.710609    2549 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
May 25 12:13:16.711147 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:13:16.711118    2549 reflector.go:200] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
May 25 12:13:16.711382 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.711359    2549 volume_manager.go:295] "The desired_state_of_world populator starts"
May 25 12:13:16.711382 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.711382    2549 volume_manager.go:297] "Starting Kubelet Volume Manager"
May 25 12:13:16.711516 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.711505    2549 desired_state_of_world_populator.go:150] "Desired state populator starts to run"
May 25 12:13:16.711610 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.711569    2549 reconstruct.go:97] "Volume reconstruction finished"
May 25 12:13:16.711610 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.711580    2549 reconciler.go:26] "Reconciler: start to sync state"
May 25 12:13:16.712381 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:13:16.712355    2549 reflector.go:200] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-142-53.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
May 25 12:13:16.712728 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:13:16.712705    2549 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-142-53.ec2.internal\" not found"
May 25 12:13:16.712841 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:13:16.711175    2549 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-53.ec2.internal.18b2cde7724194db  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-53.ec2.internal,UID:ip-10-0-142-53.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-142-53.ec2.internal,},FirstTimestamp:2026-05-25 12:13:16.702807259 +0000 UTC m=+0.418622540,LastTimestamp:2026-05-25 12:13:16.702807259 +0000 UTC m=+0.418622540,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-53.ec2.internal,}"
May 25 12:13:16.713427 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.713410    2549 factory.go:55] Registering systemd factory
May 25 12:13:16.713427 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.713431    2549 factory.go:223] Registration of the systemd container factory successfully
May 25 12:13:16.713886 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.713867    2549 factory.go:153] Registering CRI-O factory
May 25 12:13:16.713886 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.713887    2549 factory.go:223] Registration of the crio container factory successfully
May 25 12:13:16.714017 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.713936    2549 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
May 25 12:13:16.714017 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.713955    2549 factory.go:103] Registering Raw factory
May 25 12:13:16.714017 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.713974    2549 manager.go:1196] Started watching for new ooms in manager
May 25 12:13:16.714375 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.714358    2549 manager.go:319] Starting recovery of all containers
May 25 12:13:16.715357 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:13:16.715330    2549 kubelet.go:1618] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
May 25 12:13:16.715545 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:13:16.715523    2549 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-142-53.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
May 25 12:13:16.715664 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:13:16.715642    2549 reflector.go:200] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
May 25 12:13:16.716885 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.716865    2549 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-nfspq"
May 25 12:13:16.726077 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.725924    2549 manager.go:324] Recovery completed
May 25 12:13:16.729886 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.729875    2549 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 25 12:13:16.730973 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.730956    2549 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-nfspq"
May 25 12:13:16.731935 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.731917    2549 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-142-53.ec2.internal" event="NodeHasSufficientMemory"
May 25 12:13:16.732021 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.731949    2549 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-142-53.ec2.internal" event="NodeHasNoDiskPressure"
May 25 12:13:16.732021 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.731964    2549 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-142-53.ec2.internal" event="NodeHasSufficientPID"
May 25 12:13:16.732515 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.732503    2549 cpu_manager.go:222] "Starting CPU manager" policy="none"
May 25 12:13:16.732515 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.732513    2549 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
May 25 12:13:16.732611 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.732528    2549 state_mem.go:36] "Initialized new in-memory state store"
May 25 12:13:16.733988 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:13:16.733905    2549 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-53.ec2.internal.18b2cde773fe0475  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-53.ec2.internal,UID:ip-10-0-142-53.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-142-53.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-142-53.ec2.internal,},FirstTimestamp:2026-05-25 12:13:16.731933813 +0000 UTC m=+0.447749094,LastTimestamp:2026-05-25 12:13:16.731933813 +0000 UTC m=+0.447749094,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-53.ec2.internal,}"
May 25 12:13:16.734630 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.734619    2549 policy_none.go:49] "None policy: Start"
May 25 12:13:16.734663 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.734635    2549 memory_manager.go:186] "Starting memorymanager" policy="None"
May 25 12:13:16.734663 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.734647    2549 state_mem.go:35] "Initializing new in-memory state store"
May 25 12:13:16.773781 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.773762    2549 manager.go:341] "Starting Device Plugin manager"
May 25 12:13:16.784518 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:13:16.773793    2549 manager.go:517] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
May 25 12:13:16.784518 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.773803    2549 server.go:85] "Starting device plugin registration server"
May 25 12:13:16.784518 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.774041    2549 eviction_manager.go:189] "Eviction manager: starting control loop"
May 25 12:13:16.784518 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.774055    2549 container_log_manager.go:189] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
May 25 12:13:16.784518 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.774185    2549 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
May 25 12:13:16.784518 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.774276    2549 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
May 25 12:13:16.784518 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.774284    2549 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
May 25 12:13:16.784518 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:13:16.774719    2549 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
May 25 12:13:16.784518 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:13:16.774747    2549 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-142-53.ec2.internal\" not found"
May 25 12:13:16.841322 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.841292    2549 kubelet_network_linux.go:49] "Initialized iptables rules." protocol="IPv4"
May 25 12:13:16.842448 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.842432    2549 kubelet_network_linux.go:49] "Initialized iptables rules." protocol="IPv6"
May 25 12:13:16.842559 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.842456    2549 status_manager.go:230] "Starting to sync pod status with apiserver"
May 25 12:13:16.842559 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.842474    2549 watchdog_linux.go:127] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
May 25 12:13:16.842559 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.842481    2549 kubelet.go:2451] "Starting kubelet main sync loop"
May 25 12:13:16.842559 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:13:16.842523    2549 kubelet.go:2475] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
May 25 12:13:16.846010 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.845991    2549 reflector.go:430] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
May 25 12:13:16.874710 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.874689    2549 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 25 12:13:16.875598 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.875584    2549 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-142-53.ec2.internal" event="NodeHasSufficientMemory"
May 25 12:13:16.875668 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.875611    2549 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-142-53.ec2.internal" event="NodeHasNoDiskPressure"
May 25 12:13:16.875668 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.875621    2549 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-142-53.ec2.internal" event="NodeHasSufficientPID"
May 25 12:13:16.875668 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.875645    2549 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-142-53.ec2.internal"
May 25 12:13:16.888272 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.888255    2549 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-142-53.ec2.internal"
May 25 12:13:16.888326 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:13:16.888287    2549 kubelet_node_status.go:597] "Error updating node status, will retry" err="error getting node \"ip-10-0-142-53.ec2.internal\": node \"ip-10-0-142-53.ec2.internal\" not found"
May 25 12:13:16.911098 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:13:16.911079    2549 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-142-53.ec2.internal\" not found"
May 25 12:13:16.942591 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.942566    2549 kubelet.go:2537] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-53.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-142-53.ec2.internal"]
May 25 12:13:16.942675 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.942628    2549 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 25 12:13:16.943584 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.943571    2549 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-142-53.ec2.internal" event="NodeHasSufficientMemory"
May 25 12:13:16.943640 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.943598    2549 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-142-53.ec2.internal" event="NodeHasNoDiskPressure"
May 25 12:13:16.943640 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.943611    2549 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-142-53.ec2.internal" event="NodeHasSufficientPID"
May 25 12:13:16.946000 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.945988    2549 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 25 12:13:16.946131 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.946116    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-53.ec2.internal"
May 25 12:13:16.946173 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.946148    2549 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 25 12:13:16.947489 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.947463    2549 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-142-53.ec2.internal" event="NodeHasSufficientMemory"
May 25 12:13:16.947575 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.947471    2549 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-142-53.ec2.internal" event="NodeHasSufficientMemory"
May 25 12:13:16.947575 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.947496    2549 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-142-53.ec2.internal" event="NodeHasNoDiskPressure"
May 25 12:13:16.947575 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.947508    2549 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-142-53.ec2.internal" event="NodeHasSufficientPID"
May 25 12:13:16.947575 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.947516    2549 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-142-53.ec2.internal" event="NodeHasNoDiskPressure"
May 25 12:13:16.947575 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.947528    2549 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-142-53.ec2.internal" event="NodeHasSufficientPID"
May 25 12:13:16.949707 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.949689    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-53.ec2.internal"
May 25 12:13:16.949780 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.949724    2549 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 25 12:13:16.950358 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.950346    2549 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-142-53.ec2.internal" event="NodeHasSufficientMemory"
May 25 12:13:16.950453 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.950368    2549 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-142-53.ec2.internal" event="NodeHasNoDiskPressure"
May 25 12:13:16.950453 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:16.950383    2549 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-142-53.ec2.internal" event="NodeHasSufficientPID"
May 25 12:13:16.963803 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:13:16.963782    2549 kubelet.go:3336] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-53.ec2.internal\" not found" node="ip-10-0-142-53.ec2.internal"
May 25 12:13:16.966997 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:13:16.966979    2549 kubelet.go:3336] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-53.ec2.internal\" not found" node="ip-10-0-142-53.ec2.internal"
May 25 12:13:17.011804 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:13:17.011735    2549 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-142-53.ec2.internal\" not found"
May 25 12:13:17.013932 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.013914    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/dd968f903a046947131232f9282e5635-config\") pod \"kube-apiserver-proxy-ip-10-0-142-53.ec2.internal\" (UID: \"dd968f903a046947131232f9282e5635\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-53.ec2.internal"
May 25 12:13:17.013987 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.013942    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/fa968fd9574051417c15086d0b1771c9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-53.ec2.internal\" (UID: \"fa968fd9574051417c15086d0b1771c9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-53.ec2.internal"
May 25 12:13:17.013987 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.013960    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fa968fd9574051417c15086d0b1771c9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-53.ec2.internal\" (UID: \"fa968fd9574051417c15086d0b1771c9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-53.ec2.internal"
May 25 12:13:17.111854 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:13:17.111827    2549 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-142-53.ec2.internal\" not found"
May 25 12:13:17.115096 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.115083    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/fa968fd9574051417c15086d0b1771c9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-53.ec2.internal\" (UID: \"fa968fd9574051417c15086d0b1771c9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-53.ec2.internal"
May 25 12:13:17.115161 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.115106    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fa968fd9574051417c15086d0b1771c9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-53.ec2.internal\" (UID: \"fa968fd9574051417c15086d0b1771c9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-53.ec2.internal"
May 25 12:13:17.115161 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.115125    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/dd968f903a046947131232f9282e5635-config\") pod \"kube-apiserver-proxy-ip-10-0-142-53.ec2.internal\" (UID: \"dd968f903a046947131232f9282e5635\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-53.ec2.internal"
May 25 12:13:17.115266 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.115167    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/dd968f903a046947131232f9282e5635-config\") pod \"kube-apiserver-proxy-ip-10-0-142-53.ec2.internal\" (UID: \"dd968f903a046947131232f9282e5635\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-53.ec2.internal"
May 25 12:13:17.115266 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.115190    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fa968fd9574051417c15086d0b1771c9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-53.ec2.internal\" (UID: \"fa968fd9574051417c15086d0b1771c9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-53.ec2.internal"
May 25 12:13:17.115266 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.115191    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/fa968fd9574051417c15086d0b1771c9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-53.ec2.internal\" (UID: \"fa968fd9574051417c15086d0b1771c9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-53.ec2.internal"
May 25 12:13:17.212510 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:13:17.212466    2549 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-142-53.ec2.internal\" not found"
May 25 12:13:17.266860 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.266776    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-53.ec2.internal"
May 25 12:13:17.269533 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.269520    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-53.ec2.internal"
May 25 12:13:17.313049 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:13:17.313023    2549 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-142-53.ec2.internal\" not found"
May 25 12:13:17.413430 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:13:17.413410    2549 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-142-53.ec2.internal\" not found"
May 25 12:13:17.513892 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:13:17.513872    2549 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-142-53.ec2.internal\" not found"
May 25 12:13:17.538607 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.538546    2549 reflector.go:430] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
May 25 12:13:17.612955 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.612924    2549 kubelet.go:3340] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-53.ec2.internal"
May 25 12:13:17.626245 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.626217    2549 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
May 25 12:13:17.626592 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.626580    2549 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
May 25 12:13:17.626693 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.626679    2549 reflector.go:556] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
May 25 12:13:17.626726 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.626682    2549 reflector.go:556] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
May 25 12:13:17.628047 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.628034    2549 kubelet.go:3340] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-53.ec2.internal"
May 25 12:13:17.644731 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.644714    2549 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
May 25 12:13:17.691067 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.691039    2549 apiserver.go:52] "Watching apiserver"
May 25 12:13:17.703284 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.703243    2549 reflector.go:430] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
May 25 12:13:17.704090 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.704065    2549 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-tcp4j","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l8w7f","openshift-cluster-node-tuning-operator/tuned-xp8d2","openshift-dns/node-resolver-dv9dt","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-53.ec2.internal","openshift-multus/network-metrics-daemon-zf62d","openshift-ovn-kubernetes/ovnkube-node-79j85","kube-system/kube-apiserver-proxy-ip-10-0-142-53.ec2.internal","openshift-image-registry/node-ca-bjpxs","openshift-multus/multus-29966","openshift-multus/multus-additional-cni-plugins-h49mm","openshift-network-diagnostics/network-check-target-tp68k","openshift-network-operator/iptables-alerter-g4949"]
May 25 12:13:17.709496 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.709475    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-29966"
May 25 12:13:17.710143 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.710127    2549 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
May 25 12:13:17.712329 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.712307    2549 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
May 25 12:13:17.712409 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.712330    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-xp8d2"
May 25 12:13:17.712409 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.712342    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-jfqq7\""
May 25 12:13:17.712409 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.712392    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zf62d"
May 25 12:13:17.712562 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:13:17.712460    2549 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zf62d" podUID="1124f1f1-fdc7-4a81-b28a-620db95e2ff6"
May 25 12:13:17.712562 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.712480    2549 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
May 25 12:13:17.712562 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.712501    2549 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
May 25 12:13:17.712705 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.712580    2549 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
May 25 12:13:17.714871 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.714852    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-gxcf7\""
May 25 12:13:17.714965 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.714878    2549 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
May 25 12:13:17.714965 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.714889    2549 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
May 25 12:13:17.717517 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.717505    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-79j85"
May 25 12:13:17.718489 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.718472    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c66cda37-4f3e-4058-a58c-1b168db3edd7-etc-modprobe-d\") pod \"tuned-xp8d2\" (UID: \"c66cda37-4f3e-4058-a58c-1b168db3edd7\") " pod="openshift-cluster-node-tuning-operator/tuned-xp8d2"
May 25 12:13:17.718566 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.718495    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c66cda37-4f3e-4058-a58c-1b168db3edd7-etc-sysconfig\") pod \"tuned-xp8d2\" (UID: \"c66cda37-4f3e-4058-a58c-1b168db3edd7\") " pod="openshift-cluster-node-tuning-operator/tuned-xp8d2"
May 25 12:13:17.718566 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.718513    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lcss\" (UniqueName: \"kubernetes.io/projected/1124f1f1-fdc7-4a81-b28a-620db95e2ff6-kube-api-access-5lcss\") pod \"network-metrics-daemon-zf62d\" (UID: \"1124f1f1-fdc7-4a81-b28a-620db95e2ff6\") " pod="openshift-multus/network-metrics-daemon-zf62d"
May 25 12:13:17.718566 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.718528    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c66cda37-4f3e-4058-a58c-1b168db3edd7-lib-modules\") pod \"tuned-xp8d2\" (UID: \"c66cda37-4f3e-4058-a58c-1b168db3edd7\") " pod="openshift-cluster-node-tuning-operator/tuned-xp8d2"
May 25 12:13:17.718566 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.718543    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c66cda37-4f3e-4058-a58c-1b168db3edd7-sys\") pod \"tuned-xp8d2\" (UID: \"c66cda37-4f3e-4058-a58c-1b168db3edd7\") " pod="openshift-cluster-node-tuning-operator/tuned-xp8d2"
May 25 12:13:17.718751 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.718581    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c66cda37-4f3e-4058-a58c-1b168db3edd7-host\") pod \"tuned-xp8d2\" (UID: \"c66cda37-4f3e-4058-a58c-1b168db3edd7\") " pod="openshift-cluster-node-tuning-operator/tuned-xp8d2"
May 25 12:13:17.718751 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.718614    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ll5zp\" (UniqueName: \"kubernetes.io/projected/c66cda37-4f3e-4058-a58c-1b168db3edd7-kube-api-access-ll5zp\") pod \"tuned-xp8d2\" (UID: \"c66cda37-4f3e-4058-a58c-1b168db3edd7\") " pod="openshift-cluster-node-tuning-operator/tuned-xp8d2"
May 25 12:13:17.718751 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.718639    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1124f1f1-fdc7-4a81-b28a-620db95e2ff6-metrics-certs\") pod \"network-metrics-daemon-zf62d\" (UID: \"1124f1f1-fdc7-4a81-b28a-620db95e2ff6\") " pod="openshift-multus/network-metrics-daemon-zf62d"
May 25 12:13:17.718751 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.718665    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2d7e9aa3-9ae5-4e93-8e85-11d5efe2dd99-multus-socket-dir-parent\") pod \"multus-29966\" (UID: \"2d7e9aa3-9ae5-4e93-8e85-11d5efe2dd99\") " pod="openshift-multus/multus-29966"
May 25 12:13:17.718751 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.718692    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c66cda37-4f3e-4058-a58c-1b168db3edd7-etc-sysctl-conf\") pod \"tuned-xp8d2\" (UID: \"c66cda37-4f3e-4058-a58c-1b168db3edd7\") " pod="openshift-cluster-node-tuning-operator/tuned-xp8d2"
May 25 12:13:17.718751 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.718726    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c66cda37-4f3e-4058-a58c-1b168db3edd7-tmp\") pod \"tuned-xp8d2\" (UID: \"c66cda37-4f3e-4058-a58c-1b168db3edd7\") " pod="openshift-cluster-node-tuning-operator/tuned-xp8d2"
May 25 12:13:17.718984 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.718751    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2d7e9aa3-9ae5-4e93-8e85-11d5efe2dd99-system-cni-dir\") pod \"multus-29966\" (UID: \"2d7e9aa3-9ae5-4e93-8e85-11d5efe2dd99\") " pod="openshift-multus/multus-29966"
May 25 12:13:17.718984 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.718777    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2d7e9aa3-9ae5-4e93-8e85-11d5efe2dd99-host-var-lib-cni-multus\") pod \"multus-29966\" (UID: \"2d7e9aa3-9ae5-4e93-8e85-11d5efe2dd99\") " pod="openshift-multus/multus-29966"
May 25 12:13:17.718984 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.718793    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2d7e9aa3-9ae5-4e93-8e85-11d5efe2dd99-hostroot\") pod \"multus-29966\" (UID: \"2d7e9aa3-9ae5-4e93-8e85-11d5efe2dd99\") " pod="openshift-multus/multus-29966"
May 25 12:13:17.718984 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.718825    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2d7e9aa3-9ae5-4e93-8e85-11d5efe2dd99-multus-conf-dir\") pod \"multus-29966\" (UID: \"2d7e9aa3-9ae5-4e93-8e85-11d5efe2dd99\") " pod="openshift-multus/multus-29966"
May 25 12:13:17.718984 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.718839    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2d7e9aa3-9ae5-4e93-8e85-11d5efe2dd99-multus-daemon-config\") pod \"multus-29966\" (UID: \"2d7e9aa3-9ae5-4e93-8e85-11d5efe2dd99\") " pod="openshift-multus/multus-29966"
May 25 12:13:17.718984 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.718859    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c66cda37-4f3e-4058-a58c-1b168db3edd7-etc-tuned\") pod \"tuned-xp8d2\" (UID: \"c66cda37-4f3e-4058-a58c-1b168db3edd7\") " pod="openshift-cluster-node-tuning-operator/tuned-xp8d2"
May 25 12:13:17.718984 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.718875    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c66cda37-4f3e-4058-a58c-1b168db3edd7-etc-kubernetes\") pod \"tuned-xp8d2\" (UID: \"c66cda37-4f3e-4058-a58c-1b168db3edd7\") " pod="openshift-cluster-node-tuning-operator/tuned-xp8d2"
May 25 12:13:17.718984 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.718906    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2d7e9aa3-9ae5-4e93-8e85-11d5efe2dd99-cnibin\") pod \"multus-29966\" (UID: \"2d7e9aa3-9ae5-4e93-8e85-11d5efe2dd99\") " pod="openshift-multus/multus-29966"
May 25 12:13:17.718984 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.718923    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2d7e9aa3-9ae5-4e93-8e85-11d5efe2dd99-host-var-lib-cni-bin\") pod \"multus-29966\" (UID: \"2d7e9aa3-9ae5-4e93-8e85-11d5efe2dd99\") " pod="openshift-multus/multus-29966"
May 25 12:13:17.718984 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.718939    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2d7e9aa3-9ae5-4e93-8e85-11d5efe2dd99-host-var-lib-kubelet\") pod \"multus-29966\" (UID: \"2d7e9aa3-9ae5-4e93-8e85-11d5efe2dd99\") " pod="openshift-multus/multus-29966"
May 25 12:13:17.718984 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.718954    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2d7e9aa3-9ae5-4e93-8e85-11d5efe2dd99-host-run-multus-certs\") pod \"multus-29966\" (UID: \"2d7e9aa3-9ae5-4e93-8e85-11d5efe2dd99\") " pod="openshift-multus/multus-29966"
May 25 12:13:17.718984 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.718967    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2d7e9aa3-9ae5-4e93-8e85-11d5efe2dd99-etc-kubernetes\") pod \"multus-29966\" (UID: \"2d7e9aa3-9ae5-4e93-8e85-11d5efe2dd99\") " pod="openshift-multus/multus-29966"
May 25 12:13:17.718984 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.718981    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2d7e9aa3-9ae5-4e93-8e85-11d5efe2dd99-os-release\") pod \"multus-29966\" (UID: \"2d7e9aa3-9ae5-4e93-8e85-11d5efe2dd99\") " pod="openshift-multus/multus-29966"
May 25 12:13:17.719505 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.718996    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c66cda37-4f3e-4058-a58c-1b168db3edd7-run\") pod \"tuned-xp8d2\" (UID: \"c66cda37-4f3e-4058-a58c-1b168db3edd7\") " pod="openshift-cluster-node-tuning-operator/tuned-xp8d2"
May 25 12:13:17.719505 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.719010    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2d7e9aa3-9ae5-4e93-8e85-11d5efe2dd99-cni-binary-copy\") pod \"multus-29966\" (UID: \"2d7e9aa3-9ae5-4e93-8e85-11d5efe2dd99\") " pod="openshift-multus/multus-29966"
May 25 12:13:17.719505 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.719029    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2d7e9aa3-9ae5-4e93-8e85-11d5efe2dd99-host-run-k8s-cni-cncf-io\") pod \"multus-29966\" (UID: \"2d7e9aa3-9ae5-4e93-8e85-11d5efe2dd99\") " pod="openshift-multus/multus-29966"
May 25 12:13:17.719505 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.719045    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c66cda37-4f3e-4058-a58c-1b168db3edd7-etc-systemd\") pod \"tuned-xp8d2\" (UID: \"c66cda37-4f3e-4058-a58c-1b168db3edd7\") " pod="openshift-cluster-node-tuning-operator/tuned-xp8d2"
May 25 12:13:17.719505 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.719065    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nrwl\" (UniqueName: \"kubernetes.io/projected/2d7e9aa3-9ae5-4e93-8e85-11d5efe2dd99-kube-api-access-9nrwl\") pod \"multus-29966\" (UID: \"2d7e9aa3-9ae5-4e93-8e85-11d5efe2dd99\") " pod="openshift-multus/multus-29966"
May 25 12:13:17.719505 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.719085    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c66cda37-4f3e-4058-a58c-1b168db3edd7-etc-sysctl-d\") pod \"tuned-xp8d2\" (UID: \"c66cda37-4f3e-4058-a58c-1b168db3edd7\") " pod="openshift-cluster-node-tuning-operator/tuned-xp8d2"
May 25 12:13:17.719505 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.719105    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2d7e9aa3-9ae5-4e93-8e85-11d5efe2dd99-multus-cni-dir\") pod \"multus-29966\" (UID: \"2d7e9aa3-9ae5-4e93-8e85-11d5efe2dd99\") " pod="openshift-multus/multus-29966"
May 25 12:13:17.719505 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.719118    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2d7e9aa3-9ae5-4e93-8e85-11d5efe2dd99-host-run-netns\") pod \"multus-29966\" (UID: \"2d7e9aa3-9ae5-4e93-8e85-11d5efe2dd99\") " pod="openshift-multus/multus-29966"
May 25 12:13:17.719505 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.719131    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c66cda37-4f3e-4058-a58c-1b168db3edd7-var-lib-kubelet\") pod \"tuned-xp8d2\" (UID: \"c66cda37-4f3e-4058-a58c-1b168db3edd7\") " pod="openshift-cluster-node-tuning-operator/tuned-xp8d2"
May 25 12:13:17.720003 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.719975    2549 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
May 25 12:13:17.720372 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.720359    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-tcp4j"
May 25 12:13:17.720477 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.720462    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-h49mm"
May 25 12:13:17.721207 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.721156    2549 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
May 25 12:13:17.721207 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.721162    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
May 25 12:13:17.721207 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.721187    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-j48lg\""
May 25 12:13:17.721398 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.721219    2549 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
May 25 12:13:17.721398 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.721286    2549 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
May 25 12:13:17.721939 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.721925    2549 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
May 25 12:13:17.722989 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.722973    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tp68k"
May 25 12:13:17.723065 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.722996    2549 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
May 25 12:13:17.723065 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.723009    2549 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
May 25 12:13:17.723065 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:13:17.723028    2549 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tp68k" podUID="d37b4fc9-9e8b-4e07-9672-b6bffc0f46d8"
May 25 12:13:17.723424 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.723409    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-hqs9h\""
May 25 12:13:17.723667 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.723650    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-l9nx4\""
May 25 12:13:17.723981 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.723965    2549 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
May 25 12:13:17.723981 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.723971    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
May 25 12:13:17.724148 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.724016    2549 reflector.go:430] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
May 25 12:13:17.725601 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.725585    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-g4949"
May 25 12:13:17.727918 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.727901    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-rgsg9\""
May 25 12:13:17.728041 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.727916    2549 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
May 25 12:13:17.728041 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.727920    2549 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
May 25 12:13:17.728041 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.727966    2549 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
May 25 12:13:17.728280 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.728268    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l8w7f"
May 25 12:13:17.730993 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.730976    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-dv9dt"
May 25 12:13:17.731117 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.731017    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
May 25 12:13:17.731117 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.731091    2549 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
May 25 12:13:17.731338 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.731321    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-mllxb\""
May 25 12:13:17.732313 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.731844    2549 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
May 25 12:13:17.732965 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.732747    2549 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-05-24 12:08:16 +0000 UTC" deadline="2028-02-18 21:05:21.62268132 +0000 UTC"
May 25 12:13:17.732965 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.732796    2549 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15224h52m3.889888987s"
May 25 12:13:17.733307 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.733291    2549 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
May 25 12:13:17.733526 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.733510    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-2t6cp\""
May 25 12:13:17.734052 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.733768    2549 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
May 25 12:13:17.734672 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.734656    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-bjpxs"
May 25 12:13:17.737046 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.737028    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-l54fv\""
May 25 12:13:17.737153 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.737057    2549 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
May 25 12:13:17.737153 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.737073    2549 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
May 25 12:13:17.737335 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.737323    2549 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
May 25 12:13:17.742773 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.742752    2549 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-2qpq7"
May 25 12:13:17.751142 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.751121    2549 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-2qpq7"
May 25 12:13:17.812852 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.812834    2549 desired_state_of_world_populator.go:158] "Finished populating initial desired state of world"
May 25 12:13:17.819755 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.819729    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2d7e9aa3-9ae5-4e93-8e85-11d5efe2dd99-multus-cni-dir\") pod \"multus-29966\" (UID: \"2d7e9aa3-9ae5-4e93-8e85-11d5efe2dd99\") " pod="openshift-multus/multus-29966"
May 25 12:13:17.819848 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.819770    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2d7e9aa3-9ae5-4e93-8e85-11d5efe2dd99-host-run-netns\") pod \"multus-29966\" (UID: \"2d7e9aa3-9ae5-4e93-8e85-11d5efe2dd99\") " pod="openshift-multus/multus-29966"
May 25 12:13:17.819848 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.819801    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0b693280-ce91-46e7-b8c8-37d81a9b8fe1-host-run-ovn-kubernetes\") pod \"ovnkube-node-79j85\" (UID: \"0b693280-ce91-46e7-b8c8-37d81a9b8fe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-79j85"
May 25 12:13:17.819848 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.819829    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7153f197-fa4f-4da4-ba82-73f29e4dd2ac-system-cni-dir\") pod \"multus-additional-cni-plugins-h49mm\" (UID: \"7153f197-fa4f-4da4-ba82-73f29e4dd2ac\") " pod="openshift-multus/multus-additional-cni-plugins-h49mm"
May 25 12:13:17.819848 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.819840    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2d7e9aa3-9ae5-4e93-8e85-11d5efe2dd99-multus-cni-dir\") pod \"multus-29966\" (UID: \"2d7e9aa3-9ae5-4e93-8e85-11d5efe2dd99\") " pod="openshift-multus/multus-29966"
May 25 12:13:17.820060 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.819855    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7153f197-fa4f-4da4-ba82-73f29e4dd2ac-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-h49mm\" (UID: \"7153f197-fa4f-4da4-ba82-73f29e4dd2ac\") " pod="openshift-multus/multus-additional-cni-plugins-h49mm"
May 25 12:13:17.820060 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.819880    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c66cda37-4f3e-4058-a58c-1b168db3edd7-etc-modprobe-d\") pod \"tuned-xp8d2\" (UID: \"c66cda37-4f3e-4058-a58c-1b168db3edd7\") " pod="openshift-cluster-node-tuning-operator/tuned-xp8d2"
May 25 12:13:17.820060 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.819904    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c66cda37-4f3e-4058-a58c-1b168db3edd7-etc-sysconfig\") pod \"tuned-xp8d2\" (UID: \"c66cda37-4f3e-4058-a58c-1b168db3edd7\") " pod="openshift-cluster-node-tuning-operator/tuned-xp8d2"
May 25 12:13:17.820060 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.819920    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-5lcss\" (UniqueName: \"kubernetes.io/projected/1124f1f1-fdc7-4a81-b28a-620db95e2ff6-kube-api-access-5lcss\") pod \"network-metrics-daemon-zf62d\" (UID: \"1124f1f1-fdc7-4a81-b28a-620db95e2ff6\") " pod="openshift-multus/network-metrics-daemon-zf62d"
May 25 12:13:17.820060 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.819938    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0b693280-ce91-46e7-b8c8-37d81a9b8fe1-ovnkube-config\") pod \"ovnkube-node-79j85\" (UID: \"0b693280-ce91-46e7-b8c8-37d81a9b8fe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-79j85"
May 25 12:13:17.820060 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.819961    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0b693280-ce91-46e7-b8c8-37d81a9b8fe1-env-overrides\") pod \"ovnkube-node-79j85\" (UID: \"0b693280-ce91-46e7-b8c8-37d81a9b8fe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-79j85"
May 25 12:13:17.820060 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.819936    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2d7e9aa3-9ae5-4e93-8e85-11d5efe2dd99-host-run-netns\") pod \"multus-29966\" (UID: \"2d7e9aa3-9ae5-4e93-8e85-11d5efe2dd99\") " pod="openshift-multus/multus-29966"
May 25 12:13:17.820060 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.820008    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c66cda37-4f3e-4058-a58c-1b168db3edd7-etc-sysconfig\") pod \"tuned-xp8d2\" (UID: \"c66cda37-4f3e-4058-a58c-1b168db3edd7\") " pod="openshift-cluster-node-tuning-operator/tuned-xp8d2"
May 25 12:13:17.820060 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.820022    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c66cda37-4f3e-4058-a58c-1b168db3edd7-etc-modprobe-d\") pod \"tuned-xp8d2\" (UID: \"c66cda37-4f3e-4058-a58c-1b168db3edd7\") " pod="openshift-cluster-node-tuning-operator/tuned-xp8d2"
May 25 12:13:17.820060 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.820010    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7bb34bd7-c02c-42cb-b3c8-643fd893a7db-registration-dir\") pod \"aws-ebs-csi-driver-node-l8w7f\" (UID: \"7bb34bd7-c02c-42cb-b3c8-643fd893a7db\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l8w7f"
May 25 12:13:17.820060 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.820059    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7bb34bd7-c02c-42cb-b3c8-643fd893a7db-device-dir\") pod \"aws-ebs-csi-driver-node-l8w7f\" (UID: \"7bb34bd7-c02c-42cb-b3c8-643fd893a7db\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l8w7f"
May 25 12:13:17.820486 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.820088    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c66cda37-4f3e-4058-a58c-1b168db3edd7-host\") pod \"tuned-xp8d2\" (UID: \"c66cda37-4f3e-4058-a58c-1b168db3edd7\") " pod="openshift-cluster-node-tuning-operator/tuned-xp8d2"
May 25 12:13:17.820486 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.820115    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-ll5zp\" (UniqueName: \"kubernetes.io/projected/c66cda37-4f3e-4058-a58c-1b168db3edd7-kube-api-access-ll5zp\") pod \"tuned-xp8d2\" (UID: \"c66cda37-4f3e-4058-a58c-1b168db3edd7\") " pod="openshift-cluster-node-tuning-operator/tuned-xp8d2"
May 25 12:13:17.820486 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.820141    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnhxh\" (UniqueName: \"kubernetes.io/projected/61e828df-19f1-4008-bcb9-2c3a879b0178-kube-api-access-qnhxh\") pod \"node-resolver-dv9dt\" (UID: \"61e828df-19f1-4008-bcb9-2c3a879b0178\") " pod="openshift-dns/node-resolver-dv9dt"
May 25 12:13:17.820486 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.820166    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3472ac4b-9b58-4e85-85e8-f006315d8915-agent-certs\") pod \"konnectivity-agent-tcp4j\" (UID: \"3472ac4b-9b58-4e85-85e8-f006315d8915\") " pod="kube-system/konnectivity-agent-tcp4j"
May 25 12:13:17.820486 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.820174    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c66cda37-4f3e-4058-a58c-1b168db3edd7-host\") pod \"tuned-xp8d2\" (UID: \"c66cda37-4f3e-4058-a58c-1b168db3edd7\") " pod="openshift-cluster-node-tuning-operator/tuned-xp8d2"
May 25 12:13:17.820486 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.820191    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dgg9\" (UniqueName: \"kubernetes.io/projected/69c409ea-f028-429e-8a53-0f25010483bd-kube-api-access-6dgg9\") pod \"iptables-alerter-g4949\" (UID: \"69c409ea-f028-429e-8a53-0f25010483bd\") " pod="openshift-network-operator/iptables-alerter-g4949"
May 25 12:13:17.820486 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.820230    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0b693280-ce91-46e7-b8c8-37d81a9b8fe1-var-lib-openvswitch\") pod \"ovnkube-node-79j85\" (UID: \"0b693280-ce91-46e7-b8c8-37d81a9b8fe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-79j85"
May 25 12:13:17.820486 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.820255    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0b693280-ce91-46e7-b8c8-37d81a9b8fe1-run-openvswitch\") pod \"ovnkube-node-79j85\" (UID: \"0b693280-ce91-46e7-b8c8-37d81a9b8fe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-79j85"
May 25 12:13:17.820486 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.820280    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3472ac4b-9b58-4e85-85e8-f006315d8915-konnectivity-ca\") pod \"konnectivity-agent-tcp4j\" (UID: \"3472ac4b-9b58-4e85-85e8-f006315d8915\") " pod="kube-system/konnectivity-agent-tcp4j"
May 25 12:13:17.820486 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.820326    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2d7e9aa3-9ae5-4e93-8e85-11d5efe2dd99-host-var-lib-cni-multus\") pod \"multus-29966\" (UID: \"2d7e9aa3-9ae5-4e93-8e85-11d5efe2dd99\") " pod="openshift-multus/multus-29966"
May 25 12:13:17.820486 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.820345    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2d7e9aa3-9ae5-4e93-8e85-11d5efe2dd99-hostroot\") pod \"multus-29966\" (UID: \"2d7e9aa3-9ae5-4e93-8e85-11d5efe2dd99\") " pod="openshift-multus/multus-29966"
May 25 12:13:17.820486 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.820371    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2d7e9aa3-9ae5-4e93-8e85-11d5efe2dd99-host-var-lib-cni-multus\") pod \"multus-29966\" (UID: \"2d7e9aa3-9ae5-4e93-8e85-11d5efe2dd99\") " pod="openshift-multus/multus-29966"
May 25 12:13:17.820486 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.820376    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0b693280-ce91-46e7-b8c8-37d81a9b8fe1-host-cni-bin\") pod \"ovnkube-node-79j85\" (UID: \"0b693280-ce91-46e7-b8c8-37d81a9b8fe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-79j85"
May 25 12:13:17.820486 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.820411    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2d7e9aa3-9ae5-4e93-8e85-11d5efe2dd99-hostroot\") pod \"multus-29966\" (UID: \"2d7e9aa3-9ae5-4e93-8e85-11d5efe2dd99\") " pod="openshift-multus/multus-29966"
May 25 12:13:17.820486 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.820419    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/69c409ea-f028-429e-8a53-0f25010483bd-iptables-alerter-script\") pod \"iptables-alerter-g4949\" (UID: \"69c409ea-f028-429e-8a53-0f25010483bd\") " pod="openshift-network-operator/iptables-alerter-g4949"
May 25 12:13:17.820486 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.820444    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7bb34bd7-c02c-42cb-b3c8-643fd893a7db-kubelet-dir\") pod \"aws-ebs-csi-driver-node-l8w7f\" (UID: \"7bb34bd7-c02c-42cb-b3c8-643fd893a7db\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l8w7f"
May 25 12:13:17.820486 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.820465    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7bb34bd7-c02c-42cb-b3c8-643fd893a7db-etc-selinux\") pod \"aws-ebs-csi-driver-node-l8w7f\" (UID: \"7bb34bd7-c02c-42cb-b3c8-643fd893a7db\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l8w7f"
May 25 12:13:17.821134 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.820480    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzjf5\" (UniqueName: \"kubernetes.io/projected/c13dfb5f-85fd-4f8e-9a7f-f432a01720d3-kube-api-access-dzjf5\") pod \"node-ca-bjpxs\" (UID: \"c13dfb5f-85fd-4f8e-9a7f-f432a01720d3\") " pod="openshift-image-registry/node-ca-bjpxs"
May 25 12:13:17.821134 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.820497    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2d7e9aa3-9ae5-4e93-8e85-11d5efe2dd99-host-var-lib-cni-bin\") pod \"multus-29966\" (UID: \"2d7e9aa3-9ae5-4e93-8e85-11d5efe2dd99\") " pod="openshift-multus/multus-29966"
May 25 12:13:17.821134 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.820525    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2d7e9aa3-9ae5-4e93-8e85-11d5efe2dd99-host-var-lib-kubelet\") pod \"multus-29966\" (UID: \"2d7e9aa3-9ae5-4e93-8e85-11d5efe2dd99\") " pod="openshift-multus/multus-29966"
May 25 12:13:17.821134 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.820534    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2d7e9aa3-9ae5-4e93-8e85-11d5efe2dd99-host-var-lib-cni-bin\") pod \"multus-29966\" (UID: \"2d7e9aa3-9ae5-4e93-8e85-11d5efe2dd99\") " pod="openshift-multus/multus-29966"
May 25 12:13:17.821134 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.820553    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2d7e9aa3-9ae5-4e93-8e85-11d5efe2dd99-host-run-multus-certs\") pod \"multus-29966\" (UID: \"2d7e9aa3-9ae5-4e93-8e85-11d5efe2dd99\") " pod="openshift-multus/multus-29966"
May 25 12:13:17.821134 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.820561    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2d7e9aa3-9ae5-4e93-8e85-11d5efe2dd99-host-var-lib-kubelet\") pod \"multus-29966\" (UID: \"2d7e9aa3-9ae5-4e93-8e85-11d5efe2dd99\") " pod="openshift-multus/multus-29966"
May 25 12:13:17.821134 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.820583    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2d7e9aa3-9ae5-4e93-8e85-11d5efe2dd99-etc-kubernetes\") pod \"multus-29966\" (UID: \"2d7e9aa3-9ae5-4e93-8e85-11d5efe2dd99\") " pod="openshift-multus/multus-29966"
May 25 12:13:17.821134 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.820608    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0b693280-ce91-46e7-b8c8-37d81a9b8fe1-run-ovn\") pod \"ovnkube-node-79j85\" (UID: \"0b693280-ce91-46e7-b8c8-37d81a9b8fe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-79j85"
May 25 12:13:17.821134 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.820627    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2d7e9aa3-9ae5-4e93-8e85-11d5efe2dd99-etc-kubernetes\") pod \"multus-29966\" (UID: \"2d7e9aa3-9ae5-4e93-8e85-11d5efe2dd99\") " pod="openshift-multus/multus-29966"
May 25 12:13:17.821134 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.820606    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2d7e9aa3-9ae5-4e93-8e85-11d5efe2dd99-host-run-multus-certs\") pod \"multus-29966\" (UID: \"2d7e9aa3-9ae5-4e93-8e85-11d5efe2dd99\") " pod="openshift-multus/multus-29966"
May 25 12:13:17.821134 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.820633    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7brb\" (UniqueName: \"kubernetes.io/projected/0b693280-ce91-46e7-b8c8-37d81a9b8fe1-kube-api-access-g7brb\") pod \"ovnkube-node-79j85\" (UID: \"0b693280-ce91-46e7-b8c8-37d81a9b8fe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-79j85"
May 25 12:13:17.821134 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.820695    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7bb34bd7-c02c-42cb-b3c8-643fd893a7db-socket-dir\") pod \"aws-ebs-csi-driver-node-l8w7f\" (UID: \"7bb34bd7-c02c-42cb-b3c8-643fd893a7db\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l8w7f"
May 25 12:13:17.821134 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.820721    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7153f197-fa4f-4da4-ba82-73f29e4dd2ac-cnibin\") pod \"multus-additional-cni-plugins-h49mm\" (UID: \"7153f197-fa4f-4da4-ba82-73f29e4dd2ac\") " pod="openshift-multus/multus-additional-cni-plugins-h49mm"
May 25 12:13:17.821134 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.820745    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/61e828df-19f1-4008-bcb9-2c3a879b0178-hosts-file\") pod \"node-resolver-dv9dt\" (UID: \"61e828df-19f1-4008-bcb9-2c3a879b0178\") " pod="openshift-dns/node-resolver-dv9dt"
May 25 12:13:17.821134 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.820780    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56c2b\" (UniqueName: \"kubernetes.io/projected/d37b4fc9-9e8b-4e07-9672-b6bffc0f46d8-kube-api-access-56c2b\") pod \"network-check-target-tp68k\" (UID: \"d37b4fc9-9e8b-4e07-9672-b6bffc0f46d8\") " pod="openshift-network-diagnostics/network-check-target-tp68k"
May 25 12:13:17.821134 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.820816    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c13dfb5f-85fd-4f8e-9a7f-f432a01720d3-host\") pod \"node-ca-bjpxs\" (UID: \"c13dfb5f-85fd-4f8e-9a7f-f432a01720d3\") " pod="openshift-image-registry/node-ca-bjpxs"
May 25 12:13:17.821134 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.820844    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjnml\" (UniqueName: \"kubernetes.io/projected/7153f197-fa4f-4da4-ba82-73f29e4dd2ac-kube-api-access-zjnml\") pod \"multus-additional-cni-plugins-h49mm\" (UID: \"7153f197-fa4f-4da4-ba82-73f29e4dd2ac\") " pod="openshift-multus/multus-additional-cni-plugins-h49mm"
May 25 12:13:17.821799 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.820863    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2d7e9aa3-9ae5-4e93-8e85-11d5efe2dd99-host-run-k8s-cni-cncf-io\") pod \"multus-29966\" (UID: \"2d7e9aa3-9ae5-4e93-8e85-11d5efe2dd99\") " pod="openshift-multus/multus-29966"
May 25 12:13:17.821799 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.820877    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c13dfb5f-85fd-4f8e-9a7f-f432a01720d3-serviceca\") pod \"node-ca-bjpxs\" (UID: \"c13dfb5f-85fd-4f8e-9a7f-f432a01720d3\") " pod="openshift-image-registry/node-ca-bjpxs"
May 25 12:13:17.821799 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.820892    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-9nrwl\" (UniqueName: \"kubernetes.io/projected/2d7e9aa3-9ae5-4e93-8e85-11d5efe2dd99-kube-api-access-9nrwl\") pod \"multus-29966\" (UID: \"2d7e9aa3-9ae5-4e93-8e85-11d5efe2dd99\") " pod="openshift-multus/multus-29966"
May 25 12:13:17.821799 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.820916    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c66cda37-4f3e-4058-a58c-1b168db3edd7-etc-sysctl-d\") pod \"tuned-xp8d2\" (UID: \"c66cda37-4f3e-4058-a58c-1b168db3edd7\") " pod="openshift-cluster-node-tuning-operator/tuned-xp8d2"
May 25 12:13:17.821799 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.820925    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2d7e9aa3-9ae5-4e93-8e85-11d5efe2dd99-host-run-k8s-cni-cncf-io\") pod \"multus-29966\" (UID: \"2d7e9aa3-9ae5-4e93-8e85-11d5efe2dd99\") " pod="openshift-multus/multus-29966"
May 25 12:13:17.821799 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.820952    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c66cda37-4f3e-4058-a58c-1b168db3edd7-var-lib-kubelet\") pod \"tuned-xp8d2\" (UID: \"c66cda37-4f3e-4058-a58c-1b168db3edd7\") " pod="openshift-cluster-node-tuning-operator/tuned-xp8d2"
May 25 12:13:17.821799 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.820981    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0b693280-ce91-46e7-b8c8-37d81a9b8fe1-run-systemd\") pod \"ovnkube-node-79j85\" (UID: \"0b693280-ce91-46e7-b8c8-37d81a9b8fe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-79j85"
May 25 12:13:17.821799 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.821004    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0b693280-ce91-46e7-b8c8-37d81a9b8fe1-node-log\") pod \"ovnkube-node-79j85\" (UID: \"0b693280-ce91-46e7-b8c8-37d81a9b8fe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-79j85"
May 25 12:13:17.821799 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.820998    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c66cda37-4f3e-4058-a58c-1b168db3edd7-etc-sysctl-d\") pod \"tuned-xp8d2\" (UID: \"c66cda37-4f3e-4058-a58c-1b168db3edd7\") " pod="openshift-cluster-node-tuning-operator/tuned-xp8d2"
May 25 12:13:17.821799 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.821028    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c66cda37-4f3e-4058-a58c-1b168db3edd7-var-lib-kubelet\") pod \"tuned-xp8d2\" (UID: \"c66cda37-4f3e-4058-a58c-1b168db3edd7\") " pod="openshift-cluster-node-tuning-operator/tuned-xp8d2"
May 25 12:13:17.821799 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.821084    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c66cda37-4f3e-4058-a58c-1b168db3edd7-lib-modules\") pod \"tuned-xp8d2\" (UID: \"c66cda37-4f3e-4058-a58c-1b168db3edd7\") " pod="openshift-cluster-node-tuning-operator/tuned-xp8d2"
May 25 12:13:17.821799 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.821124    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/69c409ea-f028-429e-8a53-0f25010483bd-host-slash\") pod \"iptables-alerter-g4949\" (UID: \"69c409ea-f028-429e-8a53-0f25010483bd\") " pod="openshift-network-operator/iptables-alerter-g4949"
May 25 12:13:17.821799 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.821150    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7bb34bd7-c02c-42cb-b3c8-643fd893a7db-sys-fs\") pod \"aws-ebs-csi-driver-node-l8w7f\" (UID: \"7bb34bd7-c02c-42cb-b3c8-643fd893a7db\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l8w7f"
May 25 12:13:17.821799 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.821166    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c66cda37-4f3e-4058-a58c-1b168db3edd7-lib-modules\") pod \"tuned-xp8d2\" (UID: \"c66cda37-4f3e-4058-a58c-1b168db3edd7\") " pod="openshift-cluster-node-tuning-operator/tuned-xp8d2"
May 25 12:13:17.821799 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.821174    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7153f197-fa4f-4da4-ba82-73f29e4dd2ac-cni-binary-copy\") pod \"multus-additional-cni-plugins-h49mm\" (UID: \"7153f197-fa4f-4da4-ba82-73f29e4dd2ac\") " pod="openshift-multus/multus-additional-cni-plugins-h49mm"
May 25 12:13:17.821799 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.821216    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7153f197-fa4f-4da4-ba82-73f29e4dd2ac-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-h49mm\" (UID: \"7153f197-fa4f-4da4-ba82-73f29e4dd2ac\") " pod="openshift-multus/multus-additional-cni-plugins-h49mm"
May 25 12:13:17.821799 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.821244    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c66cda37-4f3e-4058-a58c-1b168db3edd7-sys\") pod \"tuned-xp8d2\" (UID: \"c66cda37-4f3e-4058-a58c-1b168db3edd7\") " pod="openshift-cluster-node-tuning-operator/tuned-xp8d2"
May 25 12:13:17.822526 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.821267    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0b693280-ce91-46e7-b8c8-37d81a9b8fe1-host-run-netns\") pod \"ovnkube-node-79j85\" (UID: \"0b693280-ce91-46e7-b8c8-37d81a9b8fe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-79j85"
May 25 12:13:17.822526 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.821288    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0b693280-ce91-46e7-b8c8-37d81a9b8fe1-host-cni-netd\") pod \"ovnkube-node-79j85\" (UID: \"0b693280-ce91-46e7-b8c8-37d81a9b8fe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-79j85"
May 25 12:13:17.822526 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.821312    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0b693280-ce91-46e7-b8c8-37d81a9b8fe1-ovn-node-metrics-cert\") pod \"ovnkube-node-79j85\" (UID: \"0b693280-ce91-46e7-b8c8-37d81a9b8fe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-79j85"
May 25 12:13:17.822526 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.821327    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c66cda37-4f3e-4058-a58c-1b168db3edd7-sys\") pod \"tuned-xp8d2\" (UID: \"c66cda37-4f3e-4058-a58c-1b168db3edd7\") " pod="openshift-cluster-node-tuning-operator/tuned-xp8d2"
May 25 12:13:17.822526 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.821339    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0b693280-ce91-46e7-b8c8-37d81a9b8fe1-ovnkube-script-lib\") pod \"ovnkube-node-79j85\" (UID: \"0b693280-ce91-46e7-b8c8-37d81a9b8fe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-79j85"
May 25 12:13:17.822526 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.821378    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46f52\" (UniqueName: \"kubernetes.io/projected/7bb34bd7-c02c-42cb-b3c8-643fd893a7db-kube-api-access-46f52\") pod \"aws-ebs-csi-driver-node-l8w7f\" (UID: \"7bb34bd7-c02c-42cb-b3c8-643fd893a7db\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l8w7f"
May 25 12:13:17.822526 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.821420    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1124f1f1-fdc7-4a81-b28a-620db95e2ff6-metrics-certs\") pod \"network-metrics-daemon-zf62d\" (UID: \"1124f1f1-fdc7-4a81-b28a-620db95e2ff6\") " pod="openshift-multus/network-metrics-daemon-zf62d"
May 25 12:13:17.822526 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:13:17.821499    2549 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 12:13:17.822526 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.821546    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2d7e9aa3-9ae5-4e93-8e85-11d5efe2dd99-multus-socket-dir-parent\") pod \"multus-29966\" (UID: \"2d7e9aa3-9ae5-4e93-8e85-11d5efe2dd99\") " pod="openshift-multus/multus-29966"
May 25 12:13:17.822526 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.821580    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2d7e9aa3-9ae5-4e93-8e85-11d5efe2dd99-multus-socket-dir-parent\") pod \"multus-29966\" (UID: \"2d7e9aa3-9ae5-4e93-8e85-11d5efe2dd99\") " pod="openshift-multus/multus-29966"
May 25 12:13:17.822526 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:13:17.821582    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1124f1f1-fdc7-4a81-b28a-620db95e2ff6-metrics-certs podName:1124f1f1-fdc7-4a81-b28a-620db95e2ff6 nodeName:}" failed. No retries permitted until 2026-05-25 12:13:18.32155224 +0000 UTC m=+2.037367507 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1124f1f1-fdc7-4a81-b28a-620db95e2ff6-metrics-certs") pod "network-metrics-daemon-zf62d" (UID: "1124f1f1-fdc7-4a81-b28a-620db95e2ff6") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 12:13:17.822526 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.821612    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c66cda37-4f3e-4058-a58c-1b168db3edd7-etc-sysctl-conf\") pod \"tuned-xp8d2\" (UID: \"c66cda37-4f3e-4058-a58c-1b168db3edd7\") " pod="openshift-cluster-node-tuning-operator/tuned-xp8d2"
May 25 12:13:17.822526 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.821628    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c66cda37-4f3e-4058-a58c-1b168db3edd7-tmp\") pod \"tuned-xp8d2\" (UID: \"c66cda37-4f3e-4058-a58c-1b168db3edd7\") " pod="openshift-cluster-node-tuning-operator/tuned-xp8d2"
May 25 12:13:17.822526 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.821706    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0b693280-ce91-46e7-b8c8-37d81a9b8fe1-etc-openvswitch\") pod \"ovnkube-node-79j85\" (UID: \"0b693280-ce91-46e7-b8c8-37d81a9b8fe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-79j85"
May 25 12:13:17.822526 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.821727    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c66cda37-4f3e-4058-a58c-1b168db3edd7-etc-sysctl-conf\") pod \"tuned-xp8d2\" (UID: \"c66cda37-4f3e-4058-a58c-1b168db3edd7\") " pod="openshift-cluster-node-tuning-operator/tuned-xp8d2"
May 25 12:13:17.822526 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.821733    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/7153f197-fa4f-4da4-ba82-73f29e4dd2ac-os-release\") pod \"multus-additional-cni-plugins-h49mm\" (UID: \"7153f197-fa4f-4da4-ba82-73f29e4dd2ac\") " pod="openshift-multus/multus-additional-cni-plugins-h49mm"
May 25 12:13:17.822526 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.821775    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2d7e9aa3-9ae5-4e93-8e85-11d5efe2dd99-system-cni-dir\") pod \"multus-29966\" (UID: \"2d7e9aa3-9ae5-4e93-8e85-11d5efe2dd99\") " pod="openshift-multus/multus-29966"
May 25 12:13:17.823268 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.821818    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2d7e9aa3-9ae5-4e93-8e85-11d5efe2dd99-multus-conf-dir\") pod \"multus-29966\" (UID: \"2d7e9aa3-9ae5-4e93-8e85-11d5efe2dd99\") " pod="openshift-multus/multus-29966"
May 25 12:13:17.823268 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.821851    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2d7e9aa3-9ae5-4e93-8e85-11d5efe2dd99-system-cni-dir\") pod \"multus-29966\" (UID: \"2d7e9aa3-9ae5-4e93-8e85-11d5efe2dd99\") " pod="openshift-multus/multus-29966"
May 25 12:13:17.823268 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.821852    2549 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
May 25 12:13:17.823268 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.821882    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2d7e9aa3-9ae5-4e93-8e85-11d5efe2dd99-multus-daemon-config\") pod \"multus-29966\" (UID: \"2d7e9aa3-9ae5-4e93-8e85-11d5efe2dd99\") " pod="openshift-multus/multus-29966"
May 25 12:13:17.823268 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.821902    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c66cda37-4f3e-4058-a58c-1b168db3edd7-etc-tuned\") pod \"tuned-xp8d2\" (UID: \"c66cda37-4f3e-4058-a58c-1b168db3edd7\") " pod="openshift-cluster-node-tuning-operator/tuned-xp8d2"
May 25 12:13:17.823268 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.821931    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0b693280-ce91-46e7-b8c8-37d81a9b8fe1-systemd-units\") pod \"ovnkube-node-79j85\" (UID: \"0b693280-ce91-46e7-b8c8-37d81a9b8fe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-79j85"
May 25 12:13:17.823268 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.821974    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2d7e9aa3-9ae5-4e93-8e85-11d5efe2dd99-multus-conf-dir\") pod \"multus-29966\" (UID: \"2d7e9aa3-9ae5-4e93-8e85-11d5efe2dd99\") " pod="openshift-multus/multus-29966"
May 25 12:13:17.823268 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.821979    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0b693280-ce91-46e7-b8c8-37d81a9b8fe1-log-socket\") pod \"ovnkube-node-79j85\" (UID: \"0b693280-ce91-46e7-b8c8-37d81a9b8fe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-79j85"
May 25 12:13:17.823268 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.822066    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0b693280-ce91-46e7-b8c8-37d81a9b8fe1-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-79j85\" (UID: \"0b693280-ce91-46e7-b8c8-37d81a9b8fe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-79j85"
May 25 12:13:17.823268 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.822112    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c66cda37-4f3e-4058-a58c-1b168db3edd7-etc-kubernetes\") pod \"tuned-xp8d2\" (UID: \"c66cda37-4f3e-4058-a58c-1b168db3edd7\") " pod="openshift-cluster-node-tuning-operator/tuned-xp8d2"
May 25 12:13:17.823268 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.822167    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2d7e9aa3-9ae5-4e93-8e85-11d5efe2dd99-cnibin\") pod \"multus-29966\" (UID: \"2d7e9aa3-9ae5-4e93-8e85-11d5efe2dd99\") " pod="openshift-multus/multus-29966"
May 25 12:13:17.823268 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.822257    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c66cda37-4f3e-4058-a58c-1b168db3edd7-etc-kubernetes\") pod \"tuned-xp8d2\" (UID: \"c66cda37-4f3e-4058-a58c-1b168db3edd7\") " pod="openshift-cluster-node-tuning-operator/tuned-xp8d2"
May 25 12:13:17.823268 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.822304    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61e828df-19f1-4008-bcb9-2c3a879b0178-tmp-dir\") pod \"node-resolver-dv9dt\" (UID: \"61e828df-19f1-4008-bcb9-2c3a879b0178\") " pod="openshift-dns/node-resolver-dv9dt"
May 25 12:13:17.823268 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.822313    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2d7e9aa3-9ae5-4e93-8e85-11d5efe2dd99-cnibin\") pod \"multus-29966\" (UID: \"2d7e9aa3-9ae5-4e93-8e85-11d5efe2dd99\") " pod="openshift-multus/multus-29966"
May 25 12:13:17.823268 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.822350    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0b693280-ce91-46e7-b8c8-37d81a9b8fe1-host-kubelet\") pod \"ovnkube-node-79j85\" (UID: \"0b693280-ce91-46e7-b8c8-37d81a9b8fe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-79j85"
May 25 12:13:17.823268 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.822378    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2d7e9aa3-9ae5-4e93-8e85-11d5efe2dd99-os-release\") pod \"multus-29966\" (UID: \"2d7e9aa3-9ae5-4e93-8e85-11d5efe2dd99\") " pod="openshift-multus/multus-29966"
May 25 12:13:17.823268 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.822409    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2d7e9aa3-9ae5-4e93-8e85-11d5efe2dd99-multus-daemon-config\") pod \"multus-29966\" (UID: \"2d7e9aa3-9ae5-4e93-8e85-11d5efe2dd99\") " pod="openshift-multus/multus-29966"
May 25 12:13:17.823268 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.822417    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c66cda37-4f3e-4058-a58c-1b168db3edd7-run\") pod \"tuned-xp8d2\" (UID: \"c66cda37-4f3e-4058-a58c-1b168db3edd7\") " pod="openshift-cluster-node-tuning-operator/tuned-xp8d2"
May 25 12:13:17.824072 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.822444    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0b693280-ce91-46e7-b8c8-37d81a9b8fe1-host-slash\") pod \"ovnkube-node-79j85\" (UID: \"0b693280-ce91-46e7-b8c8-37d81a9b8fe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-79j85"
May 25 12:13:17.824072 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.822469    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7153f197-fa4f-4da4-ba82-73f29e4dd2ac-tuning-conf-dir\") pod \"multus-additional-cni-plugins-h49mm\" (UID: \"7153f197-fa4f-4da4-ba82-73f29e4dd2ac\") " pod="openshift-multus/multus-additional-cni-plugins-h49mm"
May 25 12:13:17.824072 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.822497    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2d7e9aa3-9ae5-4e93-8e85-11d5efe2dd99-cni-binary-copy\") pod \"multus-29966\" (UID: \"2d7e9aa3-9ae5-4e93-8e85-11d5efe2dd99\") " pod="openshift-multus/multus-29966"
May 25 12:13:17.824072 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.822529    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2d7e9aa3-9ae5-4e93-8e85-11d5efe2dd99-os-release\") pod \"multus-29966\" (UID: \"2d7e9aa3-9ae5-4e93-8e85-11d5efe2dd99\") " pod="openshift-multus/multus-29966"
May 25 12:13:17.824072 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.822565    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c66cda37-4f3e-4058-a58c-1b168db3edd7-etc-systemd\") pod \"tuned-xp8d2\" (UID: \"c66cda37-4f3e-4058-a58c-1b168db3edd7\") " pod="openshift-cluster-node-tuning-operator/tuned-xp8d2"
May 25 12:13:17.824072 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.822655    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c66cda37-4f3e-4058-a58c-1b168db3edd7-etc-systemd\") pod \"tuned-xp8d2\" (UID: \"c66cda37-4f3e-4058-a58c-1b168db3edd7\") " pod="openshift-cluster-node-tuning-operator/tuned-xp8d2"
May 25 12:13:17.824072 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.822988    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c66cda37-4f3e-4058-a58c-1b168db3edd7-run\") pod \"tuned-xp8d2\" (UID: \"c66cda37-4f3e-4058-a58c-1b168db3edd7\") " pod="openshift-cluster-node-tuning-operator/tuned-xp8d2"
May 25 12:13:17.824072 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.823034    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2d7e9aa3-9ae5-4e93-8e85-11d5efe2dd99-cni-binary-copy\") pod \"multus-29966\" (UID: \"2d7e9aa3-9ae5-4e93-8e85-11d5efe2dd99\") " pod="openshift-multus/multus-29966"
May 25 12:13:17.824786 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.824768    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c66cda37-4f3e-4058-a58c-1b168db3edd7-tmp\") pod \"tuned-xp8d2\" (UID: \"c66cda37-4f3e-4058-a58c-1b168db3edd7\") " pod="openshift-cluster-node-tuning-operator/tuned-xp8d2"
May 25 12:13:17.824892 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.824878    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c66cda37-4f3e-4058-a58c-1b168db3edd7-etc-tuned\") pod \"tuned-xp8d2\" (UID: \"c66cda37-4f3e-4058-a58c-1b168db3edd7\") " pod="openshift-cluster-node-tuning-operator/tuned-xp8d2"
May 25 12:13:17.833384 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.833361    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lcss\" (UniqueName: \"kubernetes.io/projected/1124f1f1-fdc7-4a81-b28a-620db95e2ff6-kube-api-access-5lcss\") pod \"network-metrics-daemon-zf62d\" (UID: \"1124f1f1-fdc7-4a81-b28a-620db95e2ff6\") " pod="openshift-multus/network-metrics-daemon-zf62d"
May 25 12:13:17.833967 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.833951    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nrwl\" (UniqueName: \"kubernetes.io/projected/2d7e9aa3-9ae5-4e93-8e85-11d5efe2dd99-kube-api-access-9nrwl\") pod \"multus-29966\" (UID: \"2d7e9aa3-9ae5-4e93-8e85-11d5efe2dd99\") " pod="openshift-multus/multus-29966"
May 25 12:13:17.834428 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.834409    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ll5zp\" (UniqueName: \"kubernetes.io/projected/c66cda37-4f3e-4058-a58c-1b168db3edd7-kube-api-access-ll5zp\") pod \"tuned-xp8d2\" (UID: \"c66cda37-4f3e-4058-a58c-1b168db3edd7\") " pod="openshift-cluster-node-tuning-operator/tuned-xp8d2"
May 25 12:13:17.876441 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:17.876401    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd968f903a046947131232f9282e5635.slice/crio-b4c404c6ce3721afa96b64c9e660a546dd1fc023d01974082128676304ac8ead WatchSource:0}: Error finding container b4c404c6ce3721afa96b64c9e660a546dd1fc023d01974082128676304ac8ead: Status 404 returned error can't find the container with id b4c404c6ce3721afa96b64c9e660a546dd1fc023d01974082128676304ac8ead
May 25 12:13:17.876716 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:17.876688    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa968fd9574051417c15086d0b1771c9.slice/crio-3e7ceea2947bbb89e64a11abef7c9c70cab082db4f16654880966f7b677513b2 WatchSource:0}: Error finding container 3e7ceea2947bbb89e64a11abef7c9c70cab082db4f16654880966f7b677513b2: Status 404 returned error can't find the container with id 3e7ceea2947bbb89e64a11abef7c9c70cab082db4f16654880966f7b677513b2
May 25 12:13:17.879873 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.879856    2549 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 25 12:13:17.922904 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.922876    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c13dfb5f-85fd-4f8e-9a7f-f432a01720d3-serviceca\") pod \"node-ca-bjpxs\" (UID: \"c13dfb5f-85fd-4f8e-9a7f-f432a01720d3\") " pod="openshift-image-registry/node-ca-bjpxs"
May 25 12:13:17.923031 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.922910    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0b693280-ce91-46e7-b8c8-37d81a9b8fe1-run-systemd\") pod \"ovnkube-node-79j85\" (UID: \"0b693280-ce91-46e7-b8c8-37d81a9b8fe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-79j85"
May 25 12:13:17.923031 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.922935    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0b693280-ce91-46e7-b8c8-37d81a9b8fe1-node-log\") pod \"ovnkube-node-79j85\" (UID: \"0b693280-ce91-46e7-b8c8-37d81a9b8fe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-79j85"
May 25 12:13:17.923031 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.922958    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/69c409ea-f028-429e-8a53-0f25010483bd-host-slash\") pod \"iptables-alerter-g4949\" (UID: \"69c409ea-f028-429e-8a53-0f25010483bd\") " pod="openshift-network-operator/iptables-alerter-g4949"
May 25 12:13:17.923031 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.922991    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0b693280-ce91-46e7-b8c8-37d81a9b8fe1-run-systemd\") pod \"ovnkube-node-79j85\" (UID: \"0b693280-ce91-46e7-b8c8-37d81a9b8fe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-79j85"
May 25 12:13:17.923031 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.923008    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/69c409ea-f028-429e-8a53-0f25010483bd-host-slash\") pod \"iptables-alerter-g4949\" (UID: \"69c409ea-f028-429e-8a53-0f25010483bd\") " pod="openshift-network-operator/iptables-alerter-g4949"
May 25 12:13:17.923031 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.922997    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7bb34bd7-c02c-42cb-b3c8-643fd893a7db-sys-fs\") pod \"aws-ebs-csi-driver-node-l8w7f\" (UID: \"7bb34bd7-c02c-42cb-b3c8-643fd893a7db\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l8w7f"
May 25 12:13:17.923348 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.923050    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7153f197-fa4f-4da4-ba82-73f29e4dd2ac-cni-binary-copy\") pod \"multus-additional-cni-plugins-h49mm\" (UID: \"7153f197-fa4f-4da4-ba82-73f29e4dd2ac\") " pod="openshift-multus/multus-additional-cni-plugins-h49mm"
May 25 12:13:17.923348 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.923055    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0b693280-ce91-46e7-b8c8-37d81a9b8fe1-node-log\") pod \"ovnkube-node-79j85\" (UID: \"0b693280-ce91-46e7-b8c8-37d81a9b8fe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-79j85"
May 25 12:13:17.923348 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.923051    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7bb34bd7-c02c-42cb-b3c8-643fd893a7db-sys-fs\") pod \"aws-ebs-csi-driver-node-l8w7f\" (UID: \"7bb34bd7-c02c-42cb-b3c8-643fd893a7db\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l8w7f"
May 25 12:13:17.923348 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.923081    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7153f197-fa4f-4da4-ba82-73f29e4dd2ac-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-h49mm\" (UID: \"7153f197-fa4f-4da4-ba82-73f29e4dd2ac\") " pod="openshift-multus/multus-additional-cni-plugins-h49mm"
May 25 12:13:17.923348 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.923110    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0b693280-ce91-46e7-b8c8-37d81a9b8fe1-host-run-netns\") pod \"ovnkube-node-79j85\" (UID: \"0b693280-ce91-46e7-b8c8-37d81a9b8fe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-79j85"
May 25 12:13:17.923348 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.923135    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0b693280-ce91-46e7-b8c8-37d81a9b8fe1-host-cni-netd\") pod \"ovnkube-node-79j85\" (UID: \"0b693280-ce91-46e7-b8c8-37d81a9b8fe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-79j85"
May 25 12:13:17.923348 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.923160    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0b693280-ce91-46e7-b8c8-37d81a9b8fe1-ovn-node-metrics-cert\") pod \"ovnkube-node-79j85\" (UID: \"0b693280-ce91-46e7-b8c8-37d81a9b8fe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-79j85"
May 25 12:13:17.923348 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.923183    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0b693280-ce91-46e7-b8c8-37d81a9b8fe1-host-run-netns\") pod \"ovnkube-node-79j85\" (UID: \"0b693280-ce91-46e7-b8c8-37d81a9b8fe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-79j85"
May 25 12:13:17.923348 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.923186    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0b693280-ce91-46e7-b8c8-37d81a9b8fe1-ovnkube-script-lib\") pod \"ovnkube-node-79j85\" (UID: \"0b693280-ce91-46e7-b8c8-37d81a9b8fe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-79j85"
May 25 12:13:17.923348 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.923235    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0b693280-ce91-46e7-b8c8-37d81a9b8fe1-host-cni-netd\") pod \"ovnkube-node-79j85\" (UID: \"0b693280-ce91-46e7-b8c8-37d81a9b8fe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-79j85"
May 25 12:13:17.923348 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.923246    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-46f52\" (UniqueName: \"kubernetes.io/projected/7bb34bd7-c02c-42cb-b3c8-643fd893a7db-kube-api-access-46f52\") pod \"aws-ebs-csi-driver-node-l8w7f\" (UID: \"7bb34bd7-c02c-42cb-b3c8-643fd893a7db\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l8w7f"
May 25 12:13:17.923348 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.923310    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0b693280-ce91-46e7-b8c8-37d81a9b8fe1-etc-openvswitch\") pod \"ovnkube-node-79j85\" (UID: \"0b693280-ce91-46e7-b8c8-37d81a9b8fe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-79j85"
May 25 12:13:17.923348 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.923339    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/7153f197-fa4f-4da4-ba82-73f29e4dd2ac-os-release\") pod \"multus-additional-cni-plugins-h49mm\" (UID: \"7153f197-fa4f-4da4-ba82-73f29e4dd2ac\") " pod="openshift-multus/multus-additional-cni-plugins-h49mm"
May 25 12:13:17.924020 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.923371    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0b693280-ce91-46e7-b8c8-37d81a9b8fe1-systemd-units\") pod \"ovnkube-node-79j85\" (UID: \"0b693280-ce91-46e7-b8c8-37d81a9b8fe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-79j85"
May 25 12:13:17.924020 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.923395    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0b693280-ce91-46e7-b8c8-37d81a9b8fe1-log-socket\") pod \"ovnkube-node-79j85\" (UID: \"0b693280-ce91-46e7-b8c8-37d81a9b8fe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-79j85"
May 25 12:13:17.924020 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.923422    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0b693280-ce91-46e7-b8c8-37d81a9b8fe1-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-79j85\" (UID: \"0b693280-ce91-46e7-b8c8-37d81a9b8fe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-79j85"
May 25 12:13:17.924020 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.923440    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0b693280-ce91-46e7-b8c8-37d81a9b8fe1-etc-openvswitch\") pod \"ovnkube-node-79j85\" (UID: \"0b693280-ce91-46e7-b8c8-37d81a9b8fe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-79j85"
May 25 12:13:17.924020 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.923451    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61e828df-19f1-4008-bcb9-2c3a879b0178-tmp-dir\") pod \"node-resolver-dv9dt\" (UID: \"61e828df-19f1-4008-bcb9-2c3a879b0178\") " pod="openshift-dns/node-resolver-dv9dt"
May 25 12:13:17.924020 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.923472    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0b693280-ce91-46e7-b8c8-37d81a9b8fe1-log-socket\") pod \"ovnkube-node-79j85\" (UID: \"0b693280-ce91-46e7-b8c8-37d81a9b8fe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-79j85"
May 25 12:13:17.924020 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.923394    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c13dfb5f-85fd-4f8e-9a7f-f432a01720d3-serviceca\") pod \"node-ca-bjpxs\" (UID: \"c13dfb5f-85fd-4f8e-9a7f-f432a01720d3\") " pod="openshift-image-registry/node-ca-bjpxs"
May 25 12:13:17.924020 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.923491    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0b693280-ce91-46e7-b8c8-37d81a9b8fe1-systemd-units\") pod \"ovnkube-node-79j85\" (UID: \"0b693280-ce91-46e7-b8c8-37d81a9b8fe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-79j85"
May 25 12:13:17.924020 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.923519    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0b693280-ce91-46e7-b8c8-37d81a9b8fe1-host-kubelet\") pod \"ovnkube-node-79j85\" (UID: \"0b693280-ce91-46e7-b8c8-37d81a9b8fe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-79j85"
May 25 12:13:17.924020 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.923475    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0b693280-ce91-46e7-b8c8-37d81a9b8fe1-host-kubelet\") pod \"ovnkube-node-79j85\" (UID: \"0b693280-ce91-46e7-b8c8-37d81a9b8fe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-79j85"
May 25 12:13:17.924020 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.923541    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0b693280-ce91-46e7-b8c8-37d81a9b8fe1-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-79j85\" (UID: \"0b693280-ce91-46e7-b8c8-37d81a9b8fe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-79j85"
May 25 12:13:17.924020 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.923547    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/7153f197-fa4f-4da4-ba82-73f29e4dd2ac-os-release\") pod \"multus-additional-cni-plugins-h49mm\" (UID: \"7153f197-fa4f-4da4-ba82-73f29e4dd2ac\") " pod="openshift-multus/multus-additional-cni-plugins-h49mm"
May 25 12:13:17.924020 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.923562    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0b693280-ce91-46e7-b8c8-37d81a9b8fe1-host-slash\") pod \"ovnkube-node-79j85\" (UID: \"0b693280-ce91-46e7-b8c8-37d81a9b8fe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-79j85"
May 25 12:13:17.924020 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.923587    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7153f197-fa4f-4da4-ba82-73f29e4dd2ac-tuning-conf-dir\") pod \"multus-additional-cni-plugins-h49mm\" (UID: \"7153f197-fa4f-4da4-ba82-73f29e4dd2ac\") " pod="openshift-multus/multus-additional-cni-plugins-h49mm"
May 25 12:13:17.924020 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.923618    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0b693280-ce91-46e7-b8c8-37d81a9b8fe1-host-run-ovn-kubernetes\") pod \"ovnkube-node-79j85\" (UID: \"0b693280-ce91-46e7-b8c8-37d81a9b8fe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-79j85"
May 25 12:13:17.924020 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.923635    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7153f197-fa4f-4da4-ba82-73f29e4dd2ac-cni-binary-copy\") pod \"multus-additional-cni-plugins-h49mm\" (UID: \"7153f197-fa4f-4da4-ba82-73f29e4dd2ac\") " pod="openshift-multus/multus-additional-cni-plugins-h49mm"
May 25 12:13:17.924020 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.923644    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7153f197-fa4f-4da4-ba82-73f29e4dd2ac-system-cni-dir\") pod \"multus-additional-cni-plugins-h49mm\" (UID: \"7153f197-fa4f-4da4-ba82-73f29e4dd2ac\") " pod="openshift-multus/multus-additional-cni-plugins-h49mm"
May 25 12:13:17.924793 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.923660    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0b693280-ce91-46e7-b8c8-37d81a9b8fe1-host-slash\") pod \"ovnkube-node-79j85\" (UID: \"0b693280-ce91-46e7-b8c8-37d81a9b8fe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-79j85"
May 25 12:13:17.924793 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.923658    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7153f197-fa4f-4da4-ba82-73f29e4dd2ac-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-h49mm\" (UID: \"7153f197-fa4f-4da4-ba82-73f29e4dd2ac\") " pod="openshift-multus/multus-additional-cni-plugins-h49mm"
May 25 12:13:17.924793 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.923698    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7153f197-fa4f-4da4-ba82-73f29e4dd2ac-system-cni-dir\") pod \"multus-additional-cni-plugins-h49mm\" (UID: \"7153f197-fa4f-4da4-ba82-73f29e4dd2ac\") " pod="openshift-multus/multus-additional-cni-plugins-h49mm"
May 25 12:13:17.924793 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.923698    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0b693280-ce91-46e7-b8c8-37d81a9b8fe1-host-run-ovn-kubernetes\") pod \"ovnkube-node-79j85\" (UID: \"0b693280-ce91-46e7-b8c8-37d81a9b8fe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-79j85"
May 25 12:13:17.924793 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.923721    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7153f197-fa4f-4da4-ba82-73f29e4dd2ac-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-h49mm\" (UID: \"7153f197-fa4f-4da4-ba82-73f29e4dd2ac\") " pod="openshift-multus/multus-additional-cni-plugins-h49mm"
May 25 12:13:17.924793 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.923745    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7153f197-fa4f-4da4-ba82-73f29e4dd2ac-tuning-conf-dir\") pod \"multus-additional-cni-plugins-h49mm\" (UID: \"7153f197-fa4f-4da4-ba82-73f29e4dd2ac\") " pod="openshift-multus/multus-additional-cni-plugins-h49mm"
May 25 12:13:17.924793 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.923747    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0b693280-ce91-46e7-b8c8-37d81a9b8fe1-ovnkube-config\") pod \"ovnkube-node-79j85\" (UID: \"0b693280-ce91-46e7-b8c8-37d81a9b8fe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-79j85"
May 25 12:13:17.924793 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.923780    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0b693280-ce91-46e7-b8c8-37d81a9b8fe1-env-overrides\") pod \"ovnkube-node-79j85\" (UID: \"0b693280-ce91-46e7-b8c8-37d81a9b8fe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-79j85"
May 25 12:13:17.924793 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.923806    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61e828df-19f1-4008-bcb9-2c3a879b0178-tmp-dir\") pod \"node-resolver-dv9dt\" (UID: \"61e828df-19f1-4008-bcb9-2c3a879b0178\") " pod="openshift-dns/node-resolver-dv9dt"
May 25 12:13:17.924793 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.923809    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7bb34bd7-c02c-42cb-b3c8-643fd893a7db-registration-dir\") pod \"aws-ebs-csi-driver-node-l8w7f\" (UID: \"7bb34bd7-c02c-42cb-b3c8-643fd893a7db\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l8w7f"
May 25 12:13:17.924793 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.923828    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0b693280-ce91-46e7-b8c8-37d81a9b8fe1-ovnkube-script-lib\") pod \"ovnkube-node-79j85\" (UID: \"0b693280-ce91-46e7-b8c8-37d81a9b8fe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-79j85"
May 25 12:13:17.924793 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.923835    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7bb34bd7-c02c-42cb-b3c8-643fd893a7db-device-dir\") pod \"aws-ebs-csi-driver-node-l8w7f\" (UID: \"7bb34bd7-c02c-42cb-b3c8-643fd893a7db\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l8w7f"
May 25 12:13:17.924793 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.923872    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7bb34bd7-c02c-42cb-b3c8-643fd893a7db-registration-dir\") pod \"aws-ebs-csi-driver-node-l8w7f\" (UID: \"7bb34bd7-c02c-42cb-b3c8-643fd893a7db\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l8w7f"
May 25 12:13:17.924793 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.923877    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7bb34bd7-c02c-42cb-b3c8-643fd893a7db-device-dir\") pod \"aws-ebs-csi-driver-node-l8w7f\" (UID: \"7bb34bd7-c02c-42cb-b3c8-643fd893a7db\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l8w7f"
May 25 12:13:17.924793 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.923878    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-qnhxh\" (UniqueName: \"kubernetes.io/projected/61e828df-19f1-4008-bcb9-2c3a879b0178-kube-api-access-qnhxh\") pod \"node-resolver-dv9dt\" (UID: \"61e828df-19f1-4008-bcb9-2c3a879b0178\") " pod="openshift-dns/node-resolver-dv9dt"
May 25 12:13:17.924793 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.923913    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3472ac4b-9b58-4e85-85e8-f006315d8915-agent-certs\") pod \"konnectivity-agent-tcp4j\" (UID: \"3472ac4b-9b58-4e85-85e8-f006315d8915\") " pod="kube-system/konnectivity-agent-tcp4j"
May 25 12:13:17.924793 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.923936    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-6dgg9\" (UniqueName: \"kubernetes.io/projected/69c409ea-f028-429e-8a53-0f25010483bd-kube-api-access-6dgg9\") pod \"iptables-alerter-g4949\" (UID: \"69c409ea-f028-429e-8a53-0f25010483bd\") " pod="openshift-network-operator/iptables-alerter-g4949"
May 25 12:13:17.925344 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.923951    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0b693280-ce91-46e7-b8c8-37d81a9b8fe1-var-lib-openvswitch\") pod \"ovnkube-node-79j85\" (UID: \"0b693280-ce91-46e7-b8c8-37d81a9b8fe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-79j85"
May 25 12:13:17.925344 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.923966    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0b693280-ce91-46e7-b8c8-37d81a9b8fe1-run-openvswitch\") pod \"ovnkube-node-79j85\" (UID: \"0b693280-ce91-46e7-b8c8-37d81a9b8fe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-79j85"
May 25 12:13:17.925344 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.923987    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3472ac4b-9b58-4e85-85e8-f006315d8915-konnectivity-ca\") pod \"konnectivity-agent-tcp4j\" (UID: \"3472ac4b-9b58-4e85-85e8-f006315d8915\") " pod="kube-system/konnectivity-agent-tcp4j"
May 25 12:13:17.925344 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.924050    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0b693280-ce91-46e7-b8c8-37d81a9b8fe1-var-lib-openvswitch\") pod \"ovnkube-node-79j85\" (UID: \"0b693280-ce91-46e7-b8c8-37d81a9b8fe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-79j85"
May 25 12:13:17.925344 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.924121    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0b693280-ce91-46e7-b8c8-37d81a9b8fe1-run-openvswitch\") pod \"ovnkube-node-79j85\" (UID: \"0b693280-ce91-46e7-b8c8-37d81a9b8fe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-79j85"
May 25 12:13:17.925344 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.924225    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0b693280-ce91-46e7-b8c8-37d81a9b8fe1-ovnkube-config\") pod \"ovnkube-node-79j85\" (UID: \"0b693280-ce91-46e7-b8c8-37d81a9b8fe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-79j85"
May 25 12:13:17.925344 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.924225    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7153f197-fa4f-4da4-ba82-73f29e4dd2ac-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-h49mm\" (UID: \"7153f197-fa4f-4da4-ba82-73f29e4dd2ac\") " pod="openshift-multus/multus-additional-cni-plugins-h49mm"
May 25 12:13:17.925344 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.924230    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0b693280-ce91-46e7-b8c8-37d81a9b8fe1-host-cni-bin\") pod \"ovnkube-node-79j85\" (UID: \"0b693280-ce91-46e7-b8c8-37d81a9b8fe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-79j85"
May 25 12:13:17.925344 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.924269    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0b693280-ce91-46e7-b8c8-37d81a9b8fe1-host-cni-bin\") pod \"ovnkube-node-79j85\" (UID: \"0b693280-ce91-46e7-b8c8-37d81a9b8fe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-79j85"
May 25 12:13:17.925344 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.924276    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/69c409ea-f028-429e-8a53-0f25010483bd-iptables-alerter-script\") pod \"iptables-alerter-g4949\" (UID: \"69c409ea-f028-429e-8a53-0f25010483bd\") " pod="openshift-network-operator/iptables-alerter-g4949"
May 25 12:13:17.925344 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.924301    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7bb34bd7-c02c-42cb-b3c8-643fd893a7db-kubelet-dir\") pod \"aws-ebs-csi-driver-node-l8w7f\" (UID: \"7bb34bd7-c02c-42cb-b3c8-643fd893a7db\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l8w7f"
May 25 12:13:17.925344 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.924326    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7bb34bd7-c02c-42cb-b3c8-643fd893a7db-etc-selinux\") pod \"aws-ebs-csi-driver-node-l8w7f\" (UID: \"7bb34bd7-c02c-42cb-b3c8-643fd893a7db\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l8w7f"
May 25 12:13:17.925344 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.924351    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-dzjf5\" (UniqueName: \"kubernetes.io/projected/c13dfb5f-85fd-4f8e-9a7f-f432a01720d3-kube-api-access-dzjf5\") pod \"node-ca-bjpxs\" (UID: \"c13dfb5f-85fd-4f8e-9a7f-f432a01720d3\") " pod="openshift-image-registry/node-ca-bjpxs"
May 25 12:13:17.925344 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.924403    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7bb34bd7-c02c-42cb-b3c8-643fd893a7db-kubelet-dir\") pod \"aws-ebs-csi-driver-node-l8w7f\" (UID: \"7bb34bd7-c02c-42cb-b3c8-643fd893a7db\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l8w7f"
May 25 12:13:17.925344 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.924434    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0b693280-ce91-46e7-b8c8-37d81a9b8fe1-run-ovn\") pod \"ovnkube-node-79j85\" (UID: \"0b693280-ce91-46e7-b8c8-37d81a9b8fe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-79j85"
May 25 12:13:17.925344 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.924460    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-g7brb\" (UniqueName: \"kubernetes.io/projected/0b693280-ce91-46e7-b8c8-37d81a9b8fe1-kube-api-access-g7brb\") pod \"ovnkube-node-79j85\" (UID: \"0b693280-ce91-46e7-b8c8-37d81a9b8fe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-79j85"
May 25 12:13:17.925344 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.924484    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7bb34bd7-c02c-42cb-b3c8-643fd893a7db-socket-dir\") pod \"aws-ebs-csi-driver-node-l8w7f\" (UID: \"7bb34bd7-c02c-42cb-b3c8-643fd893a7db\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l8w7f"
May 25 12:13:17.925838 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.924489    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3472ac4b-9b58-4e85-85e8-f006315d8915-konnectivity-ca\") pod \"konnectivity-agent-tcp4j\" (UID: \"3472ac4b-9b58-4e85-85e8-f006315d8915\") " pod="kube-system/konnectivity-agent-tcp4j"
May 25 12:13:17.925838 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.924508    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7153f197-fa4f-4da4-ba82-73f29e4dd2ac-cnibin\") pod \"multus-additional-cni-plugins-h49mm\" (UID: \"7153f197-fa4f-4da4-ba82-73f29e4dd2ac\") " pod="openshift-multus/multus-additional-cni-plugins-h49mm"
May 25 12:13:17.925838 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.924535    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/61e828df-19f1-4008-bcb9-2c3a879b0178-hosts-file\") pod \"node-resolver-dv9dt\" (UID: \"61e828df-19f1-4008-bcb9-2c3a879b0178\") " pod="openshift-dns/node-resolver-dv9dt"
May 25 12:13:17.925838 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.924558    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-56c2b\" (UniqueName: \"kubernetes.io/projected/d37b4fc9-9e8b-4e07-9672-b6bffc0f46d8-kube-api-access-56c2b\") pod \"network-check-target-tp68k\" (UID: \"d37b4fc9-9e8b-4e07-9672-b6bffc0f46d8\") " pod="openshift-network-diagnostics/network-check-target-tp68k"
May 25 12:13:17.925838 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.924567    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0b693280-ce91-46e7-b8c8-37d81a9b8fe1-env-overrides\") pod \"ovnkube-node-79j85\" (UID: \"0b693280-ce91-46e7-b8c8-37d81a9b8fe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-79j85"
May 25 12:13:17.925838 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.924581    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c13dfb5f-85fd-4f8e-9a7f-f432a01720d3-host\") pod \"node-ca-bjpxs\" (UID: \"c13dfb5f-85fd-4f8e-9a7f-f432a01720d3\") " pod="openshift-image-registry/node-ca-bjpxs"
May 25 12:13:17.925838 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.924582    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0b693280-ce91-46e7-b8c8-37d81a9b8fe1-run-ovn\") pod \"ovnkube-node-79j85\" (UID: \"0b693280-ce91-46e7-b8c8-37d81a9b8fe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-79j85"
May 25 12:13:17.925838 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.924593    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7bb34bd7-c02c-42cb-b3c8-643fd893a7db-socket-dir\") pod \"aws-ebs-csi-driver-node-l8w7f\" (UID: \"7bb34bd7-c02c-42cb-b3c8-643fd893a7db\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l8w7f"
May 25 12:13:17.925838 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.924622    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7153f197-fa4f-4da4-ba82-73f29e4dd2ac-cnibin\") pod \"multus-additional-cni-plugins-h49mm\" (UID: \"7153f197-fa4f-4da4-ba82-73f29e4dd2ac\") " pod="openshift-multus/multus-additional-cni-plugins-h49mm"
May 25 12:13:17.925838 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.924625    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/61e828df-19f1-4008-bcb9-2c3a879b0178-hosts-file\") pod \"node-resolver-dv9dt\" (UID: \"61e828df-19f1-4008-bcb9-2c3a879b0178\") " pod="openshift-dns/node-resolver-dv9dt"
May 25 12:13:17.925838 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.924489    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7bb34bd7-c02c-42cb-b3c8-643fd893a7db-etc-selinux\") pod \"aws-ebs-csi-driver-node-l8w7f\" (UID: \"7bb34bd7-c02c-42cb-b3c8-643fd893a7db\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l8w7f"
May 25 12:13:17.925838 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.924643    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c13dfb5f-85fd-4f8e-9a7f-f432a01720d3-host\") pod \"node-ca-bjpxs\" (UID: \"c13dfb5f-85fd-4f8e-9a7f-f432a01720d3\") " pod="openshift-image-registry/node-ca-bjpxs"
May 25 12:13:17.925838 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.924650    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-zjnml\" (UniqueName: \"kubernetes.io/projected/7153f197-fa4f-4da4-ba82-73f29e4dd2ac-kube-api-access-zjnml\") pod \"multus-additional-cni-plugins-h49mm\" (UID: \"7153f197-fa4f-4da4-ba82-73f29e4dd2ac\") " pod="openshift-multus/multus-additional-cni-plugins-h49mm"
May 25 12:13:17.925838 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.924717    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/69c409ea-f028-429e-8a53-0f25010483bd-iptables-alerter-script\") pod \"iptables-alerter-g4949\" (UID: \"69c409ea-f028-429e-8a53-0f25010483bd\") " pod="openshift-network-operator/iptables-alerter-g4949"
May 25 12:13:17.925838 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.925659    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0b693280-ce91-46e7-b8c8-37d81a9b8fe1-ovn-node-metrics-cert\") pod \"ovnkube-node-79j85\" (UID: \"0b693280-ce91-46e7-b8c8-37d81a9b8fe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-79j85"
May 25 12:13:17.926315 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.926301    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3472ac4b-9b58-4e85-85e8-f006315d8915-agent-certs\") pod \"konnectivity-agent-tcp4j\" (UID: \"3472ac4b-9b58-4e85-85e8-f006315d8915\") " pod="kube-system/konnectivity-agent-tcp4j"
May 25 12:13:17.932694 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.932679    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-46f52\" (UniqueName: \"kubernetes.io/projected/7bb34bd7-c02c-42cb-b3c8-643fd893a7db-kube-api-access-46f52\") pod \"aws-ebs-csi-driver-node-l8w7f\" (UID: \"7bb34bd7-c02c-42cb-b3c8-643fd893a7db\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l8w7f"
May 25 12:13:17.936798 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:13:17.936781    2549 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 25 12:13:17.936798 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:13:17.936799    2549 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 25 12:13:17.936976 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:13:17.936808    2549 projected.go:194] Error preparing data for projected volume kube-api-access-56c2b for pod openshift-network-diagnostics/network-check-target-tp68k: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 12:13:17.936976 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:13:17.936866    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d37b4fc9-9e8b-4e07-9672-b6bffc0f46d8-kube-api-access-56c2b podName:d37b4fc9-9e8b-4e07-9672-b6bffc0f46d8 nodeName:}" failed. No retries permitted until 2026-05-25 12:13:18.436852178 +0000 UTC m=+2.152667460 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-56c2b" (UniqueName: "kubernetes.io/projected/d37b4fc9-9e8b-4e07-9672-b6bffc0f46d8-kube-api-access-56c2b") pod "network-check-target-tp68k" (UID: "d37b4fc9-9e8b-4e07-9672-b6bffc0f46d8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 12:13:17.938163 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.938135    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dgg9\" (UniqueName: \"kubernetes.io/projected/69c409ea-f028-429e-8a53-0f25010483bd-kube-api-access-6dgg9\") pod \"iptables-alerter-g4949\" (UID: \"69c409ea-f028-429e-8a53-0f25010483bd\") " pod="openshift-network-operator/iptables-alerter-g4949"
May 25 12:13:17.938311 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.938294    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzjf5\" (UniqueName: \"kubernetes.io/projected/c13dfb5f-85fd-4f8e-9a7f-f432a01720d3-kube-api-access-dzjf5\") pod \"node-ca-bjpxs\" (UID: \"c13dfb5f-85fd-4f8e-9a7f-f432a01720d3\") " pod="openshift-image-registry/node-ca-bjpxs"
May 25 12:13:17.938508 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.938493    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnhxh\" (UniqueName: \"kubernetes.io/projected/61e828df-19f1-4008-bcb9-2c3a879b0178-kube-api-access-qnhxh\") pod \"node-resolver-dv9dt\" (UID: \"61e828df-19f1-4008-bcb9-2c3a879b0178\") " pod="openshift-dns/node-resolver-dv9dt"
May 25 12:13:17.938832 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.938794    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7brb\" (UniqueName: \"kubernetes.io/projected/0b693280-ce91-46e7-b8c8-37d81a9b8fe1-kube-api-access-g7brb\") pod \"ovnkube-node-79j85\" (UID: \"0b693280-ce91-46e7-b8c8-37d81a9b8fe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-79j85"
May 25 12:13:17.939170 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:17.939148    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjnml\" (UniqueName: \"kubernetes.io/projected/7153f197-fa4f-4da4-ba82-73f29e4dd2ac-kube-api-access-zjnml\") pod \"multus-additional-cni-plugins-h49mm\" (UID: \"7153f197-fa4f-4da4-ba82-73f29e4dd2ac\") " pod="openshift-multus/multus-additional-cni-plugins-h49mm"
May 25 12:13:18.020715 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:18.020691    2549 reflector.go:430] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
May 25 12:13:18.030062 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:18.030043    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-29966"
May 25 12:13:18.036646 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:18.036624    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d7e9aa3_9ae5_4e93_8e85_11d5efe2dd99.slice/crio-bc1c6f2d475cae8648977679075f079afae0166892318c0a5dacd443bdfe51d6 WatchSource:0}: Error finding container bc1c6f2d475cae8648977679075f079afae0166892318c0a5dacd443bdfe51d6: Status 404 returned error can't find the container with id bc1c6f2d475cae8648977679075f079afae0166892318c0a5dacd443bdfe51d6
May 25 12:13:18.040544 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:18.040529    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-xp8d2"
May 25 12:13:18.067081 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:18.067058    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-79j85"
May 25 12:13:18.072341 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:18.072319    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b693280_ce91_46e7_b8c8_37d81a9b8fe1.slice/crio-b1171300b2824cd7e3100854f0e5f4b009788c01a09e366e6100177d154635f7 WatchSource:0}: Error finding container b1171300b2824cd7e3100854f0e5f4b009788c01a09e366e6100177d154635f7: Status 404 returned error can't find the container with id b1171300b2824cd7e3100854f0e5f4b009788c01a09e366e6100177d154635f7
May 25 12:13:18.076066 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:18.076050    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-tcp4j"
May 25 12:13:18.082362 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:18.082341    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3472ac4b_9b58_4e85_85e8_f006315d8915.slice/crio-6228e2fc92b72568163d83cba5f142b0073ae59ff6a21f28192f69e4f06db6f6 WatchSource:0}: Error finding container 6228e2fc92b72568163d83cba5f142b0073ae59ff6a21f28192f69e4f06db6f6: Status 404 returned error can't find the container with id 6228e2fc92b72568163d83cba5f142b0073ae59ff6a21f28192f69e4f06db6f6
May 25 12:13:18.089442 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:18.089424    2549 reflector.go:430] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
May 25 12:13:18.092253 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:18.092238    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-h49mm"
May 25 12:13:18.097846 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:18.097826    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7153f197_fa4f_4da4_ba82_73f29e4dd2ac.slice/crio-a609da79a95406148c2505f09da1eb7f01774a0bed27bffcf6e0f2e28fda6cb1 WatchSource:0}: Error finding container a609da79a95406148c2505f09da1eb7f01774a0bed27bffcf6e0f2e28fda6cb1: Status 404 returned error can't find the container with id a609da79a95406148c2505f09da1eb7f01774a0bed27bffcf6e0f2e28fda6cb1
May 25 12:13:18.106655 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:18.106641    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-g4949"
May 25 12:13:18.111899 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:18.111877    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69c409ea_f028_429e_8a53_0f25010483bd.slice/crio-e0805114da15c8afd464f5888e5216aa9e798d2c15224d23e9843f24a1051f6a WatchSource:0}: Error finding container e0805114da15c8afd464f5888e5216aa9e798d2c15224d23e9843f24a1051f6a: Status 404 returned error can't find the container with id e0805114da15c8afd464f5888e5216aa9e798d2c15224d23e9843f24a1051f6a
May 25 12:13:18.120732 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:18.120714    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l8w7f"
May 25 12:13:18.126412 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:18.126392    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bb34bd7_c02c_42cb_b3c8_643fd893a7db.slice/crio-26ab17ba8f0edea89d6ddef7baae78beed7076481283221b266853616ead8010 WatchSource:0}: Error finding container 26ab17ba8f0edea89d6ddef7baae78beed7076481283221b266853616ead8010: Status 404 returned error can't find the container with id 26ab17ba8f0edea89d6ddef7baae78beed7076481283221b266853616ead8010
May 25 12:13:18.142333 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:18.142316    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-dv9dt"
May 25 12:13:18.147837 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:18.147818    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-bjpxs"
May 25 12:13:18.148903 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:18.148882    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61e828df_19f1_4008_bcb9_2c3a879b0178.slice/crio-0389cd4c6c971030084dec6b5305319b1f0b2ec7751d80e211e133254874d32b WatchSource:0}: Error finding container 0389cd4c6c971030084dec6b5305319b1f0b2ec7751d80e211e133254874d32b: Status 404 returned error can't find the container with id 0389cd4c6c971030084dec6b5305319b1f0b2ec7751d80e211e133254874d32b
May 25 12:13:18.153284 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:13:18.153260    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc13dfb5f_85fd_4f8e_9a7f_f432a01720d3.slice/crio-0711623ce2e6b728ab712b57b1652ba911cbcac9a4db832bcd25fb5154582603 WatchSource:0}: Error finding container 0711623ce2e6b728ab712b57b1652ba911cbcac9a4db832bcd25fb5154582603: Status 404 returned error can't find the container with id 0711623ce2e6b728ab712b57b1652ba911cbcac9a4db832bcd25fb5154582603
May 25 12:13:18.326591 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:18.326507    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1124f1f1-fdc7-4a81-b28a-620db95e2ff6-metrics-certs\") pod \"network-metrics-daemon-zf62d\" (UID: \"1124f1f1-fdc7-4a81-b28a-620db95e2ff6\") " pod="openshift-multus/network-metrics-daemon-zf62d"
May 25 12:13:18.326728 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:13:18.326645    2549 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 12:13:18.326728 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:13:18.326704    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1124f1f1-fdc7-4a81-b28a-620db95e2ff6-metrics-certs podName:1124f1f1-fdc7-4a81-b28a-620db95e2ff6 nodeName:}" failed. No retries permitted until 2026-05-25 12:13:19.326690372 +0000 UTC m=+3.042505637 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1124f1f1-fdc7-4a81-b28a-620db95e2ff6-metrics-certs") pod "network-metrics-daemon-zf62d" (UID: "1124f1f1-fdc7-4a81-b28a-620db95e2ff6") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 12:13:18.527890 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:18.527832    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-56c2b\" (UniqueName: \"kubernetes.io/projected/d37b4fc9-9e8b-4e07-9672-b6bffc0f46d8-kube-api-access-56c2b\") pod \"network-check-target-tp68k\" (UID: \"d37b4fc9-9e8b-4e07-9672-b6bffc0f46d8\") " pod="openshift-network-diagnostics/network-check-target-tp68k"
May 25 12:13:18.528063 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:13:18.528008    2549 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 25 12:13:18.528063 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:13:18.528025    2549 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 25 12:13:18.528063 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:13:18.528037    2549 projected.go:194] Error preparing data for projected volume kube-api-access-56c2b for pod openshift-network-diagnostics/network-check-target-tp68k: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 12:13:18.528230 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:13:18.528089    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d37b4fc9-9e8b-4e07-9672-b6bffc0f46d8-kube-api-access-56c2b podName:d37b4fc9-9e8b-4e07-9672-b6bffc0f46d8 nodeName:}" failed. No retries permitted until 2026-05-25 12:13:19.528072353 +0000 UTC m=+3.243887625 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-56c2b" (UniqueName: "kubernetes.io/projected/d37b4fc9-9e8b-4e07-9672-b6bffc0f46d8-kube-api-access-56c2b") pod "network-check-target-tp68k" (UID: "d37b4fc9-9e8b-4e07-9672-b6bffc0f46d8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 12:13:18.708760 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:18.708680    2549 reflector.go:430] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
May 25 12:13:18.752785 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:18.752656    2549 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-05-24 12:08:17 +0000 UTC" deadline="2027-12-01 02:54:09.25972121 +0000 UTC"
May 25 12:13:18.752785 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:18.752692    2549 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13310h40m50.50703396s"
May 25 12:13:18.846090 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:18.846040    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tp68k"
May 25 12:13:18.846257 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:13:18.846156    2549 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tp68k" podUID="d37b4fc9-9e8b-4e07-9672-b6bffc0f46d8"
May 25 12:13:18.867477 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:18.867372    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-bjpxs" event={"ID":"c13dfb5f-85fd-4f8e-9a7f-f432a01720d3","Type":"ContainerStarted","Data":"0711623ce2e6b728ab712b57b1652ba911cbcac9a4db832bcd25fb5154582603"}
May 25 12:13:18.879756 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:18.879704    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-g4949" event={"ID":"69c409ea-f028-429e-8a53-0f25010483bd","Type":"ContainerStarted","Data":"e0805114da15c8afd464f5888e5216aa9e798d2c15224d23e9843f24a1051f6a"}
May 25 12:13:18.896169 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:18.896141    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-h49mm" event={"ID":"7153f197-fa4f-4da4-ba82-73f29e4dd2ac","Type":"ContainerStarted","Data":"a609da79a95406148c2505f09da1eb7f01774a0bed27bffcf6e0f2e28fda6cb1"}
May 25 12:13:18.909940 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:18.909913    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-79j85" event={"ID":"0b693280-ce91-46e7-b8c8-37d81a9b8fe1","Type":"ContainerStarted","Data":"b1171300b2824cd7e3100854f0e5f4b009788c01a09e366e6100177d154635f7"}
May 25 12:13:18.913033 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:18.913002    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-xp8d2" event={"ID":"c66cda37-4f3e-4058-a58c-1b168db3edd7","Type":"ContainerStarted","Data":"0158425cf6da81ea7d8199f31ece3560af86e5bad996215474b731ae2c6d4823"}
May 25 12:13:18.930009 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:18.929621    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-29966" event={"ID":"2d7e9aa3-9ae5-4e93-8e85-11d5efe2dd99","Type":"ContainerStarted","Data":"bc1c6f2d475cae8648977679075f079afae0166892318c0a5dacd443bdfe51d6"}
May 25 12:13:18.956819 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:18.956582    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-dv9dt" event={"ID":"61e828df-19f1-4008-bcb9-2c3a879b0178","Type":"ContainerStarted","Data":"0389cd4c6c971030084dec6b5305319b1f0b2ec7751d80e211e133254874d32b"}
May 25 12:13:18.965906 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:18.965835    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l8w7f" event={"ID":"7bb34bd7-c02c-42cb-b3c8-643fd893a7db","Type":"ContainerStarted","Data":"26ab17ba8f0edea89d6ddef7baae78beed7076481283221b266853616ead8010"}
May 25 12:13:18.987563 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:18.987530    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-tcp4j" event={"ID":"3472ac4b-9b58-4e85-85e8-f006315d8915","Type":"ContainerStarted","Data":"6228e2fc92b72568163d83cba5f142b0073ae59ff6a21f28192f69e4f06db6f6"}
May 25 12:13:19.001320 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:19.001230    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-53.ec2.internal" event={"ID":"fa968fd9574051417c15086d0b1771c9","Type":"ContainerStarted","Data":"3e7ceea2947bbb89e64a11abef7c9c70cab082db4f16654880966f7b677513b2"}
May 25 12:13:19.017306 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:19.017181    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-53.ec2.internal" event={"ID":"dd968f903a046947131232f9282e5635","Type":"ContainerStarted","Data":"b4c404c6ce3721afa96b64c9e660a546dd1fc023d01974082128676304ac8ead"}
May 25 12:13:19.333700 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:19.333615    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1124f1f1-fdc7-4a81-b28a-620db95e2ff6-metrics-certs\") pod \"network-metrics-daemon-zf62d\" (UID: \"1124f1f1-fdc7-4a81-b28a-620db95e2ff6\") " pod="openshift-multus/network-metrics-daemon-zf62d"
May 25 12:13:19.333859 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:13:19.333774    2549 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 12:13:19.333859 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:13:19.333842    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1124f1f1-fdc7-4a81-b28a-620db95e2ff6-metrics-certs podName:1124f1f1-fdc7-4a81-b28a-620db95e2ff6 nodeName:}" failed. No retries permitted until 2026-05-25 12:13:21.333820174 +0000 UTC m=+5.049635459 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1124f1f1-fdc7-4a81-b28a-620db95e2ff6-metrics-certs") pod "network-metrics-daemon-zf62d" (UID: "1124f1f1-fdc7-4a81-b28a-620db95e2ff6") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 12:13:19.386166 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:19.386138    2549 reflector.go:430] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
May 25 12:13:19.535383 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:19.535342    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-56c2b\" (UniqueName: \"kubernetes.io/projected/d37b4fc9-9e8b-4e07-9672-b6bffc0f46d8-kube-api-access-56c2b\") pod \"network-check-target-tp68k\" (UID: \"d37b4fc9-9e8b-4e07-9672-b6bffc0f46d8\") " pod="openshift-network-diagnostics/network-check-target-tp68k"
May 25 12:13:19.535545 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:13:19.535522    2549 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 25 12:13:19.535545 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:13:19.535543    2549 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 25 12:13:19.535646 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:13:19.535556    2549 projected.go:194] Error preparing data for projected volume kube-api-access-56c2b for pod openshift-network-diagnostics/network-check-target-tp68k: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 12:13:19.535646 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:13:19.535613    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d37b4fc9-9e8b-4e07-9672-b6bffc0f46d8-kube-api-access-56c2b podName:d37b4fc9-9e8b-4e07-9672-b6bffc0f46d8 nodeName:}" failed. No retries permitted until 2026-05-25 12:13:21.535594519 +0000 UTC m=+5.251409789 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-56c2b" (UniqueName: "kubernetes.io/projected/d37b4fc9-9e8b-4e07-9672-b6bffc0f46d8-kube-api-access-56c2b") pod "network-check-target-tp68k" (UID: "d37b4fc9-9e8b-4e07-9672-b6bffc0f46d8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 12:13:19.753775 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:19.753681    2549 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-05-24 12:08:17 +0000 UTC" deadline="2027-12-10 10:34:22.252359671 +0000 UTC"
May 25 12:13:19.753775 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:19.753723    2549 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13534h21m2.498640288s"
May 25 12:13:19.843388 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:19.843357    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zf62d"
May 25 12:13:19.843572 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:13:19.843493    2549 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zf62d" podUID="1124f1f1-fdc7-4a81-b28a-620db95e2ff6"
May 25 12:13:20.845532 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:20.845457    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tp68k"
May 25 12:13:20.845982 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:13:20.845582    2549 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tp68k" podUID="d37b4fc9-9e8b-4e07-9672-b6bffc0f46d8"
May 25 12:13:21.349032 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:21.348999    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1124f1f1-fdc7-4a81-b28a-620db95e2ff6-metrics-certs\") pod \"network-metrics-daemon-zf62d\" (UID: \"1124f1f1-fdc7-4a81-b28a-620db95e2ff6\") " pod="openshift-multus/network-metrics-daemon-zf62d"
May 25 12:13:21.349325 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:13:21.349174    2549 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 12:13:21.349325 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:13:21.349261    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1124f1f1-fdc7-4a81-b28a-620db95e2ff6-metrics-certs podName:1124f1f1-fdc7-4a81-b28a-620db95e2ff6 nodeName:}" failed. No retries permitted until 2026-05-25 12:13:25.349238453 +0000 UTC m=+9.065053752 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1124f1f1-fdc7-4a81-b28a-620db95e2ff6-metrics-certs") pod "network-metrics-daemon-zf62d" (UID: "1124f1f1-fdc7-4a81-b28a-620db95e2ff6") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 12:13:21.552005 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:21.551374    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-56c2b\" (UniqueName: \"kubernetes.io/projected/d37b4fc9-9e8b-4e07-9672-b6bffc0f46d8-kube-api-access-56c2b\") pod \"network-check-target-tp68k\" (UID: \"d37b4fc9-9e8b-4e07-9672-b6bffc0f46d8\") " pod="openshift-network-diagnostics/network-check-target-tp68k"
May 25 12:13:21.552005 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:13:21.551570    2549 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 25 12:13:21.552005 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:13:21.551590    2549 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 25 12:13:21.552005 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:13:21.551603    2549 projected.go:194] Error preparing data for projected volume kube-api-access-56c2b for pod openshift-network-diagnostics/network-check-target-tp68k: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 12:13:21.552005 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:13:21.551662    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d37b4fc9-9e8b-4e07-9672-b6bffc0f46d8-kube-api-access-56c2b podName:d37b4fc9-9e8b-4e07-9672-b6bffc0f46d8 nodeName:}" failed. No retries permitted until 2026-05-25 12:13:25.55164423 +0000 UTC m=+9.267459518 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-56c2b" (UniqueName: "kubernetes.io/projected/d37b4fc9-9e8b-4e07-9672-b6bffc0f46d8-kube-api-access-56c2b") pod "network-check-target-tp68k" (UID: "d37b4fc9-9e8b-4e07-9672-b6bffc0f46d8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 12:13:21.843770 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:21.843446    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zf62d"
May 25 12:13:21.843770 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:13:21.843597    2549 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zf62d" podUID="1124f1f1-fdc7-4a81-b28a-620db95e2ff6"
May 25 12:13:22.843153 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:22.843122    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tp68k"
May 25 12:13:22.843622 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:13:22.843267    2549 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tp68k" podUID="d37b4fc9-9e8b-4e07-9672-b6bffc0f46d8"
May 25 12:13:23.843774 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:23.843270    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zf62d"
May 25 12:13:23.843774 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:13:23.843420    2549 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zf62d" podUID="1124f1f1-fdc7-4a81-b28a-620db95e2ff6"
May 25 12:13:24.846616 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:24.846587    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tp68k"
May 25 12:13:24.847100 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:13:24.846711    2549 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tp68k" podUID="d37b4fc9-9e8b-4e07-9672-b6bffc0f46d8"
May 25 12:13:25.379088 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:25.379046    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1124f1f1-fdc7-4a81-b28a-620db95e2ff6-metrics-certs\") pod \"network-metrics-daemon-zf62d\" (UID: \"1124f1f1-fdc7-4a81-b28a-620db95e2ff6\") " pod="openshift-multus/network-metrics-daemon-zf62d"
May 25 12:13:25.379323 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:13:25.379293    2549 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 12:13:25.379457 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:13:25.379386    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1124f1f1-fdc7-4a81-b28a-620db95e2ff6-metrics-certs podName:1124f1f1-fdc7-4a81-b28a-620db95e2ff6 nodeName:}" failed. No retries permitted until 2026-05-25 12:13:33.379361483 +0000 UTC m=+17.095176752 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1124f1f1-fdc7-4a81-b28a-620db95e2ff6-metrics-certs") pod "network-metrics-daemon-zf62d" (UID: "1124f1f1-fdc7-4a81-b28a-620db95e2ff6") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 12:13:25.580915 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:25.580450    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-56c2b\" (UniqueName: \"kubernetes.io/projected/d37b4fc9-9e8b-4e07-9672-b6bffc0f46d8-kube-api-access-56c2b\") pod \"network-check-target-tp68k\" (UID: \"d37b4fc9-9e8b-4e07-9672-b6bffc0f46d8\") " pod="openshift-network-diagnostics/network-check-target-tp68k"
May 25 12:13:25.580915 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:13:25.580647    2549 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 25 12:13:25.580915 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:13:25.580665    2549 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 25 12:13:25.580915 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:13:25.580676    2549 projected.go:194] Error preparing data for projected volume kube-api-access-56c2b for pod openshift-network-diagnostics/network-check-target-tp68k: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 12:13:25.580915 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:13:25.580734    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d37b4fc9-9e8b-4e07-9672-b6bffc0f46d8-kube-api-access-56c2b podName:d37b4fc9-9e8b-4e07-9672-b6bffc0f46d8 nodeName:}" failed. No retries permitted until 2026-05-25 12:13:33.580717249 +0000 UTC m=+17.296532529 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-56c2b" (UniqueName: "kubernetes.io/projected/d37b4fc9-9e8b-4e07-9672-b6bffc0f46d8-kube-api-access-56c2b") pod "network-check-target-tp68k" (UID: "d37b4fc9-9e8b-4e07-9672-b6bffc0f46d8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 12:13:25.843407 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:25.843308    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zf62d"
May 25 12:13:25.843576 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:13:25.843458    2549 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zf62d" podUID="1124f1f1-fdc7-4a81-b28a-620db95e2ff6"
May 25 12:13:26.844408 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:26.843923    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tp68k"
May 25 12:13:26.844408 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:13:26.844052    2549 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tp68k" podUID="d37b4fc9-9e8b-4e07-9672-b6bffc0f46d8"
May 25 12:13:27.842870 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:27.842813    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zf62d"
May 25 12:13:27.843045 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:13:27.842964    2549 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zf62d" podUID="1124f1f1-fdc7-4a81-b28a-620db95e2ff6"
May 25 12:13:28.842684 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:28.842656    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tp68k"
May 25 12:13:28.843070 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:13:28.842745    2549 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tp68k" podUID="d37b4fc9-9e8b-4e07-9672-b6bffc0f46d8"
May 25 12:13:29.842711 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:29.842676    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zf62d"
May 25 12:13:29.843156 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:13:29.842821    2549 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zf62d" podUID="1124f1f1-fdc7-4a81-b28a-620db95e2ff6"
May 25 12:13:30.843159 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:30.843122    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tp68k"
May 25 12:13:30.843601 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:13:30.843270    2549 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tp68k" podUID="d37b4fc9-9e8b-4e07-9672-b6bffc0f46d8"
May 25 12:13:31.843377 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:31.843343    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zf62d"
May 25 12:13:31.843774 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:13:31.843463    2549 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zf62d" podUID="1124f1f1-fdc7-4a81-b28a-620db95e2ff6"
May 25 12:13:32.843312 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:32.843235    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tp68k"
May 25 12:13:32.843471 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:13:32.843352    2549 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tp68k" podUID="d37b4fc9-9e8b-4e07-9672-b6bffc0f46d8"
May 25 12:13:33.436138 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:33.436099    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1124f1f1-fdc7-4a81-b28a-620db95e2ff6-metrics-certs\") pod \"network-metrics-daemon-zf62d\" (UID: \"1124f1f1-fdc7-4a81-b28a-620db95e2ff6\") " pod="openshift-multus/network-metrics-daemon-zf62d"
May 25 12:13:33.436371 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:13:33.436246    2549 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 12:13:33.436371 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:13:33.436319    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1124f1f1-fdc7-4a81-b28a-620db95e2ff6-metrics-certs podName:1124f1f1-fdc7-4a81-b28a-620db95e2ff6 nodeName:}" failed. No retries permitted until 2026-05-25 12:13:49.436302278 +0000 UTC m=+33.152117545 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1124f1f1-fdc7-4a81-b28a-620db95e2ff6-metrics-certs") pod "network-metrics-daemon-zf62d" (UID: "1124f1f1-fdc7-4a81-b28a-620db95e2ff6") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 12:13:33.636827 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:33.636787    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-56c2b\" (UniqueName: \"kubernetes.io/projected/d37b4fc9-9e8b-4e07-9672-b6bffc0f46d8-kube-api-access-56c2b\") pod \"network-check-target-tp68k\" (UID: \"d37b4fc9-9e8b-4e07-9672-b6bffc0f46d8\") " pod="openshift-network-diagnostics/network-check-target-tp68k"
May 25 12:13:33.637012 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:13:33.636937    2549 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 25 12:13:33.637012 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:13:33.636960    2549 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 25 12:13:33.637012 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:13:33.636970    2549 projected.go:194] Error preparing data for projected volume kube-api-access-56c2b for pod openshift-network-diagnostics/network-check-target-tp68k: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 12:13:33.637163 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:13:33.637022    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d37b4fc9-9e8b-4e07-9672-b6bffc0f46d8-kube-api-access-56c2b podName:d37b4fc9-9e8b-4e07-9672-b6bffc0f46d8 nodeName:}" failed. No retries permitted until 2026-05-25 12:13:49.637005976 +0000 UTC m=+33.352821240 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-56c2b" (UniqueName: "kubernetes.io/projected/d37b4fc9-9e8b-4e07-9672-b6bffc0f46d8-kube-api-access-56c2b") pod "network-check-target-tp68k" (UID: "d37b4fc9-9e8b-4e07-9672-b6bffc0f46d8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 12:13:33.843137 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:33.843055    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zf62d"
May 25 12:13:33.843301 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:13:33.843182    2549 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zf62d" podUID="1124f1f1-fdc7-4a81-b28a-620db95e2ff6"
May 25 12:13:34.843304 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:34.843270    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tp68k"
May 25 12:13:34.843706 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:13:34.843389    2549 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tp68k" podUID="d37b4fc9-9e8b-4e07-9672-b6bffc0f46d8"
May 25 12:13:35.843516 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:35.843480    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zf62d"
May 25 12:13:35.844044 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:13:35.843613    2549 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zf62d" podUID="1124f1f1-fdc7-4a81-b28a-620db95e2ff6"
May 25 12:13:36.845700 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:36.845532    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tp68k"
May 25 12:13:36.846033 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:13:36.845762    2549 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tp68k" podUID="d37b4fc9-9e8b-4e07-9672-b6bffc0f46d8"
May 25 12:13:37.843564 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:37.843391    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zf62d"
May 25 12:13:37.843672 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:13:37.843650    2549 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zf62d" podUID="1124f1f1-fdc7-4a81-b28a-620db95e2ff6"
May 25 12:13:38.057881 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:38.057795    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-29966" event={"ID":"2d7e9aa3-9ae5-4e93-8e85-11d5efe2dd99","Type":"ContainerStarted","Data":"bca8a5cd1b243251f91501a2dbec63c7ffc1234307b1d6c507735723733b6e84"}
May 25 12:13:38.059441 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:38.059414    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-dv9dt" event={"ID":"61e828df-19f1-4008-bcb9-2c3a879b0178","Type":"ContainerStarted","Data":"4af62aaf49b8d28ff7795a0a8fd2f113b036999906e87f0ca06e9fa6fef55240"}
May 25 12:13:38.060811 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:38.060756    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l8w7f" event={"ID":"7bb34bd7-c02c-42cb-b3c8-643fd893a7db","Type":"ContainerStarted","Data":"ca385d28d3cb9a34a3e2546db818ed32ee92367b420fcf4979e920a192cf60e1"}
May 25 12:13:38.061988 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:38.061963    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-tcp4j" event={"ID":"3472ac4b-9b58-4e85-85e8-f006315d8915","Type":"ContainerStarted","Data":"8aede58ab07daa8991fd5fdb4efe4b9f873faf95a43b39b6e4bc0ffcfebad51b"}
May 25 12:13:38.063423 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:38.063400    2549 generic.go:358] "Generic (PLEG): container finished" podID="fa968fd9574051417c15086d0b1771c9" containerID="ce3448a9216cfe605196f50864e98bb0a4b07ea3b869887d856dc0212d519220" exitCode=0
May 25 12:13:38.063518 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:38.063473    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-53.ec2.internal" event={"ID":"fa968fd9574051417c15086d0b1771c9","Type":"ContainerDied","Data":"ce3448a9216cfe605196f50864e98bb0a4b07ea3b869887d856dc0212d519220"}
May 25 12:13:38.064860 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:38.064840    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-53.ec2.internal" event={"ID":"dd968f903a046947131232f9282e5635","Type":"ContainerStarted","Data":"ccb662177a11c89b1df38520d3ba3f11cb2f80cb3f32cb914f8bb7086d6b4d4b"}
May 25 12:13:38.066347 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:38.066326    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-bjpxs" event={"ID":"c13dfb5f-85fd-4f8e-9a7f-f432a01720d3","Type":"ContainerStarted","Data":"b6327db4c8d343c213f07ed6eb5d911bd4712f4ff454338ad18fd509ea034961"}
May 25 12:13:38.067970 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:38.067946    2549 generic.go:358] "Generic (PLEG): container finished" podID="7153f197-fa4f-4da4-ba82-73f29e4dd2ac" containerID="06bf31ce5f7fa7c3437b02b49caae4fa1356c4edc68151d4d3e7b8c6056581b0" exitCode=0
May 25 12:13:38.068059 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:38.068026    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-h49mm" event={"ID":"7153f197-fa4f-4da4-ba82-73f29e4dd2ac","Type":"ContainerDied","Data":"06bf31ce5f7fa7c3437b02b49caae4fa1356c4edc68151d4d3e7b8c6056581b0"}
May 25 12:13:38.071146 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:38.071117    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-79j85" event={"ID":"0b693280-ce91-46e7-b8c8-37d81a9b8fe1","Type":"ContainerStarted","Data":"70bc5a90f68464e3c8ba03c7cd40e4eb8547bec2bd4fc3e751c68cf10f08fdc5"}
May 25 12:13:38.071264 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:38.071147    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-79j85" event={"ID":"0b693280-ce91-46e7-b8c8-37d81a9b8fe1","Type":"ContainerStarted","Data":"c6c6e915690243859d58948efb4de449d2f357a7154b1de05a71c4ade49ce715"}
May 25 12:13:38.071264 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:38.071164    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-79j85" event={"ID":"0b693280-ce91-46e7-b8c8-37d81a9b8fe1","Type":"ContainerStarted","Data":"f74fa5d3a1c874f8259f79df0965f2c88885a574ec28e536bf86e23cd6049935"}
May 25 12:13:38.071264 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:38.071176    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-79j85" event={"ID":"0b693280-ce91-46e7-b8c8-37d81a9b8fe1","Type":"ContainerStarted","Data":"e5ebcd99f2761bf5872de42e2f332ecdc375555744ebb859318aa1bbfaaf9bce"}
May 25 12:13:38.071264 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:38.071213    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-79j85" event={"ID":"0b693280-ce91-46e7-b8c8-37d81a9b8fe1","Type":"ContainerStarted","Data":"893f159e6abbb877833eedf84a3dd2327e07f6f55721b28a153975f1e87e2882"}
May 25 12:13:38.071264 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:38.071227    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-79j85" event={"ID":"0b693280-ce91-46e7-b8c8-37d81a9b8fe1","Type":"ContainerStarted","Data":"c8ee82c6163d99ce1afeac2b8b447005b3ad779762b563504d882392fb296a47"}
May 25 12:13:38.072540 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:38.072472    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-xp8d2" event={"ID":"c66cda37-4f3e-4058-a58c-1b168db3edd7","Type":"ContainerStarted","Data":"59bd43b700841af8c6566555c7f765638fcf07dcb6816180323a6f1d464d2c4f"}
May 25 12:13:38.079058 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:38.079013    2549 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/multus-29966" podStartSLOduration=2.92293305 podStartE2EDuration="22.078999593s" podCreationTimestamp="2026-05-25 12:13:16 +0000 UTC" firstStartedPulling="2026-05-25 12:13:18.038149252 +0000 UTC m=+1.753964516" lastFinishedPulling="2026-05-25 12:13:37.194215774 +0000 UTC m=+20.910031059" observedRunningTime="2026-05-25 12:13:38.078226425 +0000 UTC m=+21.794041711" watchObservedRunningTime="2026-05-25 12:13:38.078999593 +0000 UTC m=+21.794814882"
May 25 12:13:38.095629 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:38.095580    2549 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/konnectivity-agent-tcp4j" podStartSLOduration=2.986128362 podStartE2EDuration="22.095564741s" podCreationTimestamp="2026-05-25 12:13:16 +0000 UTC" firstStartedPulling="2026-05-25 12:13:18.083683781 +0000 UTC m=+1.799499047" lastFinishedPulling="2026-05-25 12:13:37.193120156 +0000 UTC m=+20.908935426" observedRunningTime="2026-05-25 12:13:38.094772338 +0000 UTC m=+21.810587620" watchObservedRunningTime="2026-05-25 12:13:38.095564741 +0000 UTC m=+21.811380028"
May 25 12:13:38.134565 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:38.134523    2549 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-dns/node-resolver-dv9dt" podStartSLOduration=2.154107028 podStartE2EDuration="21.134512965s" podCreationTimestamp="2026-05-25 12:13:17 +0000 UTC" firstStartedPulling="2026-05-25 12:13:18.150554547 +0000 UTC m=+1.866369812" lastFinishedPulling="2026-05-25 12:13:37.130960469 +0000 UTC m=+20.846775749" observedRunningTime="2026-05-25 12:13:38.134123112 +0000 UTC m=+21.849938399" watchObservedRunningTime="2026-05-25 12:13:38.134512965 +0000 UTC m=+21.850328252"
May 25 12:13:38.164889 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:38.164843    2549 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-53.ec2.internal" podStartSLOduration=21.164823884 podStartE2EDuration="21.164823884s" podCreationTimestamp="2026-05-25 12:13:17 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 12:13:38.164311278 +0000 UTC m=+21.880126566" watchObservedRunningTime="2026-05-25 12:13:38.164823884 +0000 UTC m=+21.880639171"
May 25 12:13:38.185995 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:38.185816    2549 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-xp8d2" podStartSLOduration=3.102641665 podStartE2EDuration="22.185798947s" podCreationTimestamp="2026-05-25 12:13:16 +0000 UTC" firstStartedPulling="2026-05-25 12:13:18.047318464 +0000 UTC m=+1.763133730" lastFinishedPulling="2026-05-25 12:13:37.130475744 +0000 UTC m=+20.846291012" observedRunningTime="2026-05-25 12:13:38.185606097 +0000 UTC m=+21.901421385" watchObservedRunningTime="2026-05-25 12:13:38.185798947 +0000 UTC m=+21.901614247"
May 25 12:13:38.201190 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:38.201153    2549 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-image-registry/node-ca-bjpxs" podStartSLOduration=2.162314352 podStartE2EDuration="21.201142353s" podCreationTimestamp="2026-05-25 12:13:17 +0000 UTC" firstStartedPulling="2026-05-25 12:13:18.154646274 +0000 UTC m=+1.870461538" lastFinishedPulling="2026-05-25 12:13:37.19347426 +0000 UTC m=+20.909289539" observedRunningTime="2026-05-25 12:13:38.200953732 +0000 UTC m=+21.916769020" watchObservedRunningTime="2026-05-25 12:13:38.201142353 +0000 UTC m=+21.916957688"
May 25 12:13:38.714288 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:38.714262    2549 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
May 25 12:13:38.786163 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:38.786065    2549 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-05-25T12:13:38.714284588Z","UUID":"2510ea24-95d4-4cd7-ad31-c7fccb9515f6","Handler":null,"Name":"","Endpoint":""}
May 25 12:13:38.789248 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:38.789224    2549 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
May 25 12:13:38.789396 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:38.789258    2549 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
May 25 12:13:38.846535 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:38.846507    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tp68k"
May 25 12:13:38.846708 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:13:38.846614    2549 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tp68k" podUID="d37b4fc9-9e8b-4e07-9672-b6bffc0f46d8"
May 25 12:13:39.076894 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:39.076811    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l8w7f" event={"ID":"7bb34bd7-c02c-42cb-b3c8-643fd893a7db","Type":"ContainerStarted","Data":"ec86c347e6e9e83f1df8a1eeb99703bc81bbc117b5677bc22c0c0810fcf87e8d"}
May 25 12:13:39.078901 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:39.078869    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-53.ec2.internal" event={"ID":"fa968fd9574051417c15086d0b1771c9","Type":"ContainerStarted","Data":"df99742e6628be9d6f4e8bb9edf7ea5626f4846a84d9d8c3a7610671e0451b7e"}
May 25 12:13:39.080774 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:39.080688    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-g4949" event={"ID":"69c409ea-f028-429e-8a53-0f25010483bd","Type":"ContainerStarted","Data":"dc13c03e7c03d739b32b073b494fe2c45cb91de63249d8acafd7b41d9675497f"}
May 25 12:13:39.098721 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:39.098671    2549 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-53.ec2.internal" podStartSLOduration=22.098653569 podStartE2EDuration="22.098653569s" podCreationTimestamp="2026-05-25 12:13:17 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 12:13:39.098437781 +0000 UTC m=+22.814253119" watchObservedRunningTime="2026-05-25 12:13:39.098653569 +0000 UTC m=+22.814468858"
May 25 12:13:39.114035 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:39.113985    2549 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-g4949" podStartSLOduration=4.029830266 podStartE2EDuration="23.113969853s" podCreationTimestamp="2026-05-25 12:13:16 +0000 UTC" firstStartedPulling="2026-05-25 12:13:18.113207409 +0000 UTC m=+1.829022674" lastFinishedPulling="2026-05-25 12:13:37.197346982 +0000 UTC m=+20.913162261" observedRunningTime="2026-05-25 12:13:39.11383424 +0000 UTC m=+22.829649529" watchObservedRunningTime="2026-05-25 12:13:39.113969853 +0000 UTC m=+22.829785214"
May 25 12:13:39.843153 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:39.843067    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zf62d"
May 25 12:13:39.843322 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:13:39.843189    2549 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zf62d" podUID="1124f1f1-fdc7-4a81-b28a-620db95e2ff6"
May 25 12:13:40.088987 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:40.088937    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-79j85" event={"ID":"0b693280-ce91-46e7-b8c8-37d81a9b8fe1","Type":"ContainerStarted","Data":"f151d7b373644bc70d11dada81c4a833be6044cb8a33a05b2abbab2c91638345"}
May 25 12:13:40.091090 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:40.091061    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l8w7f" event={"ID":"7bb34bd7-c02c-42cb-b3c8-643fd893a7db","Type":"ContainerStarted","Data":"55da2548577165988c416c1a34620c90b1bf55907a1976f4fcee4ed2e3018c38"}
May 25 12:13:40.111239 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:40.111154    2549 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l8w7f" podStartSLOduration=1.77321333 podStartE2EDuration="23.11113819s" podCreationTimestamp="2026-05-25 12:13:17 +0000 UTC" firstStartedPulling="2026-05-25 12:13:18.127795887 +0000 UTC m=+1.843611152" lastFinishedPulling="2026-05-25 12:13:39.465720736 +0000 UTC m=+23.181536012" observedRunningTime="2026-05-25 12:13:40.110662171 +0000 UTC m=+23.826477458" watchObservedRunningTime="2026-05-25 12:13:40.11113819 +0000 UTC m=+23.826953479"
May 25 12:13:40.845997 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:40.845961    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tp68k"
May 25 12:13:40.846169 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:13:40.846078    2549 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tp68k" podUID="d37b4fc9-9e8b-4e07-9672-b6bffc0f46d8"
May 25 12:13:41.432617 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:41.432580    2549 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-tcp4j"
May 25 12:13:41.433474 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:41.433449    2549 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-tcp4j"
May 25 12:13:41.843269 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:41.843182    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zf62d"
May 25 12:13:41.843423 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:13:41.843332    2549 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zf62d" podUID="1124f1f1-fdc7-4a81-b28a-620db95e2ff6"
May 25 12:13:42.095458 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:42.095398    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-tcp4j"
May 25 12:13:42.095847 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:42.095826    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-tcp4j"
May 25 12:13:42.843091 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:42.842876    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tp68k"
May 25 12:13:42.843671 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:13:42.843143    2549 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tp68k" podUID="d37b4fc9-9e8b-4e07-9672-b6bffc0f46d8"
May 25 12:13:43.098468 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:43.098394    2549 generic.go:358] "Generic (PLEG): container finished" podID="7153f197-fa4f-4da4-ba82-73f29e4dd2ac" containerID="b882613ab57faf869968a802e310d966047d034c0016c4dd3d4b5a65c338219a" exitCode=0
May 25 12:13:43.098625 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:43.098474    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-h49mm" event={"ID":"7153f197-fa4f-4da4-ba82-73f29e4dd2ac","Type":"ContainerDied","Data":"b882613ab57faf869968a802e310d966047d034c0016c4dd3d4b5a65c338219a"}
May 25 12:13:43.101590 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:43.101550    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-79j85" event={"ID":"0b693280-ce91-46e7-b8c8-37d81a9b8fe1","Type":"ContainerStarted","Data":"0ce409eea5a6a0c073e3aab037289fbad60ab01bd9bc0fdc71a78b2337f82d1f"}
May 25 12:13:43.101843 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:43.101826    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-79j85"
May 25 12:13:43.101909 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:43.101856    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-79j85"
May 25 12:13:43.117016 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:43.116985    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-79j85"
May 25 12:13:43.155829 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:43.155779    2549 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-79j85" podStartSLOduration=7.965956551 podStartE2EDuration="27.155767789s" podCreationTimestamp="2026-05-25 12:13:16 +0000 UTC" firstStartedPulling="2026-05-25 12:13:18.073658993 +0000 UTC m=+1.789474258" lastFinishedPulling="2026-05-25 12:13:37.26347023 +0000 UTC m=+20.979285496" observedRunningTime="2026-05-25 12:13:43.155432762 +0000 UTC m=+26.871248050" watchObservedRunningTime="2026-05-25 12:13:43.155767789 +0000 UTC m=+26.871583076"
May 25 12:13:43.843018 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:43.842989    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zf62d"
May 25 12:13:43.843168 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:13:43.843142    2549 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zf62d" podUID="1124f1f1-fdc7-4a81-b28a-620db95e2ff6"
May 25 12:13:44.100982 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:44.100709    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-tp68k"]
May 25 12:13:44.101116 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:44.101007    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tp68k"
May 25 12:13:44.101116 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:13:44.101085    2549 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tp68k" podUID="d37b4fc9-9e8b-4e07-9672-b6bffc0f46d8"
May 25 12:13:44.103245 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:44.103190    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-zf62d"]
May 25 12:13:44.105922 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:44.105896    2549 generic.go:358] "Generic (PLEG): container finished" podID="7153f197-fa4f-4da4-ba82-73f29e4dd2ac" containerID="5cbde016d7a4a0b5d7cc31b6b5437e109d11b487d9b6e453d8dccdcb2c425b23" exitCode=0
May 25 12:13:44.106050 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:44.105971    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-h49mm" event={"ID":"7153f197-fa4f-4da4-ba82-73f29e4dd2ac","Type":"ContainerDied","Data":"5cbde016d7a4a0b5d7cc31b6b5437e109d11b487d9b6e453d8dccdcb2c425b23"}
May 25 12:13:44.106191 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:44.106158    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zf62d"
May 25 12:13:44.106361 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:13:44.106294    2549 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zf62d" podUID="1124f1f1-fdc7-4a81-b28a-620db95e2ff6"
May 25 12:13:44.106587 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:44.106568    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-79j85"
May 25 12:13:44.124336 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:44.124317    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-79j85"
May 25 12:13:45.109330 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:45.109247    2549 generic.go:358] "Generic (PLEG): container finished" podID="7153f197-fa4f-4da4-ba82-73f29e4dd2ac" containerID="0834d6bbcf8a5c01a07bb2600c19140cb2039ab8dd822455112d6381eb435d9c" exitCode=0
May 25 12:13:45.109715 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:45.109322    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-h49mm" event={"ID":"7153f197-fa4f-4da4-ba82-73f29e4dd2ac","Type":"ContainerDied","Data":"0834d6bbcf8a5c01a07bb2600c19140cb2039ab8dd822455112d6381eb435d9c"}
May 25 12:13:45.842928 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:45.842896    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tp68k"
May 25 12:13:45.843115 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:45.842897    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zf62d"
May 25 12:13:45.843115 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:13:45.843004    2549 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tp68k" podUID="d37b4fc9-9e8b-4e07-9672-b6bffc0f46d8"
May 25 12:13:45.843115 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:13:45.843104    2549 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zf62d" podUID="1124f1f1-fdc7-4a81-b28a-620db95e2ff6"
May 25 12:13:47.842732 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:47.842697    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tp68k"
May 25 12:13:47.842732 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:47.842719    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zf62d"
May 25 12:13:47.843318 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:13:47.842823    2549 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tp68k" podUID="d37b4fc9-9e8b-4e07-9672-b6bffc0f46d8"
May 25 12:13:47.843318 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:13:47.842943    2549 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zf62d" podUID="1124f1f1-fdc7-4a81-b28a-620db95e2ff6"
May 25 12:13:49.459354 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:49.459314    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1124f1f1-fdc7-4a81-b28a-620db95e2ff6-metrics-certs\") pod \"network-metrics-daemon-zf62d\" (UID: \"1124f1f1-fdc7-4a81-b28a-620db95e2ff6\") " pod="openshift-multus/network-metrics-daemon-zf62d"
May 25 12:13:49.459790 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:13:49.459464    2549 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 12:13:49.459790 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:13:49.459539    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1124f1f1-fdc7-4a81-b28a-620db95e2ff6-metrics-certs podName:1124f1f1-fdc7-4a81-b28a-620db95e2ff6 nodeName:}" failed. No retries permitted until 2026-05-25 12:14:21.459517305 +0000 UTC m=+65.175332571 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1124f1f1-fdc7-4a81-b28a-620db95e2ff6-metrics-certs") pod "network-metrics-daemon-zf62d" (UID: "1124f1f1-fdc7-4a81-b28a-620db95e2ff6") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 12:13:49.660272 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:49.660231    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-56c2b\" (UniqueName: \"kubernetes.io/projected/d37b4fc9-9e8b-4e07-9672-b6bffc0f46d8-kube-api-access-56c2b\") pod \"network-check-target-tp68k\" (UID: \"d37b4fc9-9e8b-4e07-9672-b6bffc0f46d8\") " pod="openshift-network-diagnostics/network-check-target-tp68k"
May 25 12:13:49.660448 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:13:49.660379    2549 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 25 12:13:49.660448 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:13:49.660404    2549 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 25 12:13:49.660448 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:13:49.660418    2549 projected.go:194] Error preparing data for projected volume kube-api-access-56c2b for pod openshift-network-diagnostics/network-check-target-tp68k: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 12:13:49.660582 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:13:49.660481    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d37b4fc9-9e8b-4e07-9672-b6bffc0f46d8-kube-api-access-56c2b podName:d37b4fc9-9e8b-4e07-9672-b6bffc0f46d8 nodeName:}" failed. No retries permitted until 2026-05-25 12:14:21.660462342 +0000 UTC m=+65.376277611 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-56c2b" (UniqueName: "kubernetes.io/projected/d37b4fc9-9e8b-4e07-9672-b6bffc0f46d8-kube-api-access-56c2b") pod "network-check-target-tp68k" (UID: "d37b4fc9-9e8b-4e07-9672-b6bffc0f46d8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 12:13:49.842914 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:49.842837    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tp68k"
May 25 12:13:49.843106 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:49.842853    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zf62d"
May 25 12:13:49.843106 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:13:49.842963    2549 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tp68k" podUID="d37b4fc9-9e8b-4e07-9672-b6bffc0f46d8"
May 25 12:13:49.843106 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:13:49.843066    2549 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zf62d" podUID="1124f1f1-fdc7-4a81-b28a-620db95e2ff6"
May 25 12:13:51.092796 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:51.092608    2549 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-142-53.ec2.internal" event="NodeReady"
May 25 12:13:51.093305 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:51.092893    2549 kubelet_node_status.go:550] "Fast updating node status as it just became ready"
May 25 12:13:51.123475 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:51.123437    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-h49mm" event={"ID":"7153f197-fa4f-4da4-ba82-73f29e4dd2ac","Type":"ContainerStarted","Data":"244add402581e77f9f47dd9ea77c87649c50cb703d52e8f07bedd7a6bda04b08"}
May 25 12:13:51.148229 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:51.148179    2549 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-ml56g"]
May 25 12:13:51.166029 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:51.165994    2549 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-fxlzt"]
May 25 12:13:51.166183 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:51.166045    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-ml56g"
May 25 12:13:51.168865 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:51.168838    2549 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
May 25 12:13:51.169059 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:51.169006    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
May 25 12:13:51.169190 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:51.169153    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-c8dc6\""
May 25 12:13:51.188389 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:51.188362    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-ml56g"]
May 25 12:13:51.188507 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:51.188490    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-fxlzt"]
May 25 12:13:51.188572 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:51.188509    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-fxlzt"
May 25 12:13:51.191281 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:51.191191    2549 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
May 25 12:13:51.191281 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:51.191270    2549 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
May 25 12:13:51.191450 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:51.191213    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
May 25 12:13:51.191450 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:51.191380    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-sw9bq\""
May 25 12:13:51.273965 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:51.273932    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0ea389da-cc7e-4b86-846d-e95f5a2dae7c-cert\") pod \"ingress-canary-fxlzt\" (UID: \"0ea389da-cc7e-4b86-846d-e95f5a2dae7c\") " pod="openshift-ingress-canary/ingress-canary-fxlzt"
May 25 12:13:51.274110 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:51.273997    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krw2l\" (UniqueName: \"kubernetes.io/projected/acccf0fc-6d0f-4793-acb0-69227950e520-kube-api-access-krw2l\") pod \"dns-default-ml56g\" (UID: \"acccf0fc-6d0f-4793-acb0-69227950e520\") " pod="openshift-dns/dns-default-ml56g"
May 25 12:13:51.274110 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:51.274021    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2b5k\" (UniqueName: \"kubernetes.io/projected/0ea389da-cc7e-4b86-846d-e95f5a2dae7c-kube-api-access-c2b5k\") pod \"ingress-canary-fxlzt\" (UID: \"0ea389da-cc7e-4b86-846d-e95f5a2dae7c\") " pod="openshift-ingress-canary/ingress-canary-fxlzt"
May 25 12:13:51.274110 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:51.274086    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/acccf0fc-6d0f-4793-acb0-69227950e520-config-volume\") pod \"dns-default-ml56g\" (UID: \"acccf0fc-6d0f-4793-acb0-69227950e520\") " pod="openshift-dns/dns-default-ml56g"
May 25 12:13:51.274284 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:51.274132    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/acccf0fc-6d0f-4793-acb0-69227950e520-metrics-tls\") pod \"dns-default-ml56g\" (UID: \"acccf0fc-6d0f-4793-acb0-69227950e520\") " pod="openshift-dns/dns-default-ml56g"
May 25 12:13:51.274284 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:51.274155    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/acccf0fc-6d0f-4793-acb0-69227950e520-tmp-dir\") pod \"dns-default-ml56g\" (UID: \"acccf0fc-6d0f-4793-acb0-69227950e520\") " pod="openshift-dns/dns-default-ml56g"
May 25 12:13:51.374900 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:51.374862    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/acccf0fc-6d0f-4793-acb0-69227950e520-config-volume\") pod \"dns-default-ml56g\" (UID: \"acccf0fc-6d0f-4793-acb0-69227950e520\") " pod="openshift-dns/dns-default-ml56g"
May 25 12:13:51.375101 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:51.374924    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/acccf0fc-6d0f-4793-acb0-69227950e520-metrics-tls\") pod \"dns-default-ml56g\" (UID: \"acccf0fc-6d0f-4793-acb0-69227950e520\") " pod="openshift-dns/dns-default-ml56g"
May 25 12:13:51.375101 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:51.374941    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/acccf0fc-6d0f-4793-acb0-69227950e520-tmp-dir\") pod \"dns-default-ml56g\" (UID: \"acccf0fc-6d0f-4793-acb0-69227950e520\") " pod="openshift-dns/dns-default-ml56g"
May 25 12:13:51.375101 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:51.374964    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0ea389da-cc7e-4b86-846d-e95f5a2dae7c-cert\") pod \"ingress-canary-fxlzt\" (UID: \"0ea389da-cc7e-4b86-846d-e95f5a2dae7c\") " pod="openshift-ingress-canary/ingress-canary-fxlzt"
May 25 12:13:51.375101 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:51.374995    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-krw2l\" (UniqueName: \"kubernetes.io/projected/acccf0fc-6d0f-4793-acb0-69227950e520-kube-api-access-krw2l\") pod \"dns-default-ml56g\" (UID: \"acccf0fc-6d0f-4793-acb0-69227950e520\") " pod="openshift-dns/dns-default-ml56g"
May 25 12:13:51.375101 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:51.375011    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-c2b5k\" (UniqueName: \"kubernetes.io/projected/0ea389da-cc7e-4b86-846d-e95f5a2dae7c-kube-api-access-c2b5k\") pod \"ingress-canary-fxlzt\" (UID: \"0ea389da-cc7e-4b86-846d-e95f5a2dae7c\") " pod="openshift-ingress-canary/ingress-canary-fxlzt"
May 25 12:13:51.375411 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:13:51.375109    2549 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 25 12:13:51.375411 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:13:51.375184    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0ea389da-cc7e-4b86-846d-e95f5a2dae7c-cert podName:0ea389da-cc7e-4b86-846d-e95f5a2dae7c nodeName:}" failed. No retries permitted until 2026-05-25 12:13:51.875163654 +0000 UTC m=+35.590978919 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/0ea389da-cc7e-4b86-846d-e95f5a2dae7c-cert") pod "ingress-canary-fxlzt" (UID: "0ea389da-cc7e-4b86-846d-e95f5a2dae7c") : secret "canary-serving-cert" not found
May 25 12:13:51.375411 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:13:51.375109    2549 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 25 12:13:51.375411 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:13:51.375273    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/acccf0fc-6d0f-4793-acb0-69227950e520-metrics-tls podName:acccf0fc-6d0f-4793-acb0-69227950e520 nodeName:}" failed. No retries permitted until 2026-05-25 12:13:51.875254432 +0000 UTC m=+35.591069712 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/acccf0fc-6d0f-4793-acb0-69227950e520-metrics-tls") pod "dns-default-ml56g" (UID: "acccf0fc-6d0f-4793-acb0-69227950e520") : secret "dns-default-metrics-tls" not found
May 25 12:13:51.375564 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:51.375410    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/acccf0fc-6d0f-4793-acb0-69227950e520-tmp-dir\") pod \"dns-default-ml56g\" (UID: \"acccf0fc-6d0f-4793-acb0-69227950e520\") " pod="openshift-dns/dns-default-ml56g"
May 25 12:13:51.375564 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:51.375529    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/acccf0fc-6d0f-4793-acb0-69227950e520-config-volume\") pod \"dns-default-ml56g\" (UID: \"acccf0fc-6d0f-4793-acb0-69227950e520\") " pod="openshift-dns/dns-default-ml56g"
May 25 12:13:51.388401 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:51.388374    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-krw2l\" (UniqueName: \"kubernetes.io/projected/acccf0fc-6d0f-4793-acb0-69227950e520-kube-api-access-krw2l\") pod \"dns-default-ml56g\" (UID: \"acccf0fc-6d0f-4793-acb0-69227950e520\") " pod="openshift-dns/dns-default-ml56g"
May 25 12:13:51.388572 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:51.388552    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2b5k\" (UniqueName: \"kubernetes.io/projected/0ea389da-cc7e-4b86-846d-e95f5a2dae7c-kube-api-access-c2b5k\") pod \"ingress-canary-fxlzt\" (UID: \"0ea389da-cc7e-4b86-846d-e95f5a2dae7c\") " pod="openshift-ingress-canary/ingress-canary-fxlzt"
May 25 12:13:51.842682 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:51.842645    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tp68k"
May 25 12:13:51.842863 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:51.842645    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zf62d"
May 25 12:13:51.845585 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:51.845564    2549 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
May 25 12:13:51.846689 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:51.846666    2549 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
May 25 12:13:51.846689 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:51.846676    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-nmk2h\""
May 25 12:13:51.846861 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:51.846680    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-rrs9z\""
May 25 12:13:51.846861 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:51.846676    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
May 25 12:13:51.879283 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:51.879249    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0ea389da-cc7e-4b86-846d-e95f5a2dae7c-cert\") pod \"ingress-canary-fxlzt\" (UID: \"0ea389da-cc7e-4b86-846d-e95f5a2dae7c\") " pod="openshift-ingress-canary/ingress-canary-fxlzt"
May 25 12:13:51.879466 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:51.879315    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/acccf0fc-6d0f-4793-acb0-69227950e520-metrics-tls\") pod \"dns-default-ml56g\" (UID: \"acccf0fc-6d0f-4793-acb0-69227950e520\") " pod="openshift-dns/dns-default-ml56g"
May 25 12:13:51.879466 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:13:51.879401    2549 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 25 12:13:51.879466 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:13:51.879410    2549 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 25 12:13:51.879466 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:13:51.879466    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/acccf0fc-6d0f-4793-acb0-69227950e520-metrics-tls podName:acccf0fc-6d0f-4793-acb0-69227950e520 nodeName:}" failed. No retries permitted until 2026-05-25 12:13:52.879452449 +0000 UTC m=+36.595267727 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/acccf0fc-6d0f-4793-acb0-69227950e520-metrics-tls") pod "dns-default-ml56g" (UID: "acccf0fc-6d0f-4793-acb0-69227950e520") : secret "dns-default-metrics-tls" not found
May 25 12:13:51.879618 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:13:51.879479    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0ea389da-cc7e-4b86-846d-e95f5a2dae7c-cert podName:0ea389da-cc7e-4b86-846d-e95f5a2dae7c nodeName:}" failed. No retries permitted until 2026-05-25 12:13:52.87947329 +0000 UTC m=+36.595288555 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/0ea389da-cc7e-4b86-846d-e95f5a2dae7c-cert") pod "ingress-canary-fxlzt" (UID: "0ea389da-cc7e-4b86-846d-e95f5a2dae7c") : secret "canary-serving-cert" not found
May 25 12:13:52.128257 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:52.128162    2549 generic.go:358] "Generic (PLEG): container finished" podID="7153f197-fa4f-4da4-ba82-73f29e4dd2ac" containerID="244add402581e77f9f47dd9ea77c87649c50cb703d52e8f07bedd7a6bda04b08" exitCode=0
May 25 12:13:52.128683 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:52.128249    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-h49mm" event={"ID":"7153f197-fa4f-4da4-ba82-73f29e4dd2ac","Type":"ContainerDied","Data":"244add402581e77f9f47dd9ea77c87649c50cb703d52e8f07bedd7a6bda04b08"}
May 25 12:13:52.885365 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:52.885332    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/acccf0fc-6d0f-4793-acb0-69227950e520-metrics-tls\") pod \"dns-default-ml56g\" (UID: \"acccf0fc-6d0f-4793-acb0-69227950e520\") " pod="openshift-dns/dns-default-ml56g"
May 25 12:13:52.885503 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:52.885373    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0ea389da-cc7e-4b86-846d-e95f5a2dae7c-cert\") pod \"ingress-canary-fxlzt\" (UID: \"0ea389da-cc7e-4b86-846d-e95f5a2dae7c\") " pod="openshift-ingress-canary/ingress-canary-fxlzt"
May 25 12:13:52.885503 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:13:52.885467    2549 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 25 12:13:52.885597 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:13:52.885516    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0ea389da-cc7e-4b86-846d-e95f5a2dae7c-cert podName:0ea389da-cc7e-4b86-846d-e95f5a2dae7c nodeName:}" failed. No retries permitted until 2026-05-25 12:13:54.885504056 +0000 UTC m=+38.601319335 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/0ea389da-cc7e-4b86-846d-e95f5a2dae7c-cert") pod "ingress-canary-fxlzt" (UID: "0ea389da-cc7e-4b86-846d-e95f5a2dae7c") : secret "canary-serving-cert" not found
May 25 12:13:52.885597 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:13:52.885467    2549 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 25 12:13:52.885597 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:13:52.885579    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/acccf0fc-6d0f-4793-acb0-69227950e520-metrics-tls podName:acccf0fc-6d0f-4793-acb0-69227950e520 nodeName:}" failed. No retries permitted until 2026-05-25 12:13:54.885567831 +0000 UTC m=+38.601383108 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/acccf0fc-6d0f-4793-acb0-69227950e520-metrics-tls") pod "dns-default-ml56g" (UID: "acccf0fc-6d0f-4793-acb0-69227950e520") : secret "dns-default-metrics-tls" not found
May 25 12:13:53.132381 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:53.132348    2549 generic.go:358] "Generic (PLEG): container finished" podID="7153f197-fa4f-4da4-ba82-73f29e4dd2ac" containerID="e2100b2363ea830b97e74a95dde5e8d689ea9b978be6e9a0051c9d684ccd1fcc" exitCode=0
May 25 12:13:53.132922 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:53.132390    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-h49mm" event={"ID":"7153f197-fa4f-4da4-ba82-73f29e4dd2ac","Type":"ContainerDied","Data":"e2100b2363ea830b97e74a95dde5e8d689ea9b978be6e9a0051c9d684ccd1fcc"}
May 25 12:13:54.137189 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:54.137145    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-h49mm" event={"ID":"7153f197-fa4f-4da4-ba82-73f29e4dd2ac","Type":"ContainerStarted","Data":"1e93a2b0b17f8db56edf62223443341b18118ec617f4a37ff15eaca235a75c85"}
May 25 12:13:54.164776 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:54.164732    2549 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-h49mm" podStartSLOduration=5.331698232 podStartE2EDuration="38.164717223s" podCreationTimestamp="2026-05-25 12:13:16 +0000 UTC" firstStartedPulling="2026-05-25 12:13:18.099322249 +0000 UTC m=+1.815137514" lastFinishedPulling="2026-05-25 12:13:50.932341241 +0000 UTC m=+34.648156505" observedRunningTime="2026-05-25 12:13:54.162988579 +0000 UTC m=+37.878803878" watchObservedRunningTime="2026-05-25 12:13:54.164717223 +0000 UTC m=+37.880532510"
May 25 12:13:54.901164 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:54.901133    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/acccf0fc-6d0f-4793-acb0-69227950e520-metrics-tls\") pod \"dns-default-ml56g\" (UID: \"acccf0fc-6d0f-4793-acb0-69227950e520\") " pod="openshift-dns/dns-default-ml56g"
May 25 12:13:54.901164 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:54.901171    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0ea389da-cc7e-4b86-846d-e95f5a2dae7c-cert\") pod \"ingress-canary-fxlzt\" (UID: \"0ea389da-cc7e-4b86-846d-e95f5a2dae7c\") " pod="openshift-ingress-canary/ingress-canary-fxlzt"
May 25 12:13:54.901429 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:13:54.901312    2549 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 25 12:13:54.901429 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:13:54.901377    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/acccf0fc-6d0f-4793-acb0-69227950e520-metrics-tls podName:acccf0fc-6d0f-4793-acb0-69227950e520 nodeName:}" failed. No retries permitted until 2026-05-25 12:13:58.901358582 +0000 UTC m=+42.617173849 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/acccf0fc-6d0f-4793-acb0-69227950e520-metrics-tls") pod "dns-default-ml56g" (UID: "acccf0fc-6d0f-4793-acb0-69227950e520") : secret "dns-default-metrics-tls" not found
May 25 12:13:54.901429 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:13:54.901391    2549 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 25 12:13:54.901580 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:13:54.901436    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0ea389da-cc7e-4b86-846d-e95f5a2dae7c-cert podName:0ea389da-cc7e-4b86-846d-e95f5a2dae7c nodeName:}" failed. No retries permitted until 2026-05-25 12:13:58.901422086 +0000 UTC m=+42.617237351 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/0ea389da-cc7e-4b86-846d-e95f5a2dae7c-cert") pod "ingress-canary-fxlzt" (UID: "0ea389da-cc7e-4b86-846d-e95f5a2dae7c") : secret "canary-serving-cert" not found
May 25 12:13:58.929511 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:58.929469    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0ea389da-cc7e-4b86-846d-e95f5a2dae7c-cert\") pod \"ingress-canary-fxlzt\" (UID: \"0ea389da-cc7e-4b86-846d-e95f5a2dae7c\") " pod="openshift-ingress-canary/ingress-canary-fxlzt"
May 25 12:13:58.929937 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:13:58.929566    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/acccf0fc-6d0f-4793-acb0-69227950e520-metrics-tls\") pod \"dns-default-ml56g\" (UID: \"acccf0fc-6d0f-4793-acb0-69227950e520\") " pod="openshift-dns/dns-default-ml56g"
May 25 12:13:58.929937 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:13:58.929618    2549 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 25 12:13:58.929937 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:13:58.929681    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0ea389da-cc7e-4b86-846d-e95f5a2dae7c-cert podName:0ea389da-cc7e-4b86-846d-e95f5a2dae7c nodeName:}" failed. No retries permitted until 2026-05-25 12:14:06.929665703 +0000 UTC m=+50.645480967 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/0ea389da-cc7e-4b86-846d-e95f5a2dae7c-cert") pod "ingress-canary-fxlzt" (UID: "0ea389da-cc7e-4b86-846d-e95f5a2dae7c") : secret "canary-serving-cert" not found
May 25 12:13:58.929937 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:13:58.929681    2549 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 25 12:13:58.929937 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:13:58.929733    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/acccf0fc-6d0f-4793-acb0-69227950e520-metrics-tls podName:acccf0fc-6d0f-4793-acb0-69227950e520 nodeName:}" failed. No retries permitted until 2026-05-25 12:14:06.929717552 +0000 UTC m=+50.645532836 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/acccf0fc-6d0f-4793-acb0-69227950e520-metrics-tls") pod "dns-default-ml56g" (UID: "acccf0fc-6d0f-4793-acb0-69227950e520") : secret "dns-default-metrics-tls" not found
May 25 12:14:06.983655 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:14:06.983614    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/acccf0fc-6d0f-4793-acb0-69227950e520-metrics-tls\") pod \"dns-default-ml56g\" (UID: \"acccf0fc-6d0f-4793-acb0-69227950e520\") " pod="openshift-dns/dns-default-ml56g"
May 25 12:14:06.983655 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:14:06.983660    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0ea389da-cc7e-4b86-846d-e95f5a2dae7c-cert\") pod \"ingress-canary-fxlzt\" (UID: \"0ea389da-cc7e-4b86-846d-e95f5a2dae7c\") " pod="openshift-ingress-canary/ingress-canary-fxlzt"
May 25 12:14:06.984268 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:14:06.983750    2549 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 25 12:14:06.984268 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:14:06.983767    2549 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 25 12:14:06.984268 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:14:06.983817    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0ea389da-cc7e-4b86-846d-e95f5a2dae7c-cert podName:0ea389da-cc7e-4b86-846d-e95f5a2dae7c nodeName:}" failed. No retries permitted until 2026-05-25 12:14:22.983799481 +0000 UTC m=+66.699614746 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/0ea389da-cc7e-4b86-846d-e95f5a2dae7c-cert") pod "ingress-canary-fxlzt" (UID: "0ea389da-cc7e-4b86-846d-e95f5a2dae7c") : secret "canary-serving-cert" not found
May 25 12:14:06.984268 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:14:06.983831    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/acccf0fc-6d0f-4793-acb0-69227950e520-metrics-tls podName:acccf0fc-6d0f-4793-acb0-69227950e520 nodeName:}" failed. No retries permitted until 2026-05-25 12:14:22.983825237 +0000 UTC m=+66.699640502 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/acccf0fc-6d0f-4793-acb0-69227950e520-metrics-tls") pod "dns-default-ml56g" (UID: "acccf0fc-6d0f-4793-acb0-69227950e520") : secret "dns-default-metrics-tls" not found
May 25 12:14:16.121885 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:14:16.121854    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-79j85"
May 25 12:14:21.481435 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:14:21.481394    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1124f1f1-fdc7-4a81-b28a-620db95e2ff6-metrics-certs\") pod \"network-metrics-daemon-zf62d\" (UID: \"1124f1f1-fdc7-4a81-b28a-620db95e2ff6\") " pod="openshift-multus/network-metrics-daemon-zf62d"
May 25 12:14:21.484406 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:14:21.484386    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
May 25 12:14:21.492492 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:14:21.492472    2549 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
May 25 12:14:21.492575 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:14:21.492552    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1124f1f1-fdc7-4a81-b28a-620db95e2ff6-metrics-certs podName:1124f1f1-fdc7-4a81-b28a-620db95e2ff6 nodeName:}" failed. No retries permitted until 2026-05-25 12:15:25.492531111 +0000 UTC m=+129.208346391 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1124f1f1-fdc7-4a81-b28a-620db95e2ff6-metrics-certs") pod "network-metrics-daemon-zf62d" (UID: "1124f1f1-fdc7-4a81-b28a-620db95e2ff6") : secret "metrics-daemon-secret" not found
May 25 12:14:21.682909 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:14:21.682873    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-56c2b\" (UniqueName: \"kubernetes.io/projected/d37b4fc9-9e8b-4e07-9672-b6bffc0f46d8-kube-api-access-56c2b\") pod \"network-check-target-tp68k\" (UID: \"d37b4fc9-9e8b-4e07-9672-b6bffc0f46d8\") " pod="openshift-network-diagnostics/network-check-target-tp68k"
May 25 12:14:21.685953 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:14:21.685923    2549 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
May 25 12:14:21.695894 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:14:21.695871    2549 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
May 25 12:14:21.705692 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:14:21.705675    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-56c2b\" (UniqueName: \"kubernetes.io/projected/d37b4fc9-9e8b-4e07-9672-b6bffc0f46d8-kube-api-access-56c2b\") pod \"network-check-target-tp68k\" (UID: \"d37b4fc9-9e8b-4e07-9672-b6bffc0f46d8\") " pod="openshift-network-diagnostics/network-check-target-tp68k"
May 25 12:14:21.854776 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:14:21.854747    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-nmk2h\""
May 25 12:14:21.863121 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:14:21.863103    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tp68k"
May 25 12:14:22.048511 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:14:22.048479    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-tp68k"]
May 25 12:14:22.052080 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:14:22.052055    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd37b4fc9_9e8b_4e07_9672_b6bffc0f46d8.slice/crio-a2351a758852ed98b7b39f869d4b0c4a195d433471aa8d3203a61eb6b10c68e7 WatchSource:0}: Error finding container a2351a758852ed98b7b39f869d4b0c4a195d433471aa8d3203a61eb6b10c68e7: Status 404 returned error can't find the container with id a2351a758852ed98b7b39f869d4b0c4a195d433471aa8d3203a61eb6b10c68e7
May 25 12:14:22.187635 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:14:22.187552    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-tp68k" event={"ID":"d37b4fc9-9e8b-4e07-9672-b6bffc0f46d8","Type":"ContainerStarted","Data":"a2351a758852ed98b7b39f869d4b0c4a195d433471aa8d3203a61eb6b10c68e7"}
May 25 12:14:22.992114 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:14:22.992071    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/acccf0fc-6d0f-4793-acb0-69227950e520-metrics-tls\") pod \"dns-default-ml56g\" (UID: \"acccf0fc-6d0f-4793-acb0-69227950e520\") " pod="openshift-dns/dns-default-ml56g"
May 25 12:14:22.992621 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:14:22.992134    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0ea389da-cc7e-4b86-846d-e95f5a2dae7c-cert\") pod \"ingress-canary-fxlzt\" (UID: \"0ea389da-cc7e-4b86-846d-e95f5a2dae7c\") " pod="openshift-ingress-canary/ingress-canary-fxlzt"
May 25 12:14:22.992621 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:14:22.992253    2549 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 25 12:14:22.992621 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:14:22.992283    2549 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 25 12:14:22.992621 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:14:22.992328    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/acccf0fc-6d0f-4793-acb0-69227950e520-metrics-tls podName:acccf0fc-6d0f-4793-acb0-69227950e520 nodeName:}" failed. No retries permitted until 2026-05-25 12:14:54.992307566 +0000 UTC m=+98.708122844 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/acccf0fc-6d0f-4793-acb0-69227950e520-metrics-tls") pod "dns-default-ml56g" (UID: "acccf0fc-6d0f-4793-acb0-69227950e520") : secret "dns-default-metrics-tls" not found
May 25 12:14:22.992621 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:14:22.992343    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0ea389da-cc7e-4b86-846d-e95f5a2dae7c-cert podName:0ea389da-cc7e-4b86-846d-e95f5a2dae7c nodeName:}" failed. No retries permitted until 2026-05-25 12:14:54.992337423 +0000 UTC m=+98.708152687 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/0ea389da-cc7e-4b86-846d-e95f5a2dae7c-cert") pod "ingress-canary-fxlzt" (UID: "0ea389da-cc7e-4b86-846d-e95f5a2dae7c") : secret "canary-serving-cert" not found
May 25 12:14:25.194276 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:14:25.194181    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-tp68k" event={"ID":"d37b4fc9-9e8b-4e07-9672-b6bffc0f46d8","Type":"ContainerStarted","Data":"d7bf0132ea7048ea9593b9b5512fa0042c39c83612b482bd8231a57a0c1ab280"}
May 25 12:14:25.194610 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:14:25.194301    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-tp68k"
May 25 12:14:25.214913 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:14:25.214875    2549 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-tp68k" podStartSLOduration=66.409754636 podStartE2EDuration="1m9.214863871s" podCreationTimestamp="2026-05-25 12:13:16 +0000 UTC" firstStartedPulling="2026-05-25 12:14:22.054330111 +0000 UTC m=+65.770145376" lastFinishedPulling="2026-05-25 12:14:24.859439329 +0000 UTC m=+68.575254611" observedRunningTime="2026-05-25 12:14:25.214217952 +0000 UTC m=+68.930033231" watchObservedRunningTime="2026-05-25 12:14:25.214863871 +0000 UTC m=+68.930679158"
May 25 12:14:46.250843 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:14:46.250698    2549 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-697587599-f2jsk"]
May 25 12:14:46.253266 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:14:46.253244    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-f2jsk"
May 25 12:14:46.256126 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:14:46.256107    2549 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
May 25 12:14:46.257280 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:14:46.257263    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-ztzph\""
May 25 12:14:46.257359 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:14:46.257303    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
May 25 12:14:46.257359 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:14:46.257315    2549 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
May 25 12:14:46.262063 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:14:46.262043    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-697587599-f2jsk"]
May 25 12:14:46.332367 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:14:46.332335    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d452639f-6ffb-4b6c-9640-67b753e08fdf-samples-operator-tls\") pod \"cluster-samples-operator-697587599-f2jsk\" (UID: \"d452639f-6ffb-4b6c-9640-67b753e08fdf\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-f2jsk"
May 25 12:14:46.332516 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:14:46.332377    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2sgc\" (UniqueName: \"kubernetes.io/projected/d452639f-6ffb-4b6c-9640-67b753e08fdf-kube-api-access-n2sgc\") pod \"cluster-samples-operator-697587599-f2jsk\" (UID: \"d452639f-6ffb-4b6c-9640-67b753e08fdf\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-f2jsk"
May 25 12:14:46.355052 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:14:46.355018    2549 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-7t27f"]
May 25 12:14:46.356853 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:14:46.356837    2549 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-78bdfb888c-nxz4w"]
May 25 12:14:46.356986 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:14:46.356969    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-7t27f"
May 25 12:14:46.359854 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:14:46.359466    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-78bdfb888c-nxz4w"
May 25 12:14:46.360035 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:14:46.360009    2549 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-5b7954bdbb-7n4rl"]
May 25 12:14:46.361158 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:14:46.361110    2549 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
May 25 12:14:46.361314 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:14:46.361293    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-vncff\""
May 25 12:14:46.361479 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:14:46.361184    2549 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
May 25 12:14:46.361586 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:14:46.361226    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
May 25 12:14:46.361708 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:14:46.361184    2549 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
May 25 12:14:46.362358 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:14:46.362337    2549 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
May 25 12:14:46.362452 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:14:46.362349    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-vk4hv\""
May 25 12:14:46.362735 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:14:46.362715    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-5b7954bdbb-7n4rl"
May 25 12:14:46.364847 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:14:46.364831    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
May 25 12:14:46.365534 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:14:46.365519    2549 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
May 25 12:14:46.365765 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:14:46.365748    2549 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
May 25 12:14:46.366160 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:14:46.366146    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
May 25 12:14:46.366268 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:14:46.366241    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
May 25 12:14:46.367210 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:14:46.367176    2549 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
May 25 12:14:46.367556 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:14:46.367540    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
May 25 12:14:46.367867 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:14:46.367849    2549 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
May 25 12:14:46.367867 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:14:46.367858    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-ltkjm\""
May 25 12:14:46.369259 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:14:46.369238    2549 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
May 25 12:14:46.370118 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:14:46.369799    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-7t27f"]
May 25 12:14:46.371647 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:14:46.371630    2549 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
May 25 12:14:46.374855 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:14:46.374809    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-5b7954bdbb-7n4rl"]
May 25 12:14:46.384888 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:14:46.384872    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-78bdfb888c-nxz4w"]
May 25 12:14:46.433188 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:14:46.433159    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/bf795e4e-8902-4aa8-be41-2671139da816-snapshots\") pod \"insights-operator-78bdfb888c-nxz4w\" (UID: \"bf795e4e-8902-4aa8-be41-2671139da816\") " pod="openshift-insights/insights-operator-78bdfb888c-nxz4w"
May 25 12:14:46.433188 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:14:46.433188    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1dc72bef-ba17-4ede-88bb-0e1a18c43bbf-config\") pod \"kube-storage-version-migrator-operator-565644f49-7t27f\" (UID: \"1dc72bef-ba17-4ede-88bb-0e1a18c43bbf\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-7t27f"
May 25 12:14:46.433404 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:14:46.433229    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1dc72bef-ba17-4ede-88bb-0e1a18c43bbf-serving-cert\") pod \"kube-storage-version-migrator-operator-565644f49-7t27f\" (UID: \"1dc72bef-ba17-4ede-88bb-0e1a18c43bbf\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-7t27f"
May 25 12:14:46.433404 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:14:46.433284    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/bf795e4e-8902-4aa8-be41-2671139da816-serving-cert\") pod \"insights-operator-78bdfb888c-nxz4w\" (UID: \"bf795e4e-8902-4aa8-be41-2671139da816\") " pod="openshift-insights/insights-operator-78bdfb888c-nxz4w"
May 25 12:14:46.433404 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:14:46.433315    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/90cc9eab-ce97-480c-b971-7487792a2833-default-certificate\") pod \"router-default-5b7954bdbb-7n4rl\" (UID: \"90cc9eab-ce97-480c-b971-7487792a2833\") " pod="openshift-ingress/router-default-5b7954bdbb-7n4rl"
May 25 12:14:46.433404 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:14:46.433360    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/90cc9eab-ce97-480c-b971-7487792a2833-service-ca-bundle\") pod \"router-default-5b7954bdbb-7n4rl\" (UID: \"90cc9eab-ce97-480c-b971-7487792a2833\") " pod="openshift-ingress/router-default-5b7954bdbb-7n4rl"
May 25 12:14:46.433602 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:14:46.433409    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bf795e4e-8902-4aa8-be41-2671139da816-tmp\") pod \"insights-operator-78bdfb888c-nxz4w\" (UID: \"bf795e4e-8902-4aa8-be41-2671139da816\") " pod="openshift-insights/insights-operator-78bdfb888c-nxz4w"
May 25 12:14:46.433602 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:14:46.433434    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bf795e4e-8902-4aa8-be41-2671139da816-trusted-ca-bundle\") pod \"insights-operator-78bdfb888c-nxz4w\" (UID: \"bf795e4e-8902-4aa8-be41-2671139da816\") " pod="openshift-insights/insights-operator-78bdfb888c-nxz4w"
May 25 12:14:46.433602 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:14:46.433468    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t784c\" (UniqueName: \"kubernetes.io/projected/1dc72bef-ba17-4ede-88bb-0e1a18c43bbf-kube-api-access-t784c\") pod \"kube-storage-version-migrator-operator-565644f49-7t27f\" (UID: \"1dc72bef-ba17-4ede-88bb-0e1a18c43bbf\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-7t27f"
May 25 12:14:46.433602 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:14:46.433494    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/90cc9eab-ce97-480c-b971-7487792a2833-stats-auth\") pod \"router-default-5b7954bdbb-7n4rl\" (UID: \"90cc9eab-ce97-480c-b971-7487792a2833\") " pod="openshift-ingress/router-default-5b7954bdbb-7n4rl"
May 25 12:14:46.433602 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:14:46.433531    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bf795e4e-8902-4aa8-be41-2671139da816-service-ca-bundle\") pod \"insights-operator-78bdfb888c-nxz4w\" (UID: \"bf795e4e-8902-4aa8-be41-2671139da816\") " pod="openshift-insights/insights-operator-78bdfb888c-nxz4w"
May 25 12:14:46.433602 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:14:46.433571    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ld45d\" (UniqueName: \"kubernetes.io/projected/90cc9eab-ce97-480c-b971-7487792a2833-kube-api-access-ld45d\") pod \"router-default-5b7954bdbb-7n4rl\" (UID: \"90cc9eab-ce97-480c-b971-7487792a2833\") " pod="openshift-ingress/router-default-5b7954bdbb-7n4rl"
May 25 12:14:46.433879 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:14:46.433611    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgnsf\" (UniqueName: \"kubernetes.io/projected/bf795e4e-8902-4aa8-be41-2671139da816-kube-api-access-cgnsf\") pod \"insights-operator-78bdfb888c-nxz4w\" (UID: \"bf795e4e-8902-4aa8-be41-2671139da816\") " pod="openshift-insights/insights-operator-78bdfb888c-nxz4w"
May 25 12:14:46.433879 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:14:46.433644    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90cc9eab-ce97-480c-b971-7487792a2833-metrics-certs\") pod \"router-default-5b7954bdbb-7n4rl\" (UID: \"90cc9eab-ce97-480c-b971-7487792a2833\") " pod="openshift-ingress/router-default-5b7954bdbb-7n4rl"
May 25 12:14:46.433879 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:14:46.433682    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d452639f-6ffb-4b6c-9640-67b753e08fdf-samples-operator-tls\") pod \"cluster-samples-operator-697587599-f2jsk\" (UID: \"d452639f-6ffb-4b6c-9640-67b753e08fdf\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-f2jsk"
May 25 12:14:46.433879 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:14:46.433726    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-n2sgc\" (UniqueName: \"kubernetes.io/projected/d452639f-6ffb-4b6c-9640-67b753e08fdf-kube-api-access-n2sgc\") pod \"cluster-samples-operator-697587599-f2jsk\" (UID: \"d452639f-6ffb-4b6c-9640-67b753e08fdf\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-f2jsk"
May 25 12:14:46.433879 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:14:46.433795    2549 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
May 25 12:14:46.433879 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:14:46.433844    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d452639f-6ffb-4b6c-9640-67b753e08fdf-samples-operator-tls podName:d452639f-6ffb-4b6c-9640-67b753e08fdf nodeName:}" failed. No retries permitted until 2026-05-25 12:14:46.93382914 +0000 UTC m=+90.649644410 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/d452639f-6ffb-4b6c-9640-67b753e08fdf-samples-operator-tls") pod "cluster-samples-operator-697587599-f2jsk" (UID: "d452639f-6ffb-4b6c-9640-67b753e08fdf") : secret "samples-operator-tls" not found
May 25 12:14:46.449209 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:14:46.449177    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2sgc\" (UniqueName: \"kubernetes.io/projected/d452639f-6ffb-4b6c-9640-67b753e08fdf-kube-api-access-n2sgc\") pod \"cluster-samples-operator-697587599-f2jsk\" (UID: \"d452639f-6ffb-4b6c-9640-67b753e08fdf\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-f2jsk"
May 25 12:14:46.534268 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:14:46.534156    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/bf795e4e-8902-4aa8-be41-2671139da816-snapshots\") pod \"insights-operator-78bdfb888c-nxz4w\" (UID: \"bf795e4e-8902-4aa8-be41-2671139da816\") " pod="openshift-insights/insights-operator-78bdfb888c-nxz4w"
May 25 12:14:46.534268 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:14:46.534213    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1dc72bef-ba17-4ede-88bb-0e1a18c43bbf-config\") pod \"kube-storage-version-migrator-operator-565644f49-7t27f\" (UID: \"1dc72bef-ba17-4ede-88bb-0e1a18c43bbf\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-7t27f"
May 25 12:14:46.534268 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:14:46.534247    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1dc72bef-ba17-4ede-88bb-0e1a18c43bbf-serving-cert\") pod \"kube-storage-version-migrator-operator-565644f49-7t27f\" (UID: \"1dc72bef-ba17-4ede-88bb-0e1a18c43bbf\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-7t27f"
May 25 12:14:46.534547 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:14:46.534276    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/bf795e4e-8902-4aa8-be41-2671139da816-serving-cert\") pod \"insights-operator-78bdfb888c-nxz4w\" (UID: \"bf795e4e-8902-4aa8-be41-2671139da816\") " pod="openshift-insights/insights-operator-78bdfb888c-nxz4w"
May 25 12:14:46.534547 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:14:46.534299    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/90cc9eab-ce97-480c-b971-7487792a2833-default-certificate\") pod \"router-default-5b7954bdbb-7n4rl\" (UID: \"90cc9eab-ce97-480c-b971-7487792a2833\") " pod="openshift-ingress/router-default-5b7954bdbb-7n4rl"
May 25 12:14:46.534547 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:14:46.534328    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/90cc9eab-ce97-480c-b971-7487792a2833-service-ca-bundle\") pod \"router-default-5b7954bdbb-7n4rl\" (UID: \"90cc9eab-ce97-480c-b971-7487792a2833\") " pod="openshift-ingress/router-default-5b7954bdbb-7n4rl"
May 25 12:14:46.534547 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:14:46.534361    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bf795e4e-8902-4aa8-be41-2671139da816-tmp\") pod \"insights-operator-78bdfb888c-nxz4w\" (UID: \"bf795e4e-8902-4aa8-be41-2671139da816\") " pod="openshift-insights/insights-operator-78bdfb888c-nxz4w"
May 25 12:14:46.534547 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:14:46.534494    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/90cc9eab-ce97-480c-b971-7487792a2833-service-ca-bundle podName:90cc9eab-ce97-480c-b971-7487792a2833 nodeName:}" failed. No retries permitted until 2026-05-25 12:14:47.034469438 +0000 UTC m=+90.750284726 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/90cc9eab-ce97-480c-b971-7487792a2833-service-ca-bundle") pod "router-default-5b7954bdbb-7n4rl" (UID: "90cc9eab-ce97-480c-b971-7487792a2833") : configmap references non-existent config key: service-ca.crt
May 25 12:14:46.534852 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:14:46.534830    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bf795e4e-8902-4aa8-be41-2671139da816-tmp\") pod \"insights-operator-78bdfb888c-nxz4w\" (UID: \"bf795e4e-8902-4aa8-be41-2671139da816\") " pod="openshift-insights/insights-operator-78bdfb888c-nxz4w"
May 25 12:14:46.534913 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:14:46.534843    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1dc72bef-ba17-4ede-88bb-0e1a18c43bbf-config\") pod \"kube-storage-version-migrator-operator-565644f49-7t27f\" (UID: \"1dc72bef-ba17-4ede-88bb-0e1a18c43bbf\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-7t27f"
May 25 12:14:46.534913 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:14:46.534854    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bf795e4e-8902-4aa8-be41-2671139da816-trusted-ca-bundle\") pod \"insights-operator-78bdfb888c-nxz4w\" (UID: \"bf795e4e-8902-4aa8-be41-2671139da816\") " pod="openshift-insights/insights-operator-78bdfb888c-nxz4w"
May 25 12:14:46.534913 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:14:46.534877    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/bf795e4e-8902-4aa8-be41-2671139da816-snapshots\") pod \"insights-operator-78bdfb888c-nxz4w\" (UID: \"bf795e4e-8902-4aa8-be41-2671139da816\") " pod="openshift-insights/insights-operator-78bdfb888c-nxz4w"
May 25 12:14:46.535059 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:14:46.534911    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-t784c\" (UniqueName: \"kubernetes.io/projected/1dc72bef-ba17-4ede-88bb-0e1a18c43bbf-kube-api-access-t784c\") pod \"kube-storage-version-migrator-operator-565644f49-7t27f\" (UID: \"1dc72bef-ba17-4ede-88bb-0e1a18c43bbf\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-7t27f"
May 25 12:14:46.535059 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:14:46.534937    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/90cc9eab-ce97-480c-b971-7487792a2833-stats-auth\") pod \"router-default-5b7954bdbb-7n4rl\" (UID: \"90cc9eab-ce97-480c-b971-7487792a2833\") " pod="openshift-ingress/router-default-5b7954bdbb-7n4rl"
May 25 12:14:46.535059 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:14:46.534971    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bf795e4e-8902-4aa8-be41-2671139da816-service-ca-bundle\") pod \"insights-operator-78bdfb888c-nxz4w\" (UID: \"bf795e4e-8902-4aa8-be41-2671139da816\") " pod="openshift-insights/insights-operator-78bdfb888c-nxz4w"
May 25 12:14:46.535224 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:14:46.535083    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-ld45d\" (UniqueName: \"kubernetes.io/projected/90cc9eab-ce97-480c-b971-7487792a2833-kube-api-access-ld45d\") pod \"router-default-5b7954bdbb-7n4rl\" (UID: \"90cc9eab-ce97-480c-b971-7487792a2833\") " pod="openshift-ingress/router-default-5b7954bdbb-7n4rl"
May 25 12:14:46.535224 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:14:46.535128    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-cgnsf\" (UniqueName: \"kubernetes.io/projected/bf795e4e-8902-4aa8-be41-2671139da816-kube-api-access-cgnsf\") pod \"insights-operator-78bdfb888c-nxz4w\" (UID: \"bf795e4e-8902-4aa8-be41-2671139da816\") " pod="openshift-insights/insights-operator-78bdfb888c-nxz4w"
May 25 12:14:46.535224 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:14:46.535168    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90cc9eab-ce97-480c-b971-7487792a2833-metrics-certs\") pod \"router-default-5b7954bdbb-7n4rl\" (UID: \"90cc9eab-ce97-480c-b971-7487792a2833\") " pod="openshift-ingress/router-default-5b7954bdbb-7n4rl"
May 25 12:14:46.535384 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:14:46.535323    2549 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
May 25 12:14:46.535438 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:14:46.535388    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90cc9eab-ce97-480c-b971-7487792a2833-metrics-certs podName:90cc9eab-ce97-480c-b971-7487792a2833 nodeName:}" failed. No retries permitted until 2026-05-25 12:14:47.035370666 +0000 UTC m=+90.751185932 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/90cc9eab-ce97-480c-b971-7487792a2833-metrics-certs") pod "router-default-5b7954bdbb-7n4rl" (UID: "90cc9eab-ce97-480c-b971-7487792a2833") : secret "router-metrics-certs-default" not found
May 25 12:14:46.535562 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:14:46.535516    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bf795e4e-8902-4aa8-be41-2671139da816-service-ca-bundle\") pod \"insights-operator-78bdfb888c-nxz4w\" (UID: \"bf795e4e-8902-4aa8-be41-2671139da816\") " pod="openshift-insights/insights-operator-78bdfb888c-nxz4w"
May 25 12:14:46.535766 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:14:46.535746    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bf795e4e-8902-4aa8-be41-2671139da816-trusted-ca-bundle\") pod \"insights-operator-78bdfb888c-nxz4w\" (UID: \"bf795e4e-8902-4aa8-be41-2671139da816\") " pod="openshift-insights/insights-operator-78bdfb888c-nxz4w"
May 25 12:14:46.536712 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:14:46.536689    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1dc72bef-ba17-4ede-88bb-0e1a18c43bbf-serving-cert\") pod \"kube-storage-version-migrator-operator-565644f49-7t27f\" (UID: \"1dc72bef-ba17-4ede-88bb-0e1a18c43bbf\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-7t27f"
May 25 12:14:46.536953 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:14:46.536936    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/90cc9eab-ce97-480c-b971-7487792a2833-default-certificate\") pod \"router-default-5b7954bdbb-7n4rl\" (UID: \"90cc9eab-ce97-480c-b971-7487792a2833\") " pod="openshift-ingress/router-default-5b7954bdbb-7n4rl"
May 25 12:14:46.537401 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:14:46.537377    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/bf795e4e-8902-4aa8-be41-2671139da816-serving-cert\") pod \"insights-operator-78bdfb888c-nxz4w\" (UID: \"bf795e4e-8902-4aa8-be41-2671139da816\") " pod="openshift-insights/insights-operator-78bdfb888c-nxz4w"
May 25 12:14:46.537613 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:14:46.537590    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/90cc9eab-ce97-480c-b971-7487792a2833-stats-auth\") pod \"router-default-5b7954bdbb-7n4rl\" (UID: \"90cc9eab-ce97-480c-b971-7487792a2833\") " pod="openshift-ingress/router-default-5b7954bdbb-7n4rl"
May 25 12:14:46.545929 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:14:46.545908    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t784c\" (UniqueName: \"kubernetes.io/projected/1dc72bef-ba17-4ede-88bb-0e1a18c43bbf-kube-api-access-t784c\") pod \"kube-storage-version-migrator-operator-565644f49-7t27f\" (UID: \"1dc72bef-ba17-4ede-88bb-0e1a18c43bbf\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-7t27f"
May 25 12:14:46.546719 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:14:46.546703    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgnsf\" (UniqueName: \"kubernetes.io/projected/bf795e4e-8902-4aa8-be41-2671139da816-kube-api-access-cgnsf\") pod \"insights-operator-78bdfb888c-nxz4w\" (UID: \"bf795e4e-8902-4aa8-be41-2671139da816\") " pod="openshift-insights/insights-operator-78bdfb888c-nxz4w"
May 25 12:14:46.547324 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:14:46.547309    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ld45d\" (UniqueName: \"kubernetes.io/projected/90cc9eab-ce97-480c-b971-7487792a2833-kube-api-access-ld45d\") pod \"router-default-5b7954bdbb-7n4rl\" (UID: \"90cc9eab-ce97-480c-b971-7487792a2833\") " pod="openshift-ingress/router-default-5b7954bdbb-7n4rl"
May 25 12:14:46.670742 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:14:46.670705    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-7t27f"
May 25 12:14:46.677562 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:14:46.677542    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-78bdfb888c-nxz4w"
May 25 12:14:46.798106 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:14:46.798027    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-7t27f"]
May 25 12:14:46.800804 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:14:46.800764    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1dc72bef_ba17_4ede_88bb_0e1a18c43bbf.slice/crio-372f5fba47dbd28c7366ab9095d3f9cca6476f47b0d924776189c0db7ae2eaaa WatchSource:0}: Error finding container 372f5fba47dbd28c7366ab9095d3f9cca6476f47b0d924776189c0db7ae2eaaa: Status 404 returned error can't find the container with id 372f5fba47dbd28c7366ab9095d3f9cca6476f47b0d924776189c0db7ae2eaaa
May 25 12:14:46.817144 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:14:46.817122    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-78bdfb888c-nxz4w"]
May 25 12:14:46.820101 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:14:46.820076    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf795e4e_8902_4aa8_be41_2671139da816.slice/crio-686c9573549f2c7157fecb7637b2dfb7f0bafbb39ba076880de0048e1914740d WatchSource:0}: Error finding container 686c9573549f2c7157fecb7637b2dfb7f0bafbb39ba076880de0048e1914740d: Status 404 returned error can't find the container with id 686c9573549f2c7157fecb7637b2dfb7f0bafbb39ba076880de0048e1914740d
May 25 12:14:46.938120 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:14:46.938078    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d452639f-6ffb-4b6c-9640-67b753e08fdf-samples-operator-tls\") pod \"cluster-samples-operator-697587599-f2jsk\" (UID: \"d452639f-6ffb-4b6c-9640-67b753e08fdf\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-f2jsk"
May 25 12:14:46.938306 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:14:46.938211    2549 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
May 25 12:14:46.938306 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:14:46.938280    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d452639f-6ffb-4b6c-9640-67b753e08fdf-samples-operator-tls podName:d452639f-6ffb-4b6c-9640-67b753e08fdf nodeName:}" failed. No retries permitted until 2026-05-25 12:14:47.938263195 +0000 UTC m=+91.654078462 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/d452639f-6ffb-4b6c-9640-67b753e08fdf-samples-operator-tls") pod "cluster-samples-operator-697587599-f2jsk" (UID: "d452639f-6ffb-4b6c-9640-67b753e08fdf") : secret "samples-operator-tls" not found
May 25 12:14:47.038811 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:14:47.038772    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/90cc9eab-ce97-480c-b971-7487792a2833-service-ca-bundle\") pod \"router-default-5b7954bdbb-7n4rl\" (UID: \"90cc9eab-ce97-480c-b971-7487792a2833\") " pod="openshift-ingress/router-default-5b7954bdbb-7n4rl"
May 25 12:14:47.038906 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:14:47.038842    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90cc9eab-ce97-480c-b971-7487792a2833-metrics-certs\") pod \"router-default-5b7954bdbb-7n4rl\" (UID: \"90cc9eab-ce97-480c-b971-7487792a2833\") " pod="openshift-ingress/router-default-5b7954bdbb-7n4rl"
May 25 12:14:47.038963 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:14:47.038934    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/90cc9eab-ce97-480c-b971-7487792a2833-service-ca-bundle podName:90cc9eab-ce97-480c-b971-7487792a2833 nodeName:}" failed. No retries permitted until 2026-05-25 12:14:48.038915507 +0000 UTC m=+91.754730789 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/90cc9eab-ce97-480c-b971-7487792a2833-service-ca-bundle") pod "router-default-5b7954bdbb-7n4rl" (UID: "90cc9eab-ce97-480c-b971-7487792a2833") : configmap references non-existent config key: service-ca.crt
May 25 12:14:47.038963 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:14:47.038936    2549 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
May 25 12:14:47.039151 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:14:47.039024    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90cc9eab-ce97-480c-b971-7487792a2833-metrics-certs podName:90cc9eab-ce97-480c-b971-7487792a2833 nodeName:}" failed. No retries permitted until 2026-05-25 12:14:48.039011725 +0000 UTC m=+91.754826990 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/90cc9eab-ce97-480c-b971-7487792a2833-metrics-certs") pod "router-default-5b7954bdbb-7n4rl" (UID: "90cc9eab-ce97-480c-b971-7487792a2833") : secret "router-metrics-certs-default" not found
May 25 12:14:47.237104 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:14:47.237068    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-78bdfb888c-nxz4w" event={"ID":"bf795e4e-8902-4aa8-be41-2671139da816","Type":"ContainerStarted","Data":"686c9573549f2c7157fecb7637b2dfb7f0bafbb39ba076880de0048e1914740d"}
May 25 12:14:47.237930 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:14:47.237908    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-7t27f" event={"ID":"1dc72bef-ba17-4ede-88bb-0e1a18c43bbf","Type":"ContainerStarted","Data":"372f5fba47dbd28c7366ab9095d3f9cca6476f47b0d924776189c0db7ae2eaaa"}
May 25 12:14:47.945917 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:14:47.945862    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d452639f-6ffb-4b6c-9640-67b753e08fdf-samples-operator-tls\") pod \"cluster-samples-operator-697587599-f2jsk\" (UID: \"d452639f-6ffb-4b6c-9640-67b753e08fdf\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-f2jsk"
May 25 12:14:47.946417 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:14:47.946002    2549 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
May 25 12:14:47.946417 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:14:47.946062    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d452639f-6ffb-4b6c-9640-67b753e08fdf-samples-operator-tls podName:d452639f-6ffb-4b6c-9640-67b753e08fdf nodeName:}" failed. No retries permitted until 2026-05-25 12:14:49.94604806 +0000 UTC m=+93.661863326 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/d452639f-6ffb-4b6c-9640-67b753e08fdf-samples-operator-tls") pod "cluster-samples-operator-697587599-f2jsk" (UID: "d452639f-6ffb-4b6c-9640-67b753e08fdf") : secret "samples-operator-tls" not found
May 25 12:14:48.046401 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:14:48.046364    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/90cc9eab-ce97-480c-b971-7487792a2833-service-ca-bundle\") pod \"router-default-5b7954bdbb-7n4rl\" (UID: \"90cc9eab-ce97-480c-b971-7487792a2833\") " pod="openshift-ingress/router-default-5b7954bdbb-7n4rl"
May 25 12:14:48.046776 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:14:48.046439    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90cc9eab-ce97-480c-b971-7487792a2833-metrics-certs\") pod \"router-default-5b7954bdbb-7n4rl\" (UID: \"90cc9eab-ce97-480c-b971-7487792a2833\") " pod="openshift-ingress/router-default-5b7954bdbb-7n4rl"
May 25 12:14:48.046776 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:14:48.046538    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/90cc9eab-ce97-480c-b971-7487792a2833-service-ca-bundle podName:90cc9eab-ce97-480c-b971-7487792a2833 nodeName:}" failed. No retries permitted until 2026-05-25 12:14:50.046519853 +0000 UTC m=+93.762335137 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/90cc9eab-ce97-480c-b971-7487792a2833-service-ca-bundle") pod "router-default-5b7954bdbb-7n4rl" (UID: "90cc9eab-ce97-480c-b971-7487792a2833") : configmap references non-existent config key: service-ca.crt
May 25 12:14:48.046776 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:14:48.046587    2549 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
May 25 12:14:48.046776 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:14:48.046645    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90cc9eab-ce97-480c-b971-7487792a2833-metrics-certs podName:90cc9eab-ce97-480c-b971-7487792a2833 nodeName:}" failed. No retries permitted until 2026-05-25 12:14:50.046632132 +0000 UTC m=+93.762447411 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/90cc9eab-ce97-480c-b971-7487792a2833-metrics-certs") pod "router-default-5b7954bdbb-7n4rl" (UID: "90cc9eab-ce97-480c-b971-7487792a2833") : secret "router-metrics-certs-default" not found
May 25 12:14:49.243591 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:14:49.243536    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-78bdfb888c-nxz4w" event={"ID":"bf795e4e-8902-4aa8-be41-2671139da816","Type":"ContainerStarted","Data":"32e4d277473ee027b7d875c1cb90246b2ab52d06be2cdc907133393062c94259"}
May 25 12:14:49.244764 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:14:49.244740    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-7t27f" event={"ID":"1dc72bef-ba17-4ede-88bb-0e1a18c43bbf","Type":"ContainerStarted","Data":"41d86f35477064ceb561aa9f63448747e6adb2b73da01bf99719192959b9f80e"}
May 25 12:14:49.290246 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:14:49.290044    2549 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-7t27f" podStartSLOduration=0.913606862 podStartE2EDuration="3.290027268s" podCreationTimestamp="2026-05-25 12:14:46 +0000 UTC" firstStartedPulling="2026-05-25 12:14:46.802574624 +0000 UTC m=+90.518389893" lastFinishedPulling="2026-05-25 12:14:49.178995029 +0000 UTC m=+92.894810299" observedRunningTime="2026-05-25 12:14:49.289166265 +0000 UTC m=+93.004981552" watchObservedRunningTime="2026-05-25 12:14:49.290027268 +0000 UTC m=+93.005842554"
May 25 12:14:49.290246 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:14:49.290124    2549 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-insights/insights-operator-78bdfb888c-nxz4w" podStartSLOduration=0.93411708 podStartE2EDuration="3.290118434s" podCreationTimestamp="2026-05-25 12:14:46 +0000 UTC" firstStartedPulling="2026-05-25 12:14:46.821920058 +0000 UTC m=+90.537735327" lastFinishedPulling="2026-05-25 12:14:49.177921411 +0000 UTC m=+92.893736681" observedRunningTime="2026-05-25 12:14:49.263969862 +0000 UTC m=+92.979785149" watchObservedRunningTime="2026-05-25 12:14:49.290118434 +0000 UTC m=+93.005933723"
May 25 12:14:49.960586 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:14:49.960535    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d452639f-6ffb-4b6c-9640-67b753e08fdf-samples-operator-tls\") pod \"cluster-samples-operator-697587599-f2jsk\" (UID: \"d452639f-6ffb-4b6c-9640-67b753e08fdf\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-f2jsk"
May 25 12:14:49.960792 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:14:49.960692    2549 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
May 25 12:14:49.960792 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:14:49.960761    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d452639f-6ffb-4b6c-9640-67b753e08fdf-samples-operator-tls podName:d452639f-6ffb-4b6c-9640-67b753e08fdf nodeName:}" failed. No retries permitted until 2026-05-25 12:14:53.960745543 +0000 UTC m=+97.676560814 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/d452639f-6ffb-4b6c-9640-67b753e08fdf-samples-operator-tls") pod "cluster-samples-operator-697587599-f2jsk" (UID: "d452639f-6ffb-4b6c-9640-67b753e08fdf") : secret "samples-operator-tls" not found
May 25 12:14:50.060966 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:14:50.060924    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/90cc9eab-ce97-480c-b971-7487792a2833-service-ca-bundle\") pod \"router-default-5b7954bdbb-7n4rl\" (UID: \"90cc9eab-ce97-480c-b971-7487792a2833\") " pod="openshift-ingress/router-default-5b7954bdbb-7n4rl"
May 25 12:14:50.061111 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:14:50.060983    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90cc9eab-ce97-480c-b971-7487792a2833-metrics-certs\") pod \"router-default-5b7954bdbb-7n4rl\" (UID: \"90cc9eab-ce97-480c-b971-7487792a2833\") " pod="openshift-ingress/router-default-5b7954bdbb-7n4rl"
May 25 12:14:50.061111 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:14:50.061061    2549 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
May 25 12:14:50.061111 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:14:50.061106    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90cc9eab-ce97-480c-b971-7487792a2833-metrics-certs podName:90cc9eab-ce97-480c-b971-7487792a2833 nodeName:}" failed. No retries permitted until 2026-05-25 12:14:54.061091549 +0000 UTC m=+97.776906815 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/90cc9eab-ce97-480c-b971-7487792a2833-metrics-certs") pod "router-default-5b7954bdbb-7n4rl" (UID: "90cc9eab-ce97-480c-b971-7487792a2833") : secret "router-metrics-certs-default" not found
May 25 12:14:50.061300 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:14:50.061119    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/90cc9eab-ce97-480c-b971-7487792a2833-service-ca-bundle podName:90cc9eab-ce97-480c-b971-7487792a2833 nodeName:}" failed. No retries permitted until 2026-05-25 12:14:54.061113656 +0000 UTC m=+97.776928921 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/90cc9eab-ce97-480c-b971-7487792a2833-service-ca-bundle") pod "router-default-5b7954bdbb-7n4rl" (UID: "90cc9eab-ce97-480c-b971-7487792a2833") : configmap references non-existent config key: service-ca.crt
May 25 12:14:52.600611 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:14:52.600583    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-dv9dt_61e828df-19f1-4008-bcb9-2c3a879b0178/dns-node-resolver/0.log"
May 25 12:14:53.110628 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:14:53.110601    2549 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-67d667c7d-9c25t"]
May 25 12:14:53.112475 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:14:53.112451    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-67d667c7d-9c25t"
May 25 12:14:53.115074 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:14:53.115052    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-lvcd4\""
May 25 12:14:53.115273 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:14:53.115182    2549 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
May 25 12:14:53.116253 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:14:53.116229    2549 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
May 25 12:14:53.116253 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:14:53.116249    2549 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
May 25 12:14:53.116401 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:14:53.116285    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
May 25 12:14:53.122236 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:14:53.122218    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-67d667c7d-9c25t"]
May 25 12:14:53.185486 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:14:53.185452    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/0f2f55d1-f249-41eb-a962-b8716725f2b7-signing-key\") pod \"service-ca-67d667c7d-9c25t\" (UID: \"0f2f55d1-f249-41eb-a962-b8716725f2b7\") " pod="openshift-service-ca/service-ca-67d667c7d-9c25t"
May 25 12:14:53.185656 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:14:53.185602    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/0f2f55d1-f249-41eb-a962-b8716725f2b7-signing-cabundle\") pod \"service-ca-67d667c7d-9c25t\" (UID: \"0f2f55d1-f249-41eb-a962-b8716725f2b7\") " pod="openshift-service-ca/service-ca-67d667c7d-9c25t"
May 25 12:14:53.185656 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:14:53.185630    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwsbg\" (UniqueName: \"kubernetes.io/projected/0f2f55d1-f249-41eb-a962-b8716725f2b7-kube-api-access-gwsbg\") pod \"service-ca-67d667c7d-9c25t\" (UID: \"0f2f55d1-f249-41eb-a962-b8716725f2b7\") " pod="openshift-service-ca/service-ca-67d667c7d-9c25t"
May 25 12:14:53.286450 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:14:53.286418    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-gwsbg\" (UniqueName: \"kubernetes.io/projected/0f2f55d1-f249-41eb-a962-b8716725f2b7-kube-api-access-gwsbg\") pod \"service-ca-67d667c7d-9c25t\" (UID: \"0f2f55d1-f249-41eb-a962-b8716725f2b7\") " pod="openshift-service-ca/service-ca-67d667c7d-9c25t"
May 25 12:14:53.286621 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:14:53.286459    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/0f2f55d1-f249-41eb-a962-b8716725f2b7-signing-key\") pod \"service-ca-67d667c7d-9c25t\" (UID: \"0f2f55d1-f249-41eb-a962-b8716725f2b7\") " pod="openshift-service-ca/service-ca-67d667c7d-9c25t"
May 25 12:14:53.286621 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:14:53.286600    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/0f2f55d1-f249-41eb-a962-b8716725f2b7-signing-cabundle\") pod \"service-ca-67d667c7d-9c25t\" (UID: \"0f2f55d1-f249-41eb-a962-b8716725f2b7\") " pod="openshift-service-ca/service-ca-67d667c7d-9c25t"
May 25 12:14:53.287236 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:14:53.287189    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/0f2f55d1-f249-41eb-a962-b8716725f2b7-signing-cabundle\") pod \"service-ca-67d667c7d-9c25t\" (UID: \"0f2f55d1-f249-41eb-a962-b8716725f2b7\") " pod="openshift-service-ca/service-ca-67d667c7d-9c25t"
May 25 12:14:53.288794 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:14:53.288775    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/0f2f55d1-f249-41eb-a962-b8716725f2b7-signing-key\") pod \"service-ca-67d667c7d-9c25t\" (UID: \"0f2f55d1-f249-41eb-a962-b8716725f2b7\") " pod="openshift-service-ca/service-ca-67d667c7d-9c25t"
May 25 12:14:53.296806 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:14:53.296788    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwsbg\" (UniqueName: \"kubernetes.io/projected/0f2f55d1-f249-41eb-a962-b8716725f2b7-kube-api-access-gwsbg\") pod \"service-ca-67d667c7d-9c25t\" (UID: \"0f2f55d1-f249-41eb-a962-b8716725f2b7\") " pod="openshift-service-ca/service-ca-67d667c7d-9c25t"
May 25 12:14:53.402783 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:14:53.402706    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-bjpxs_c13dfb5f-85fd-4f8e-9a7f-f432a01720d3/node-ca/0.log"
May 25 12:14:53.421875 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:14:53.421848    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-67d667c7d-9c25t"
May 25 12:14:53.542775 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:14:53.542743    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-67d667c7d-9c25t"]
May 25 12:14:53.545595 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:14:53.545568    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f2f55d1_f249_41eb_a962_b8716725f2b7.slice/crio-296df9c9e04022eb227e81bd3d8d12b8896d24839256854f806c2dde67987ffd WatchSource:0}: Error finding container 296df9c9e04022eb227e81bd3d8d12b8896d24839256854f806c2dde67987ffd: Status 404 returned error can't find the container with id 296df9c9e04022eb227e81bd3d8d12b8896d24839256854f806c2dde67987ffd
May 25 12:14:53.991537 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:14:53.991496    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d452639f-6ffb-4b6c-9640-67b753e08fdf-samples-operator-tls\") pod \"cluster-samples-operator-697587599-f2jsk\" (UID: \"d452639f-6ffb-4b6c-9640-67b753e08fdf\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-f2jsk"
May 25 12:14:53.991943 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:14:53.991650    2549 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
May 25 12:14:53.991943 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:14:53.991718    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d452639f-6ffb-4b6c-9640-67b753e08fdf-samples-operator-tls podName:d452639f-6ffb-4b6c-9640-67b753e08fdf nodeName:}" failed. No retries permitted until 2026-05-25 12:15:01.991702039 +0000 UTC m=+105.707517304 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/d452639f-6ffb-4b6c-9640-67b753e08fdf-samples-operator-tls") pod "cluster-samples-operator-697587599-f2jsk" (UID: "d452639f-6ffb-4b6c-9640-67b753e08fdf") : secret "samples-operator-tls" not found
May 25 12:14:54.092406 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:14:54.092369    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90cc9eab-ce97-480c-b971-7487792a2833-metrics-certs\") pod \"router-default-5b7954bdbb-7n4rl\" (UID: \"90cc9eab-ce97-480c-b971-7487792a2833\") " pod="openshift-ingress/router-default-5b7954bdbb-7n4rl"
May 25 12:14:54.092587 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:14:54.092458    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/90cc9eab-ce97-480c-b971-7487792a2833-service-ca-bundle\") pod \"router-default-5b7954bdbb-7n4rl\" (UID: \"90cc9eab-ce97-480c-b971-7487792a2833\") " pod="openshift-ingress/router-default-5b7954bdbb-7n4rl"
May 25 12:14:54.092587 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:14:54.092527    2549 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
May 25 12:14:54.092703 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:14:54.092603    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90cc9eab-ce97-480c-b971-7487792a2833-metrics-certs podName:90cc9eab-ce97-480c-b971-7487792a2833 nodeName:}" failed. No retries permitted until 2026-05-25 12:15:02.092582753 +0000 UTC m=+105.808398039 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/90cc9eab-ce97-480c-b971-7487792a2833-metrics-certs") pod "router-default-5b7954bdbb-7n4rl" (UID: "90cc9eab-ce97-480c-b971-7487792a2833") : secret "router-metrics-certs-default" not found
May 25 12:14:54.092703 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:14:54.092623    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/90cc9eab-ce97-480c-b971-7487792a2833-service-ca-bundle podName:90cc9eab-ce97-480c-b971-7487792a2833 nodeName:}" failed. No retries permitted until 2026-05-25 12:15:02.092613495 +0000 UTC m=+105.808428761 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/90cc9eab-ce97-480c-b971-7487792a2833-service-ca-bundle") pod "router-default-5b7954bdbb-7n4rl" (UID: "90cc9eab-ce97-480c-b971-7487792a2833") : configmap references non-existent config key: service-ca.crt
May 25 12:14:54.254933 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:14:54.254846    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-67d667c7d-9c25t" event={"ID":"0f2f55d1-f249-41eb-a962-b8716725f2b7","Type":"ContainerStarted","Data":"296df9c9e04022eb227e81bd3d8d12b8896d24839256854f806c2dde67987ffd"}
May 25 12:14:54.999672 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:14:54.999617    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0ea389da-cc7e-4b86-846d-e95f5a2dae7c-cert\") pod \"ingress-canary-fxlzt\" (UID: \"0ea389da-cc7e-4b86-846d-e95f5a2dae7c\") " pod="openshift-ingress-canary/ingress-canary-fxlzt"
May 25 12:14:55.000102 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:14:54.999780    2549 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 25 12:14:55.000102 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:14:54.999783    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/acccf0fc-6d0f-4793-acb0-69227950e520-metrics-tls\") pod \"dns-default-ml56g\" (UID: \"acccf0fc-6d0f-4793-acb0-69227950e520\") " pod="openshift-dns/dns-default-ml56g"
May 25 12:14:55.000102 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:14:54.999856    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0ea389da-cc7e-4b86-846d-e95f5a2dae7c-cert podName:0ea389da-cc7e-4b86-846d-e95f5a2dae7c nodeName:}" failed. No retries permitted until 2026-05-25 12:15:58.999834786 +0000 UTC m=+162.715650051 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/0ea389da-cc7e-4b86-846d-e95f5a2dae7c-cert") pod "ingress-canary-fxlzt" (UID: "0ea389da-cc7e-4b86-846d-e95f5a2dae7c") : secret "canary-serving-cert" not found
May 25 12:14:55.000102 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:14:54.999876    2549 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 25 12:14:55.000102 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:14:54.999943    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/acccf0fc-6d0f-4793-acb0-69227950e520-metrics-tls podName:acccf0fc-6d0f-4793-acb0-69227950e520 nodeName:}" failed. No retries permitted until 2026-05-25 12:15:58.999927413 +0000 UTC m=+162.715742681 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/acccf0fc-6d0f-4793-acb0-69227950e520-metrics-tls") pod "dns-default-ml56g" (UID: "acccf0fc-6d0f-4793-acb0-69227950e520") : secret "dns-default-metrics-tls" not found
May 25 12:14:55.258083 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:14:55.257989    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-67d667c7d-9c25t" event={"ID":"0f2f55d1-f249-41eb-a962-b8716725f2b7","Type":"ContainerStarted","Data":"4140e4695080393646149ae833c4f8b8d2c1ccfe9aa8d828ee06708c57bfda8d"}
May 25 12:14:55.282853 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:14:55.282790    2549 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-service-ca/service-ca-67d667c7d-9c25t" podStartSLOduration=0.717320923 podStartE2EDuration="2.282777576s" podCreationTimestamp="2026-05-25 12:14:53 +0000 UTC" firstStartedPulling="2026-05-25 12:14:53.547396365 +0000 UTC m=+97.263211630" lastFinishedPulling="2026-05-25 12:14:55.112853013 +0000 UTC m=+98.828668283" observedRunningTime="2026-05-25 12:14:55.282286653 +0000 UTC m=+98.998101940" watchObservedRunningTime="2026-05-25 12:14:55.282777576 +0000 UTC m=+98.998592863"
May 25 12:14:56.199291 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:14:56.199254    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-tp68k"
May 25 12:15:02.055080 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:02.055028    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d452639f-6ffb-4b6c-9640-67b753e08fdf-samples-operator-tls\") pod \"cluster-samples-operator-697587599-f2jsk\" (UID: \"d452639f-6ffb-4b6c-9640-67b753e08fdf\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-f2jsk"
May 25 12:15:02.057590 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:02.057556    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d452639f-6ffb-4b6c-9640-67b753e08fdf-samples-operator-tls\") pod \"cluster-samples-operator-697587599-f2jsk\" (UID: \"d452639f-6ffb-4b6c-9640-67b753e08fdf\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-f2jsk"
May 25 12:15:02.155417 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:02.155377    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/90cc9eab-ce97-480c-b971-7487792a2833-service-ca-bundle\") pod \"router-default-5b7954bdbb-7n4rl\" (UID: \"90cc9eab-ce97-480c-b971-7487792a2833\") " pod="openshift-ingress/router-default-5b7954bdbb-7n4rl"
May 25 12:15:02.155598 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:02.155438    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90cc9eab-ce97-480c-b971-7487792a2833-metrics-certs\") pod \"router-default-5b7954bdbb-7n4rl\" (UID: \"90cc9eab-ce97-480c-b971-7487792a2833\") " pod="openshift-ingress/router-default-5b7954bdbb-7n4rl"
May 25 12:15:02.155720 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:15:02.155595    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/90cc9eab-ce97-480c-b971-7487792a2833-service-ca-bundle podName:90cc9eab-ce97-480c-b971-7487792a2833 nodeName:}" failed. No retries permitted until 2026-05-25 12:15:18.15557179 +0000 UTC m=+121.871387068 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/90cc9eab-ce97-480c-b971-7487792a2833-service-ca-bundle") pod "router-default-5b7954bdbb-7n4rl" (UID: "90cc9eab-ce97-480c-b971-7487792a2833") : configmap references non-existent config key: service-ca.crt
May 25 12:15:02.157766 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:02.157746    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90cc9eab-ce97-480c-b971-7487792a2833-metrics-certs\") pod \"router-default-5b7954bdbb-7n4rl\" (UID: \"90cc9eab-ce97-480c-b971-7487792a2833\") " pod="openshift-ingress/router-default-5b7954bdbb-7n4rl"
May 25 12:15:02.161639 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:02.161619    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-f2jsk"
May 25 12:15:02.287140 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:02.287104    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-697587599-f2jsk"]
May 25 12:15:03.274414 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:03.274374    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-f2jsk" event={"ID":"d452639f-6ffb-4b6c-9640-67b753e08fdf","Type":"ContainerStarted","Data":"14d30fc482e9d7022c525e68f18c0ed09cee16c26710deb05a1eaf20c77bd6a3"}
May 25 12:15:04.278277 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:04.278170    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-f2jsk" event={"ID":"d452639f-6ffb-4b6c-9640-67b753e08fdf","Type":"ContainerStarted","Data":"0e75eb2677aba715ccfcc7a58994412a93f97d8637b8d56d9f0ddbef7b58fb02"}
May 25 12:15:04.278277 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:04.278224    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-f2jsk" event={"ID":"d452639f-6ffb-4b6c-9640-67b753e08fdf","Type":"ContainerStarted","Data":"06ebe0dd955b25d2a9eed1cc89762b23b5183467bb84557887a9ba7b1679ed79"}
May 25 12:15:13.917271 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:13.917215    2549 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-f2jsk" podStartSLOduration=26.275933818 podStartE2EDuration="27.917182372s" podCreationTimestamp="2026-05-25 12:14:46 +0000 UTC" firstStartedPulling="2026-05-25 12:15:02.32912433 +0000 UTC m=+106.044939597" lastFinishedPulling="2026-05-25 12:15:03.970372887 +0000 UTC m=+107.686188151" observedRunningTime="2026-05-25 12:15:04.300347624 +0000 UTC m=+108.016162910" watchObservedRunningTime="2026-05-25 12:15:13.917182372 +0000 UTC m=+117.632997666"
May 25 12:15:13.917788 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:13.917506    2549 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-d4d8f94f9-pdv7d"]
May 25 12:15:13.920382 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:13.920361    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-d4d8f94f9-pdv7d"
May 25 12:15:13.922928 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:13.922911    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-2bzlg\""
May 25 12:15:13.923021 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:13.922913    2549 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
May 25 12:15:13.924004 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:13.923983    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
May 25 12:15:13.928076 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:13.928053    2549 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-2hm2r"]
May 25 12:15:13.931341 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:13.931319    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-d4d8f94f9-pdv7d"]
May 25 12:15:13.931437 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:13.931426    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-2hm2r"
May 25 12:15:13.934038 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:13.934014    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-82qnj\""
May 25 12:15:13.934150 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:13.934130    2549 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
May 25 12:15:13.934361 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:13.934348    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
May 25 12:15:13.940090 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:13.940068    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-2hm2r"]
May 25 12:15:14.024553 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:14.024523    2549 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-796b7fd686-7ctpn"]
May 25 12:15:14.027648 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:14.027626    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-796b7fd686-7ctpn"
May 25 12:15:14.031365 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:14.031345    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
May 25 12:15:14.032418 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:14.032401    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
May 25 12:15:14.034119 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:14.034100    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-s7xbv\""
May 25 12:15:14.035296 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:14.035279    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
May 25 12:15:14.042500 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:14.042471    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-796b7fd686-7ctpn"]
May 25 12:15:14.042642 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:14.042512    2549 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
May 25 12:15:14.047231 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:14.047192    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/69226669-8b67-4826-97c7-92970d34863a-crio-socket\") pod \"insights-runtime-extractor-2hm2r\" (UID: \"69226669-8b67-4826-97c7-92970d34863a\") " pod="openshift-insights/insights-runtime-extractor-2hm2r"
May 25 12:15:14.047341 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:14.047288    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/69226669-8b67-4826-97c7-92970d34863a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2hm2r\" (UID: \"69226669-8b67-4826-97c7-92970d34863a\") " pod="openshift-insights/insights-runtime-extractor-2hm2r"
May 25 12:15:14.047341 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:14.047328    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fg668\" (UniqueName: \"kubernetes.io/projected/69226669-8b67-4826-97c7-92970d34863a-kube-api-access-fg668\") pod \"insights-runtime-extractor-2hm2r\" (UID: \"69226669-8b67-4826-97c7-92970d34863a\") " pod="openshift-insights/insights-runtime-extractor-2hm2r"
May 25 12:15:14.047435 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:14.047360    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/69226669-8b67-4826-97c7-92970d34863a-data-volume\") pod \"insights-runtime-extractor-2hm2r\" (UID: \"69226669-8b67-4826-97c7-92970d34863a\") " pod="openshift-insights/insights-runtime-extractor-2hm2r"
May 25 12:15:14.047435 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:14.047383    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b7f50355-0097-4b6a-8b1a-01a3f7fd1d26-nginx-conf\") pod \"networking-console-plugin-d4d8f94f9-pdv7d\" (UID: \"b7f50355-0097-4b6a-8b1a-01a3f7fd1d26\") " pod="openshift-network-console/networking-console-plugin-d4d8f94f9-pdv7d"
May 25 12:15:14.047500 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:14.047478    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b7f50355-0097-4b6a-8b1a-01a3f7fd1d26-networking-console-plugin-cert\") pod \"networking-console-plugin-d4d8f94f9-pdv7d\" (UID: \"b7f50355-0097-4b6a-8b1a-01a3f7fd1d26\") " pod="openshift-network-console/networking-console-plugin-d4d8f94f9-pdv7d"
May 25 12:15:14.047538 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:14.047524    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/69226669-8b67-4826-97c7-92970d34863a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2hm2r\" (UID: \"69226669-8b67-4826-97c7-92970d34863a\") " pod="openshift-insights/insights-runtime-extractor-2hm2r"
May 25 12:15:14.078349 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:14.078313    2549 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-796b7fd686-7ctpn"]
May 25 12:15:14.078563 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:15:14.078535    2549 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[bound-sa-token ca-trust-extracted image-registry-private-configuration installation-pull-secrets kube-api-access-9xrjn registry-certificates registry-tls trusted-ca], unattached volumes=[], failed to process volumes=[bound-sa-token ca-trust-extracted image-registry-private-configuration installation-pull-secrets kube-api-access-9xrjn registry-certificates registry-tls trusted-ca]: context canceled" pod="openshift-image-registry/image-registry-796b7fd686-7ctpn" podUID="935872f3-87a2-40f0-86ea-acb3079f632b"
May 25 12:15:14.148681 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:14.148638    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/69226669-8b67-4826-97c7-92970d34863a-crio-socket\") pod \"insights-runtime-extractor-2hm2r\" (UID: \"69226669-8b67-4826-97c7-92970d34863a\") " pod="openshift-insights/insights-runtime-extractor-2hm2r"
May 25 12:15:14.148867 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:14.148703    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/935872f3-87a2-40f0-86ea-acb3079f632b-image-registry-private-configuration\") pod \"image-registry-796b7fd686-7ctpn\" (UID: \"935872f3-87a2-40f0-86ea-acb3079f632b\") " pod="openshift-image-registry/image-registry-796b7fd686-7ctpn"
May 25 12:15:14.148867 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:14.148739    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/69226669-8b67-4826-97c7-92970d34863a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2hm2r\" (UID: \"69226669-8b67-4826-97c7-92970d34863a\") " pod="openshift-insights/insights-runtime-extractor-2hm2r"
May 25 12:15:14.148867 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:14.148772    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/69226669-8b67-4826-97c7-92970d34863a-crio-socket\") pod \"insights-runtime-extractor-2hm2r\" (UID: \"69226669-8b67-4826-97c7-92970d34863a\") " pod="openshift-insights/insights-runtime-extractor-2hm2r"
May 25 12:15:14.148867 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:14.148784    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/935872f3-87a2-40f0-86ea-acb3079f632b-trusted-ca\") pod \"image-registry-796b7fd686-7ctpn\" (UID: \"935872f3-87a2-40f0-86ea-acb3079f632b\") " pod="openshift-image-registry/image-registry-796b7fd686-7ctpn"
May 25 12:15:14.149046 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:14.148922    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xrjn\" (UniqueName: \"kubernetes.io/projected/935872f3-87a2-40f0-86ea-acb3079f632b-kube-api-access-9xrjn\") pod \"image-registry-796b7fd686-7ctpn\" (UID: \"935872f3-87a2-40f0-86ea-acb3079f632b\") " pod="openshift-image-registry/image-registry-796b7fd686-7ctpn"
May 25 12:15:14.149046 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:14.148972    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-fg668\" (UniqueName: \"kubernetes.io/projected/69226669-8b67-4826-97c7-92970d34863a-kube-api-access-fg668\") pod \"insights-runtime-extractor-2hm2r\" (UID: \"69226669-8b67-4826-97c7-92970d34863a\") " pod="openshift-insights/insights-runtime-extractor-2hm2r"
May 25 12:15:14.149046 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:14.149031    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/935872f3-87a2-40f0-86ea-acb3079f632b-registry-certificates\") pod \"image-registry-796b7fd686-7ctpn\" (UID: \"935872f3-87a2-40f0-86ea-acb3079f632b\") " pod="openshift-image-registry/image-registry-796b7fd686-7ctpn"
May 25 12:15:14.149175 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:14.149071    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/69226669-8b67-4826-97c7-92970d34863a-data-volume\") pod \"insights-runtime-extractor-2hm2r\" (UID: \"69226669-8b67-4826-97c7-92970d34863a\") " pod="openshift-insights/insights-runtime-extractor-2hm2r"
May 25 12:15:14.149258 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:14.149192    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b7f50355-0097-4b6a-8b1a-01a3f7fd1d26-nginx-conf\") pod \"networking-console-plugin-d4d8f94f9-pdv7d\" (UID: \"b7f50355-0097-4b6a-8b1a-01a3f7fd1d26\") " pod="openshift-network-console/networking-console-plugin-d4d8f94f9-pdv7d"
May 25 12:15:14.149313 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:14.149258    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b7f50355-0097-4b6a-8b1a-01a3f7fd1d26-networking-console-plugin-cert\") pod \"networking-console-plugin-d4d8f94f9-pdv7d\" (UID: \"b7f50355-0097-4b6a-8b1a-01a3f7fd1d26\") " pod="openshift-network-console/networking-console-plugin-d4d8f94f9-pdv7d"
May 25 12:15:14.149406 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:14.149387    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/69226669-8b67-4826-97c7-92970d34863a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2hm2r\" (UID: \"69226669-8b67-4826-97c7-92970d34863a\") " pod="openshift-insights/insights-runtime-extractor-2hm2r"
May 25 12:15:14.149461 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:14.149401    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/69226669-8b67-4826-97c7-92970d34863a-data-volume\") pod \"insights-runtime-extractor-2hm2r\" (UID: \"69226669-8b67-4826-97c7-92970d34863a\") " pod="openshift-insights/insights-runtime-extractor-2hm2r"
May 25 12:15:14.149517 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:14.149455    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/935872f3-87a2-40f0-86ea-acb3079f632b-ca-trust-extracted\") pod \"image-registry-796b7fd686-7ctpn\" (UID: \"935872f3-87a2-40f0-86ea-acb3079f632b\") " pod="openshift-image-registry/image-registry-796b7fd686-7ctpn"
May 25 12:15:14.149517 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:14.149489    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/935872f3-87a2-40f0-86ea-acb3079f632b-registry-tls\") pod \"image-registry-796b7fd686-7ctpn\" (UID: \"935872f3-87a2-40f0-86ea-acb3079f632b\") " pod="openshift-image-registry/image-registry-796b7fd686-7ctpn"
May 25 12:15:14.149618 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:14.149520    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/935872f3-87a2-40f0-86ea-acb3079f632b-installation-pull-secrets\") pod \"image-registry-796b7fd686-7ctpn\" (UID: \"935872f3-87a2-40f0-86ea-acb3079f632b\") " pod="openshift-image-registry/image-registry-796b7fd686-7ctpn"
May 25 12:15:14.149618 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:14.149548    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/935872f3-87a2-40f0-86ea-acb3079f632b-bound-sa-token\") pod \"image-registry-796b7fd686-7ctpn\" (UID: \"935872f3-87a2-40f0-86ea-acb3079f632b\") " pod="openshift-image-registry/image-registry-796b7fd686-7ctpn"
May 25 12:15:14.149851 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:14.149819    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/69226669-8b67-4826-97c7-92970d34863a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2hm2r\" (UID: \"69226669-8b67-4826-97c7-92970d34863a\") " pod="openshift-insights/insights-runtime-extractor-2hm2r"
May 25 12:15:14.149957 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:14.149942    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b7f50355-0097-4b6a-8b1a-01a3f7fd1d26-nginx-conf\") pod \"networking-console-plugin-d4d8f94f9-pdv7d\" (UID: \"b7f50355-0097-4b6a-8b1a-01a3f7fd1d26\") " pod="openshift-network-console/networking-console-plugin-d4d8f94f9-pdv7d"
May 25 12:15:14.151125 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:14.151106    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/69226669-8b67-4826-97c7-92970d34863a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2hm2r\" (UID: \"69226669-8b67-4826-97c7-92970d34863a\") " pod="openshift-insights/insights-runtime-extractor-2hm2r"
May 25 12:15:14.151696 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:14.151680    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b7f50355-0097-4b6a-8b1a-01a3f7fd1d26-networking-console-plugin-cert\") pod \"networking-console-plugin-d4d8f94f9-pdv7d\" (UID: \"b7f50355-0097-4b6a-8b1a-01a3f7fd1d26\") " pod="openshift-network-console/networking-console-plugin-d4d8f94f9-pdv7d"
May 25 12:15:14.167348 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:14.167285    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fg668\" (UniqueName: \"kubernetes.io/projected/69226669-8b67-4826-97c7-92970d34863a-kube-api-access-fg668\") pod \"insights-runtime-extractor-2hm2r\" (UID: \"69226669-8b67-4826-97c7-92970d34863a\") " pod="openshift-insights/insights-runtime-extractor-2hm2r"
May 25 12:15:14.229399 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:14.229361    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-d4d8f94f9-pdv7d"
May 25 12:15:14.240158 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:14.240130    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-2hm2r"
May 25 12:15:14.249966 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:14.249941    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/935872f3-87a2-40f0-86ea-acb3079f632b-ca-trust-extracted\") pod \"image-registry-796b7fd686-7ctpn\" (UID: \"935872f3-87a2-40f0-86ea-acb3079f632b\") " pod="openshift-image-registry/image-registry-796b7fd686-7ctpn"
May 25 12:15:14.250099 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:14.249974    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/935872f3-87a2-40f0-86ea-acb3079f632b-registry-tls\") pod \"image-registry-796b7fd686-7ctpn\" (UID: \"935872f3-87a2-40f0-86ea-acb3079f632b\") " pod="openshift-image-registry/image-registry-796b7fd686-7ctpn"
May 25 12:15:14.250099 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:14.249998    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/935872f3-87a2-40f0-86ea-acb3079f632b-installation-pull-secrets\") pod \"image-registry-796b7fd686-7ctpn\" (UID: \"935872f3-87a2-40f0-86ea-acb3079f632b\") " pod="openshift-image-registry/image-registry-796b7fd686-7ctpn"
May 25 12:15:14.250227 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:14.250150    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/935872f3-87a2-40f0-86ea-acb3079f632b-bound-sa-token\") pod \"image-registry-796b7fd686-7ctpn\" (UID: \"935872f3-87a2-40f0-86ea-acb3079f632b\") " pod="openshift-image-registry/image-registry-796b7fd686-7ctpn"
May 25 12:15:14.250308 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:14.250266    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/935872f3-87a2-40f0-86ea-acb3079f632b-image-registry-private-configuration\") pod \"image-registry-796b7fd686-7ctpn\" (UID: \"935872f3-87a2-40f0-86ea-acb3079f632b\") " pod="openshift-image-registry/image-registry-796b7fd686-7ctpn"
May 25 12:15:14.250377 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:14.250321    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/935872f3-87a2-40f0-86ea-acb3079f632b-trusted-ca\") pod \"image-registry-796b7fd686-7ctpn\" (UID: \"935872f3-87a2-40f0-86ea-acb3079f632b\") " pod="openshift-image-registry/image-registry-796b7fd686-7ctpn"
May 25 12:15:14.250377 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:14.250348    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-9xrjn\" (UniqueName: \"kubernetes.io/projected/935872f3-87a2-40f0-86ea-acb3079f632b-kube-api-access-9xrjn\") pod \"image-registry-796b7fd686-7ctpn\" (UID: \"935872f3-87a2-40f0-86ea-acb3079f632b\") " pod="openshift-image-registry/image-registry-796b7fd686-7ctpn"
May 25 12:15:14.250484 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:14.250372    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/935872f3-87a2-40f0-86ea-acb3079f632b-ca-trust-extracted\") pod \"image-registry-796b7fd686-7ctpn\" (UID: \"935872f3-87a2-40f0-86ea-acb3079f632b\") " pod="openshift-image-registry/image-registry-796b7fd686-7ctpn"
May 25 12:15:14.250484 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:14.250412    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/935872f3-87a2-40f0-86ea-acb3079f632b-registry-certificates\") pod \"image-registry-796b7fd686-7ctpn\" (UID: \"935872f3-87a2-40f0-86ea-acb3079f632b\") " pod="openshift-image-registry/image-registry-796b7fd686-7ctpn"
May 25 12:15:14.251287 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:14.251243    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/935872f3-87a2-40f0-86ea-acb3079f632b-registry-certificates\") pod \"image-registry-796b7fd686-7ctpn\" (UID: \"935872f3-87a2-40f0-86ea-acb3079f632b\") " pod="openshift-image-registry/image-registry-796b7fd686-7ctpn"
May 25 12:15:14.251385 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:14.251328    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/935872f3-87a2-40f0-86ea-acb3079f632b-trusted-ca\") pod \"image-registry-796b7fd686-7ctpn\" (UID: \"935872f3-87a2-40f0-86ea-acb3079f632b\") " pod="openshift-image-registry/image-registry-796b7fd686-7ctpn"
May 25 12:15:14.253190 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:14.253169    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/935872f3-87a2-40f0-86ea-acb3079f632b-registry-tls\") pod \"image-registry-796b7fd686-7ctpn\" (UID: \"935872f3-87a2-40f0-86ea-acb3079f632b\") " pod="openshift-image-registry/image-registry-796b7fd686-7ctpn"
May 25 12:15:14.253329 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:14.253310    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/935872f3-87a2-40f0-86ea-acb3079f632b-image-registry-private-configuration\") pod \"image-registry-796b7fd686-7ctpn\" (UID: \"935872f3-87a2-40f0-86ea-acb3079f632b\") " pod="openshift-image-registry/image-registry-796b7fd686-7ctpn"
May 25 12:15:14.253672 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:14.253646    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/935872f3-87a2-40f0-86ea-acb3079f632b-installation-pull-secrets\") pod \"image-registry-796b7fd686-7ctpn\" (UID: \"935872f3-87a2-40f0-86ea-acb3079f632b\") " pod="openshift-image-registry/image-registry-796b7fd686-7ctpn"
May 25 12:15:14.261690 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:14.261659    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/935872f3-87a2-40f0-86ea-acb3079f632b-bound-sa-token\") pod \"image-registry-796b7fd686-7ctpn\" (UID: \"935872f3-87a2-40f0-86ea-acb3079f632b\") " pod="openshift-image-registry/image-registry-796b7fd686-7ctpn"
May 25 12:15:14.261690 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:14.261675    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xrjn\" (UniqueName: \"kubernetes.io/projected/935872f3-87a2-40f0-86ea-acb3079f632b-kube-api-access-9xrjn\") pod \"image-registry-796b7fd686-7ctpn\" (UID: \"935872f3-87a2-40f0-86ea-acb3079f632b\") " pod="openshift-image-registry/image-registry-796b7fd686-7ctpn"
May 25 12:15:14.300879 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:14.300847    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-796b7fd686-7ctpn"
May 25 12:15:14.306938 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:14.306913    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-796b7fd686-7ctpn"
May 25 12:15:14.351612 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:14.351580    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/935872f3-87a2-40f0-86ea-acb3079f632b-ca-trust-extracted\") pod \"935872f3-87a2-40f0-86ea-acb3079f632b\" (UID: \"935872f3-87a2-40f0-86ea-acb3079f632b\") "
May 25 12:15:14.351775 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:14.351638    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9xrjn\" (UniqueName: \"kubernetes.io/projected/935872f3-87a2-40f0-86ea-acb3079f632b-kube-api-access-9xrjn\") pod \"935872f3-87a2-40f0-86ea-acb3079f632b\" (UID: \"935872f3-87a2-40f0-86ea-acb3079f632b\") "
May 25 12:15:14.351775 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:14.351693    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/935872f3-87a2-40f0-86ea-acb3079f632b-trusted-ca\") pod \"935872f3-87a2-40f0-86ea-acb3079f632b\" (UID: \"935872f3-87a2-40f0-86ea-acb3079f632b\") "
May 25 12:15:14.351775 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:14.351720    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/935872f3-87a2-40f0-86ea-acb3079f632b-installation-pull-secrets\") pod \"935872f3-87a2-40f0-86ea-acb3079f632b\" (UID: \"935872f3-87a2-40f0-86ea-acb3079f632b\") "
May 25 12:15:14.351775 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:14.351749    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/935872f3-87a2-40f0-86ea-acb3079f632b-registry-tls\") pod \"935872f3-87a2-40f0-86ea-acb3079f632b\" (UID: \"935872f3-87a2-40f0-86ea-acb3079f632b\") "
May 25 12:15:14.351988 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:14.351778    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/935872f3-87a2-40f0-86ea-acb3079f632b-image-registry-private-configuration\") pod \"935872f3-87a2-40f0-86ea-acb3079f632b\" (UID: \"935872f3-87a2-40f0-86ea-acb3079f632b\") "
May 25 12:15:14.351988 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:14.351824    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/935872f3-87a2-40f0-86ea-acb3079f632b-registry-certificates\") pod \"935872f3-87a2-40f0-86ea-acb3079f632b\" (UID: \"935872f3-87a2-40f0-86ea-acb3079f632b\") "
May 25 12:15:14.351988 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:14.351850    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/935872f3-87a2-40f0-86ea-acb3079f632b-bound-sa-token\") pod \"935872f3-87a2-40f0-86ea-acb3079f632b\" (UID: \"935872f3-87a2-40f0-86ea-acb3079f632b\") "
May 25 12:15:14.352269 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:14.352189    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/935872f3-87a2-40f0-86ea-acb3079f632b-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "935872f3-87a2-40f0-86ea-acb3079f632b" (UID: "935872f3-87a2-40f0-86ea-acb3079f632b"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 12:15:14.352438 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:14.352351    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/935872f3-87a2-40f0-86ea-acb3079f632b-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "935872f3-87a2-40f0-86ea-acb3079f632b" (UID: "935872f3-87a2-40f0-86ea-acb3079f632b"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 12:15:14.352839 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:14.352809    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/935872f3-87a2-40f0-86ea-acb3079f632b-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "935872f3-87a2-40f0-86ea-acb3079f632b" (UID: "935872f3-87a2-40f0-86ea-acb3079f632b"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 12:15:14.354244 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:14.354184    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/935872f3-87a2-40f0-86ea-acb3079f632b-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "935872f3-87a2-40f0-86ea-acb3079f632b" (UID: "935872f3-87a2-40f0-86ea-acb3079f632b"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 12:15:14.354344 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:14.354238    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/935872f3-87a2-40f0-86ea-acb3079f632b-kube-api-access-9xrjn" (OuterVolumeSpecName: "kube-api-access-9xrjn") pod "935872f3-87a2-40f0-86ea-acb3079f632b" (UID: "935872f3-87a2-40f0-86ea-acb3079f632b"). InnerVolumeSpecName "kube-api-access-9xrjn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 12:15:14.354386 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:14.354339    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/935872f3-87a2-40f0-86ea-acb3079f632b-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "935872f3-87a2-40f0-86ea-acb3079f632b" (UID: "935872f3-87a2-40f0-86ea-acb3079f632b"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 12:15:14.354386 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:14.354345    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/935872f3-87a2-40f0-86ea-acb3079f632b-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "935872f3-87a2-40f0-86ea-acb3079f632b" (UID: "935872f3-87a2-40f0-86ea-acb3079f632b"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 12:15:14.354615 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:14.354595    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/935872f3-87a2-40f0-86ea-acb3079f632b-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "935872f3-87a2-40f0-86ea-acb3079f632b" (UID: "935872f3-87a2-40f0-86ea-acb3079f632b"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 12:15:14.368775 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:14.368741    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-d4d8f94f9-pdv7d"]
May 25 12:15:14.371369 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:15:14.371345    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7f50355_0097_4b6a_8b1a_01a3f7fd1d26.slice/crio-d9118309615b901fa4044b3ff988ad4d6d71cf040c2c3091b16851141101c62c WatchSource:0}: Error finding container d9118309615b901fa4044b3ff988ad4d6d71cf040c2c3091b16851141101c62c: Status 404 returned error can't find the container with id d9118309615b901fa4044b3ff988ad4d6d71cf040c2c3091b16851141101c62c
May 25 12:15:14.391100 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:14.391070    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-2hm2r"]
May 25 12:15:14.394008 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:15:14.393984    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69226669_8b67_4826_97c7_92970d34863a.slice/crio-9082d6f7f4cf589b0071a29a7795e3bacbcf5b2956d7dcee06205827ee5ed245 WatchSource:0}: Error finding container 9082d6f7f4cf589b0071a29a7795e3bacbcf5b2956d7dcee06205827ee5ed245: Status 404 returned error can't find the container with id 9082d6f7f4cf589b0071a29a7795e3bacbcf5b2956d7dcee06205827ee5ed245
May 25 12:15:14.452797 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:14.452771    2549 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/935872f3-87a2-40f0-86ea-acb3079f632b-ca-trust-extracted\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
May 25 12:15:14.452898 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:14.452801    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9xrjn\" (UniqueName: \"kubernetes.io/projected/935872f3-87a2-40f0-86ea-acb3079f632b-kube-api-access-9xrjn\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
May 25 12:15:14.452898 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:14.452816    2549 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/935872f3-87a2-40f0-86ea-acb3079f632b-trusted-ca\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
May 25 12:15:14.452898 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:14.452831    2549 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/935872f3-87a2-40f0-86ea-acb3079f632b-installation-pull-secrets\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
May 25 12:15:14.452898 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:14.452845    2549 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/935872f3-87a2-40f0-86ea-acb3079f632b-registry-tls\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
May 25 12:15:14.452898 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:14.452860    2549 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/935872f3-87a2-40f0-86ea-acb3079f632b-image-registry-private-configuration\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
May 25 12:15:14.452898 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:14.452875    2549 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/935872f3-87a2-40f0-86ea-acb3079f632b-registry-certificates\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
May 25 12:15:14.452898 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:14.452888    2549 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/935872f3-87a2-40f0-86ea-acb3079f632b-bound-sa-token\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
May 25 12:15:15.304494 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:15.304450    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-d4d8f94f9-pdv7d" event={"ID":"b7f50355-0097-4b6a-8b1a-01a3f7fd1d26","Type":"ContainerStarted","Data":"d9118309615b901fa4044b3ff988ad4d6d71cf040c2c3091b16851141101c62c"}
May 25 12:15:15.306393 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:15.306367    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2hm2r" event={"ID":"69226669-8b67-4826-97c7-92970d34863a","Type":"ContainerStarted","Data":"7b3b6002e825bd82e5899b6b23cc37d8242100280a181d491adab38b7105c935"}
May 25 12:15:15.306519 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:15.306389    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-796b7fd686-7ctpn"
May 25 12:15:15.306519 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:15.306402    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2hm2r" event={"ID":"69226669-8b67-4826-97c7-92970d34863a","Type":"ContainerStarted","Data":"2479e5f5143131ce8f04acfa31dfd6ab4f8f5a142784607a9e34b5da15d047a5"}
May 25 12:15:15.306519 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:15.306419    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2hm2r" event={"ID":"69226669-8b67-4826-97c7-92970d34863a","Type":"ContainerStarted","Data":"9082d6f7f4cf589b0071a29a7795e3bacbcf5b2956d7dcee06205827ee5ed245"}
May 25 12:15:15.342145 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:15.342111    2549 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-796b7fd686-7ctpn"]
May 25 12:15:15.347395 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:15.347365    2549 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-796b7fd686-7ctpn"]
May 25 12:15:16.309810 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:16.309779    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-d4d8f94f9-pdv7d" event={"ID":"b7f50355-0097-4b6a-8b1a-01a3f7fd1d26","Type":"ContainerStarted","Data":"fab4f19df10a8e45cefdf6fe4e8b6687e39bcb375c18031c9e81751a34e89392"}
May 25 12:15:16.327474 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:16.327412    2549 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-d4d8f94f9-pdv7d" podStartSLOduration=2.117763964 podStartE2EDuration="3.327393398s" podCreationTimestamp="2026-05-25 12:15:13 +0000 UTC" firstStartedPulling="2026-05-25 12:15:14.373034387 +0000 UTC m=+118.088849651" lastFinishedPulling="2026-05-25 12:15:15.582663812 +0000 UTC m=+119.298479085" observedRunningTime="2026-05-25 12:15:16.326459367 +0000 UTC m=+120.042274654" watchObservedRunningTime="2026-05-25 12:15:16.327393398 +0000 UTC m=+120.043208686"
May 25 12:15:16.846050 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:16.846009    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="935872f3-87a2-40f0-86ea-acb3079f632b" path="/var/lib/kubelet/pods/935872f3-87a2-40f0-86ea-acb3079f632b/volumes"
May 25 12:15:17.313635 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:17.313595    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2hm2r" event={"ID":"69226669-8b67-4826-97c7-92970d34863a","Type":"ContainerStarted","Data":"4a4cf6a470009cbacc0b0e918bbf8c6e3432af7de7d9dcf112d8008762db9bd4"}
May 25 12:15:17.334909 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:17.334860    2549 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-2hm2r" podStartSLOduration=2.059346362 podStartE2EDuration="4.334845487s" podCreationTimestamp="2026-05-25 12:15:13 +0000 UTC" firstStartedPulling="2026-05-25 12:15:14.455811201 +0000 UTC m=+118.171626469" lastFinishedPulling="2026-05-25 12:15:16.731310312 +0000 UTC m=+120.447125594" observedRunningTime="2026-05-25 12:15:17.334248234 +0000 UTC m=+121.050063523" watchObservedRunningTime="2026-05-25 12:15:17.334845487 +0000 UTC m=+121.050660787"
May 25 12:15:18.187955 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:18.187912    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/90cc9eab-ce97-480c-b971-7487792a2833-service-ca-bundle\") pod \"router-default-5b7954bdbb-7n4rl\" (UID: \"90cc9eab-ce97-480c-b971-7487792a2833\") " pod="openshift-ingress/router-default-5b7954bdbb-7n4rl"
May 25 12:15:18.188527 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:18.188506    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/90cc9eab-ce97-480c-b971-7487792a2833-service-ca-bundle\") pod \"router-default-5b7954bdbb-7n4rl\" (UID: \"90cc9eab-ce97-480c-b971-7487792a2833\") " pod="openshift-ingress/router-default-5b7954bdbb-7n4rl"
May 25 12:15:18.484993 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:18.484908    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-ltkjm\""
May 25 12:15:18.493261 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:18.493240    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-5b7954bdbb-7n4rl"
May 25 12:15:18.617953 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:18.617922    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-5b7954bdbb-7n4rl"]
May 25 12:15:18.621099 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:15:18.621075    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90cc9eab_ce97_480c_b971_7487792a2833.slice/crio-fa6f3854cd6149bdafc74fde39adc2bcc61070b7aac62cd4000469b6e89e387f WatchSource:0}: Error finding container fa6f3854cd6149bdafc74fde39adc2bcc61070b7aac62cd4000469b6e89e387f: Status 404 returned error can't find the container with id fa6f3854cd6149bdafc74fde39adc2bcc61070b7aac62cd4000469b6e89e387f
May 25 12:15:19.319962 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:19.319923    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-5b7954bdbb-7n4rl" event={"ID":"90cc9eab-ce97-480c-b971-7487792a2833","Type":"ContainerStarted","Data":"cb56dbcc17ab1c772d8c4de2166b52427297291e179dbfffe36d87e6320ca17c"}
May 25 12:15:19.320137 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:19.319968    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-5b7954bdbb-7n4rl" event={"ID":"90cc9eab-ce97-480c-b971-7487792a2833","Type":"ContainerStarted","Data":"fa6f3854cd6149bdafc74fde39adc2bcc61070b7aac62cd4000469b6e89e387f"}
May 25 12:15:19.341431 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:19.341387    2549 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ingress/router-default-5b7954bdbb-7n4rl" podStartSLOduration=33.341374253 podStartE2EDuration="33.341374253s" podCreationTimestamp="2026-05-25 12:14:46 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 12:15:19.34070067 +0000 UTC m=+123.056516000" watchObservedRunningTime="2026-05-25 12:15:19.341374253 +0000 UTC m=+123.057189534"
May 25 12:15:19.493889 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:19.493852    2549 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-5b7954bdbb-7n4rl"
May 25 12:15:19.496445 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:19.496418    2549 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-5b7954bdbb-7n4rl"
May 25 12:15:20.323434 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:20.323399    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-5b7954bdbb-7n4rl"
May 25 12:15:20.324655 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:20.324637    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-5b7954bdbb-7n4rl"
May 25 12:15:21.555360 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:21.555329    2549 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-9wkp6"]
May 25 12:15:21.558845 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:21.558818    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-9wkp6"
May 25 12:15:21.561464 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:21.561442    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
May 25 12:15:21.561571 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:21.561494    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-pf4ll\""
May 25 12:15:21.569050 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:21.569031    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-9wkp6"]
May 25 12:15:21.617721 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:21.617694    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/46e2b2e0-2156-41bc-9ce7-3a67562ed0e5-tls-certificates\") pod \"prometheus-operator-admission-webhook-6c4bbc67b7-9wkp6\" (UID: \"46e2b2e0-2156-41bc-9ce7-3a67562ed0e5\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-9wkp6"
May 25 12:15:21.718968 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:21.718937    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/46e2b2e0-2156-41bc-9ce7-3a67562ed0e5-tls-certificates\") pod \"prometheus-operator-admission-webhook-6c4bbc67b7-9wkp6\" (UID: \"46e2b2e0-2156-41bc-9ce7-3a67562ed0e5\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-9wkp6"
May 25 12:15:21.721303 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:21.721282    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/46e2b2e0-2156-41bc-9ce7-3a67562ed0e5-tls-certificates\") pod \"prometheus-operator-admission-webhook-6c4bbc67b7-9wkp6\" (UID: \"46e2b2e0-2156-41bc-9ce7-3a67562ed0e5\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-9wkp6"
May 25 12:15:21.869378 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:21.869342    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-9wkp6"
May 25 12:15:21.990795 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:21.990766    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-9wkp6"]
May 25 12:15:21.993844 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:15:21.993810    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46e2b2e0_2156_41bc_9ce7_3a67562ed0e5.slice/crio-5d8813ffcff04e4bb96e934bcfcef98b0fa3024a0759a245657ba6535ba29bf6 WatchSource:0}: Error finding container 5d8813ffcff04e4bb96e934bcfcef98b0fa3024a0759a245657ba6535ba29bf6: Status 404 returned error can't find the container with id 5d8813ffcff04e4bb96e934bcfcef98b0fa3024a0759a245657ba6535ba29bf6
May 25 12:15:22.159116 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:22.159040    2549 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console/console-7654684cf-5vvhl"]
May 25 12:15:22.164205 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:22.164179    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7654684cf-5vvhl"
May 25 12:15:22.166947 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:22.166927    2549 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
May 25 12:15:22.167069 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:22.167009    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
May 25 12:15:22.168430 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:22.168304    2549 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
May 25 12:15:22.168430 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:22.168320    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-vwn8c\""
May 25 12:15:22.168430 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:22.168322    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
May 25 12:15:22.168430 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:22.168305    2549 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
May 25 12:15:22.168430 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:22.168385    2549 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
May 25 12:15:22.168430 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:22.168390    2549 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
May 25 12:15:22.171737 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:22.171717    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7654684cf-5vvhl"]
May 25 12:15:22.222450 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:22.222423    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/620aafc8-f648-4809-8abe-094c0f66f3de-console-config\") pod \"console-7654684cf-5vvhl\" (UID: \"620aafc8-f648-4809-8abe-094c0f66f3de\") " pod="openshift-console/console-7654684cf-5vvhl"
May 25 12:15:22.222598 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:22.222458    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/620aafc8-f648-4809-8abe-094c0f66f3de-console-oauth-config\") pod \"console-7654684cf-5vvhl\" (UID: \"620aafc8-f648-4809-8abe-094c0f66f3de\") " pod="openshift-console/console-7654684cf-5vvhl"
May 25 12:15:22.222598 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:22.222486    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/620aafc8-f648-4809-8abe-094c0f66f3de-oauth-serving-cert\") pod \"console-7654684cf-5vvhl\" (UID: \"620aafc8-f648-4809-8abe-094c0f66f3de\") " pod="openshift-console/console-7654684cf-5vvhl"
May 25 12:15:22.222598 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:22.222552    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/620aafc8-f648-4809-8abe-094c0f66f3de-console-serving-cert\") pod \"console-7654684cf-5vvhl\" (UID: \"620aafc8-f648-4809-8abe-094c0f66f3de\") " pod="openshift-console/console-7654684cf-5vvhl"
May 25 12:15:22.222754 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:22.222622    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctrrg\" (UniqueName: \"kubernetes.io/projected/620aafc8-f648-4809-8abe-094c0f66f3de-kube-api-access-ctrrg\") pod \"console-7654684cf-5vvhl\" (UID: \"620aafc8-f648-4809-8abe-094c0f66f3de\") " pod="openshift-console/console-7654684cf-5vvhl"
May 25 12:15:22.222754 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:22.222656    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/620aafc8-f648-4809-8abe-094c0f66f3de-service-ca\") pod \"console-7654684cf-5vvhl\" (UID: \"620aafc8-f648-4809-8abe-094c0f66f3de\") " pod="openshift-console/console-7654684cf-5vvhl"
May 25 12:15:22.323762 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:22.323727    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/620aafc8-f648-4809-8abe-094c0f66f3de-console-serving-cert\") pod \"console-7654684cf-5vvhl\" (UID: \"620aafc8-f648-4809-8abe-094c0f66f3de\") " pod="openshift-console/console-7654684cf-5vvhl"
May 25 12:15:22.323954 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:22.323784    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-ctrrg\" (UniqueName: \"kubernetes.io/projected/620aafc8-f648-4809-8abe-094c0f66f3de-kube-api-access-ctrrg\") pod \"console-7654684cf-5vvhl\" (UID: \"620aafc8-f648-4809-8abe-094c0f66f3de\") " pod="openshift-console/console-7654684cf-5vvhl"
May 25 12:15:22.323954 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:22.323810    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/620aafc8-f648-4809-8abe-094c0f66f3de-service-ca\") pod \"console-7654684cf-5vvhl\" (UID: \"620aafc8-f648-4809-8abe-094c0f66f3de\") " pod="openshift-console/console-7654684cf-5vvhl"
May 25 12:15:22.323954 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:22.323842    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/620aafc8-f648-4809-8abe-094c0f66f3de-console-config\") pod \"console-7654684cf-5vvhl\" (UID: \"620aafc8-f648-4809-8abe-094c0f66f3de\") " pod="openshift-console/console-7654684cf-5vvhl"
May 25 12:15:22.323954 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:22.323859    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/620aafc8-f648-4809-8abe-094c0f66f3de-console-oauth-config\") pod \"console-7654684cf-5vvhl\" (UID: \"620aafc8-f648-4809-8abe-094c0f66f3de\") " pod="openshift-console/console-7654684cf-5vvhl"
May 25 12:15:22.323954 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:22.323876    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/620aafc8-f648-4809-8abe-094c0f66f3de-oauth-serving-cert\") pod \"console-7654684cf-5vvhl\" (UID: \"620aafc8-f648-4809-8abe-094c0f66f3de\") " pod="openshift-console/console-7654684cf-5vvhl"
May 25 12:15:22.324572 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:22.324543    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/620aafc8-f648-4809-8abe-094c0f66f3de-service-ca\") pod \"console-7654684cf-5vvhl\" (UID: \"620aafc8-f648-4809-8abe-094c0f66f3de\") " pod="openshift-console/console-7654684cf-5vvhl"
May 25 12:15:22.324687 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:22.324570    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/620aafc8-f648-4809-8abe-094c0f66f3de-console-config\") pod \"console-7654684cf-5vvhl\" (UID: \"620aafc8-f648-4809-8abe-094c0f66f3de\") " pod="openshift-console/console-7654684cf-5vvhl"
May 25 12:15:22.324687 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:22.324654    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/620aafc8-f648-4809-8abe-094c0f66f3de-oauth-serving-cert\") pod \"console-7654684cf-5vvhl\" (UID: \"620aafc8-f648-4809-8abe-094c0f66f3de\") " pod="openshift-console/console-7654684cf-5vvhl"
May 25 12:15:22.326350 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:22.326323    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/620aafc8-f648-4809-8abe-094c0f66f3de-console-serving-cert\") pod \"console-7654684cf-5vvhl\" (UID: \"620aafc8-f648-4809-8abe-094c0f66f3de\") " pod="openshift-console/console-7654684cf-5vvhl"
May 25 12:15:22.326437 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:22.326407    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/620aafc8-f648-4809-8abe-094c0f66f3de-console-oauth-config\") pod \"console-7654684cf-5vvhl\" (UID: \"620aafc8-f648-4809-8abe-094c0f66f3de\") " pod="openshift-console/console-7654684cf-5vvhl"
May 25 12:15:22.328538 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:22.328515    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-9wkp6" event={"ID":"46e2b2e0-2156-41bc-9ce7-3a67562ed0e5","Type":"ContainerStarted","Data":"5d8813ffcff04e4bb96e934bcfcef98b0fa3024a0759a245657ba6535ba29bf6"}
May 25 12:15:22.334163 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:22.334145    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctrrg\" (UniqueName: \"kubernetes.io/projected/620aafc8-f648-4809-8abe-094c0f66f3de-kube-api-access-ctrrg\") pod \"console-7654684cf-5vvhl\" (UID: \"620aafc8-f648-4809-8abe-094c0f66f3de\") " pod="openshift-console/console-7654684cf-5vvhl"
May 25 12:15:22.473840 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:22.473762    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7654684cf-5vvhl"
May 25 12:15:22.610066 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:22.610038    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7654684cf-5vvhl"]
May 25 12:15:22.613380 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:15:22.613351    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod620aafc8_f648_4809_8abe_094c0f66f3de.slice/crio-cc68baf0fe0338572716deb27207a2bbfce1c1ac6f4b5e652c40830da36c19ec WatchSource:0}: Error finding container cc68baf0fe0338572716deb27207a2bbfce1c1ac6f4b5e652c40830da36c19ec: Status 404 returned error can't find the container with id cc68baf0fe0338572716deb27207a2bbfce1c1ac6f4b5e652c40830da36c19ec
May 25 12:15:23.331757 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:23.331722    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7654684cf-5vvhl" event={"ID":"620aafc8-f648-4809-8abe-094c0f66f3de","Type":"ContainerStarted","Data":"cc68baf0fe0338572716deb27207a2bbfce1c1ac6f4b5e652c40830da36c19ec"}
May 25 12:15:23.332949 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:23.332928    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-9wkp6" event={"ID":"46e2b2e0-2156-41bc-9ce7-3a67562ed0e5","Type":"ContainerStarted","Data":"042ae816666d6752f24c793f3b44b9cee1abb604efef3f08e0e518cae9ac7d09"}
May 25 12:15:23.333243 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:23.333222    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-9wkp6"
May 25 12:15:23.337810 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:23.337789    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-9wkp6"
May 25 12:15:23.354281 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:23.354242    2549 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-9wkp6" podStartSLOduration=1.183609302 podStartE2EDuration="2.354229946s" podCreationTimestamp="2026-05-25 12:15:21 +0000 UTC" firstStartedPulling="2026-05-25 12:15:21.996093895 +0000 UTC m=+125.711909160" lastFinishedPulling="2026-05-25 12:15:23.166714539 +0000 UTC m=+126.882529804" observedRunningTime="2026-05-25 12:15:23.352680154 +0000 UTC m=+127.068495440" watchObservedRunningTime="2026-05-25 12:15:23.354229946 +0000 UTC m=+127.070045233"
May 25 12:15:23.637630 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:23.637591    2549 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-7779bbdd78-mmnkt"]
May 25 12:15:23.640829 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:23.640805    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-7779bbdd78-mmnkt"
May 25 12:15:23.644984 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:23.644757    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
May 25 12:15:23.644984 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:23.644773    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
May 25 12:15:23.644984 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:23.644790    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-fb9kx\""
May 25 12:15:23.644984 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:23.644808    2549 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
May 25 12:15:23.644984 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:23.644760    2549 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
May 25 12:15:23.644984 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:23.644811    2549 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
May 25 12:15:23.651839 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:23.651815    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-7779bbdd78-mmnkt"]
May 25 12:15:23.735718 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:23.735679    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/03d5c580-b34a-4949-ac56-082edfcb0b51-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7779bbdd78-mmnkt\" (UID: \"03d5c580-b34a-4949-ac56-082edfcb0b51\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-mmnkt"
May 25 12:15:23.735909 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:23.735816    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7pj9\" (UniqueName: \"kubernetes.io/projected/03d5c580-b34a-4949-ac56-082edfcb0b51-kube-api-access-h7pj9\") pod \"prometheus-operator-7779bbdd78-mmnkt\" (UID: \"03d5c580-b34a-4949-ac56-082edfcb0b51\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-mmnkt"
May 25 12:15:23.735909 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:23.735867    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/03d5c580-b34a-4949-ac56-082edfcb0b51-prometheus-operator-tls\") pod \"prometheus-operator-7779bbdd78-mmnkt\" (UID: \"03d5c580-b34a-4949-ac56-082edfcb0b51\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-mmnkt"
May 25 12:15:23.735909 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:23.735894    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/03d5c580-b34a-4949-ac56-082edfcb0b51-metrics-client-ca\") pod \"prometheus-operator-7779bbdd78-mmnkt\" (UID: \"03d5c580-b34a-4949-ac56-082edfcb0b51\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-mmnkt"
May 25 12:15:23.836891 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:23.836857    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/03d5c580-b34a-4949-ac56-082edfcb0b51-metrics-client-ca\") pod \"prometheus-operator-7779bbdd78-mmnkt\" (UID: \"03d5c580-b34a-4949-ac56-082edfcb0b51\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-mmnkt"
May 25 12:15:23.837066 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:23.837032    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/03d5c580-b34a-4949-ac56-082edfcb0b51-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7779bbdd78-mmnkt\" (UID: \"03d5c580-b34a-4949-ac56-082edfcb0b51\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-mmnkt"
May 25 12:15:23.837147 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:23.837122    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-h7pj9\" (UniqueName: \"kubernetes.io/projected/03d5c580-b34a-4949-ac56-082edfcb0b51-kube-api-access-h7pj9\") pod \"prometheus-operator-7779bbdd78-mmnkt\" (UID: \"03d5c580-b34a-4949-ac56-082edfcb0b51\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-mmnkt"
May 25 12:15:23.837228 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:23.837160    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/03d5c580-b34a-4949-ac56-082edfcb0b51-prometheus-operator-tls\") pod \"prometheus-operator-7779bbdd78-mmnkt\" (UID: \"03d5c580-b34a-4949-ac56-082edfcb0b51\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-mmnkt"
May 25 12:15:23.837690 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:23.837665    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/03d5c580-b34a-4949-ac56-082edfcb0b51-metrics-client-ca\") pod \"prometheus-operator-7779bbdd78-mmnkt\" (UID: \"03d5c580-b34a-4949-ac56-082edfcb0b51\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-mmnkt"
May 25 12:15:23.839872 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:23.839845    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/03d5c580-b34a-4949-ac56-082edfcb0b51-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7779bbdd78-mmnkt\" (UID: \"03d5c580-b34a-4949-ac56-082edfcb0b51\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-mmnkt"
May 25 12:15:23.839975 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:23.839956    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/03d5c580-b34a-4949-ac56-082edfcb0b51-prometheus-operator-tls\") pod \"prometheus-operator-7779bbdd78-mmnkt\" (UID: \"03d5c580-b34a-4949-ac56-082edfcb0b51\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-mmnkt"
May 25 12:15:23.848371 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:23.848351    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7pj9\" (UniqueName: \"kubernetes.io/projected/03d5c580-b34a-4949-ac56-082edfcb0b51-kube-api-access-h7pj9\") pod \"prometheus-operator-7779bbdd78-mmnkt\" (UID: \"03d5c580-b34a-4949-ac56-082edfcb0b51\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-mmnkt"
May 25 12:15:23.951894 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:23.951811    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-7779bbdd78-mmnkt"
May 25 12:15:24.098873 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:24.098838    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-7779bbdd78-mmnkt"]
May 25 12:15:24.102024 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:15:24.101994    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03d5c580_b34a_4949_ac56_082edfcb0b51.slice/crio-3f9b242d1d233ad331e3d5aaced1607659ed40cd2db6a8299dd136306c56602b WatchSource:0}: Error finding container 3f9b242d1d233ad331e3d5aaced1607659ed40cd2db6a8299dd136306c56602b: Status 404 returned error can't find the container with id 3f9b242d1d233ad331e3d5aaced1607659ed40cd2db6a8299dd136306c56602b
May 25 12:15:24.337342 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:24.337254    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7779bbdd78-mmnkt" event={"ID":"03d5c580-b34a-4949-ac56-082edfcb0b51","Type":"ContainerStarted","Data":"3f9b242d1d233ad331e3d5aaced1607659ed40cd2db6a8299dd136306c56602b"}
May 25 12:15:25.555063 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:25.555023    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1124f1f1-fdc7-4a81-b28a-620db95e2ff6-metrics-certs\") pod \"network-metrics-daemon-zf62d\" (UID: \"1124f1f1-fdc7-4a81-b28a-620db95e2ff6\") " pod="openshift-multus/network-metrics-daemon-zf62d"
May 25 12:15:25.557626 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:25.557603    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1124f1f1-fdc7-4a81-b28a-620db95e2ff6-metrics-certs\") pod \"network-metrics-daemon-zf62d\" (UID: \"1124f1f1-fdc7-4a81-b28a-620db95e2ff6\") " pod="openshift-multus/network-metrics-daemon-zf62d"
May 25 12:15:25.759279 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:25.759244    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-rrs9z\""
May 25 12:15:25.766867 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:25.766837    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zf62d"
May 25 12:15:26.085321 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:26.085290    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-zf62d"]
May 25 12:15:26.088481 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:15:26.088446    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1124f1f1_fdc7_4a81_b28a_620db95e2ff6.slice/crio-f7d40774a1c38a663739886e94f4eedc859bb6c02ed5f8df5fed1b2b4058288b WatchSource:0}: Error finding container f7d40774a1c38a663739886e94f4eedc859bb6c02ed5f8df5fed1b2b4058288b: Status 404 returned error can't find the container with id f7d40774a1c38a663739886e94f4eedc859bb6c02ed5f8df5fed1b2b4058288b
May 25 12:15:26.344402 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:26.344366    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7779bbdd78-mmnkt" event={"ID":"03d5c580-b34a-4949-ac56-082edfcb0b51","Type":"ContainerStarted","Data":"148f9b9d7146ea137250f6701b454066e1d9b3c3f7de0637f30f307cc8a6f5d9"}
May 25 12:15:26.344577 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:26.344408    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7779bbdd78-mmnkt" event={"ID":"03d5c580-b34a-4949-ac56-082edfcb0b51","Type":"ContainerStarted","Data":"e436ebc6b0d266edd83a4fa5137ac93e0610092486cd2168d01cb57fcbb282f2"}
May 25 12:15:26.345655 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:26.345631    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7654684cf-5vvhl" event={"ID":"620aafc8-f648-4809-8abe-094c0f66f3de","Type":"ContainerStarted","Data":"056f1b05ec15a68360a22883a381de0431191d5cf5e58b08d76d06ef0a2b56ed"}
May 25 12:15:26.346611 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:26.346591    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-zf62d" event={"ID":"1124f1f1-fdc7-4a81-b28a-620db95e2ff6","Type":"ContainerStarted","Data":"f7d40774a1c38a663739886e94f4eedc859bb6c02ed5f8df5fed1b2b4058288b"}
May 25 12:15:26.384422 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:26.384371    2549 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-7779bbdd78-mmnkt" podStartSLOduration=1.532588113 podStartE2EDuration="3.384357051s" podCreationTimestamp="2026-05-25 12:15:23 +0000 UTC" firstStartedPulling="2026-05-25 12:15:24.104122795 +0000 UTC m=+127.819938063" lastFinishedPulling="2026-05-25 12:15:25.955891718 +0000 UTC m=+129.671707001" observedRunningTime="2026-05-25 12:15:26.362033118 +0000 UTC m=+130.077848404" watchObservedRunningTime="2026-05-25 12:15:26.384357051 +0000 UTC m=+130.100172337"
May 25 12:15:26.384626 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:26.384565    2549 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console/console-7654684cf-5vvhl" podStartSLOduration=1.045731358 podStartE2EDuration="4.384559941s" podCreationTimestamp="2026-05-25 12:15:22 +0000 UTC" firstStartedPulling="2026-05-25 12:15:22.615117897 +0000 UTC m=+126.330933162" lastFinishedPulling="2026-05-25 12:15:25.953946479 +0000 UTC m=+129.669761745" observedRunningTime="2026-05-25 12:15:26.383248194 +0000 UTC m=+130.099063482" watchObservedRunningTime="2026-05-25 12:15:26.384559941 +0000 UTC m=+130.100375228"
May 25 12:15:27.351059 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:27.351021    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-zf62d" event={"ID":"1124f1f1-fdc7-4a81-b28a-620db95e2ff6","Type":"ContainerStarted","Data":"886b3a6e8a56b742b37e7c230696c1bb654192055eec4fcfeb3902a354ccd195"}
May 25 12:15:27.351059 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:27.351061    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-zf62d" event={"ID":"1124f1f1-fdc7-4a81-b28a-620db95e2ff6","Type":"ContainerStarted","Data":"3aee7fd477b269a3012381153d067918af93a7f8300e54a02227e514195d4c05"}
May 25 12:15:27.371340 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:27.371238    2549 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-zf62d" podStartSLOduration=130.343905311 podStartE2EDuration="2m11.371224272s" podCreationTimestamp="2026-05-25 12:13:16 +0000 UTC" firstStartedPulling="2026-05-25 12:15:26.090188284 +0000 UTC m=+129.806003549" lastFinishedPulling="2026-05-25 12:15:27.117507236 +0000 UTC m=+130.833322510" observedRunningTime="2026-05-25 12:15:27.370626219 +0000 UTC m=+131.086441509" watchObservedRunningTime="2026-05-25 12:15:27.371224272 +0000 UTC m=+131.087039558"
May 25 12:15:28.024109 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:28.024073    2549 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-9c959559f-2s6h2"]
May 25 12:15:28.027615 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:28.027592    2549 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-54fd6bc5c4-c8zff"]
May 25 12:15:28.027801 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:28.027778    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-9c959559f-2s6h2"
May 25 12:15:28.030103 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:28.030084    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
May 25 12:15:28.030240 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:28.030089    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
May 25 12:15:28.030240 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:28.030139    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-pbxmr\""
May 25 12:15:28.031178 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:28.031155    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-c8zff"
May 25 12:15:28.033662 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:28.033582    2549 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
May 25 12:15:28.034054 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:28.034033    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
May 25 12:15:28.034295 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:28.034275    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
May 25 12:15:28.036732 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:28.036710    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-l9qkg\""
May 25 12:15:28.036901 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:28.036882    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-9c959559f-2s6h2"]
May 25 12:15:28.037998 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:28.037973    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-54fd6bc5c4-c8zff"]
May 25 12:15:28.040632 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:28.040605    2549 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-w4csw"]
May 25 12:15:28.044256 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:28.044239    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-w4csw"
May 25 12:15:28.046790 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:28.046766    2549 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
May 25 12:15:28.046896 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:28.046842    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-85pp4\""
May 25 12:15:28.047036 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:28.047020    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
May 25 12:15:28.047092 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:28.047066    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
May 25 12:15:28.177736 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:28.177698    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1b52309c-162a-413a-a514-801d0da5a5dd-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-54fd6bc5c4-c8zff\" (UID: \"1b52309c-162a-413a-a514-801d0da5a5dd\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-c8zff"
May 25 12:15:28.177897 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:28.177744    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8ddcb91f-25a2-4a72-b02a-19b0f85f855d-metrics-client-ca\") pod \"node-exporter-w4csw\" (UID: \"8ddcb91f-25a2-4a72-b02a-19b0f85f855d\") " pod="openshift-monitoring/node-exporter-w4csw"
May 25 12:15:28.177897 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:28.177827    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8ddcb91f-25a2-4a72-b02a-19b0f85f855d-node-exporter-textfile\") pod \"node-exporter-w4csw\" (UID: \"8ddcb91f-25a2-4a72-b02a-19b0f85f855d\") " pod="openshift-monitoring/node-exporter-w4csw"
May 25 12:15:28.177897 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:28.177852    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8ddcb91f-25a2-4a72-b02a-19b0f85f855d-node-exporter-tls\") pod \"node-exporter-w4csw\" (UID: \"8ddcb91f-25a2-4a72-b02a-19b0f85f855d\") " pod="openshift-monitoring/node-exporter-w4csw"
May 25 12:15:28.177897 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:28.177872    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8ddcb91f-25a2-4a72-b02a-19b0f85f855d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-w4csw\" (UID: \"8ddcb91f-25a2-4a72-b02a-19b0f85f855d\") " pod="openshift-monitoring/node-exporter-w4csw"
May 25 12:15:28.178026 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:28.177946    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/1b52309c-162a-413a-a514-801d0da5a5dd-kube-state-metrics-tls\") pod \"kube-state-metrics-54fd6bc5c4-c8zff\" (UID: \"1b52309c-162a-413a-a514-801d0da5a5dd\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-c8zff"
May 25 12:15:28.178026 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:28.177974    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-npmp8\" (UniqueName: \"kubernetes.io/projected/42ecc201-f5ef-489e-8eb3-775f91385c17-kube-api-access-npmp8\") pod \"openshift-state-metrics-9c959559f-2s6h2\" (UID: \"42ecc201-f5ef-489e-8eb3-775f91385c17\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-2s6h2"
May 25 12:15:28.178026 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:28.178001    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/42ecc201-f5ef-489e-8eb3-775f91385c17-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-9c959559f-2s6h2\" (UID: \"42ecc201-f5ef-489e-8eb3-775f91385c17\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-2s6h2"
May 25 12:15:28.178117 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:28.178027    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/42ecc201-f5ef-489e-8eb3-775f91385c17-metrics-client-ca\") pod \"openshift-state-metrics-9c959559f-2s6h2\" (UID: \"42ecc201-f5ef-489e-8eb3-775f91385c17\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-2s6h2"
May 25 12:15:28.178117 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:28.178059    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8ddcb91f-25a2-4a72-b02a-19b0f85f855d-node-exporter-wtmp\") pod \"node-exporter-w4csw\" (UID: \"8ddcb91f-25a2-4a72-b02a-19b0f85f855d\") " pod="openshift-monitoring/node-exporter-w4csw"
May 25 12:15:28.178117 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:28.178083    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2g6r6\" (UniqueName: \"kubernetes.io/projected/8ddcb91f-25a2-4a72-b02a-19b0f85f855d-kube-api-access-2g6r6\") pod \"node-exporter-w4csw\" (UID: \"8ddcb91f-25a2-4a72-b02a-19b0f85f855d\") " pod="openshift-monitoring/node-exporter-w4csw"
May 25 12:15:28.178237 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:28.178129    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/1b52309c-162a-413a-a514-801d0da5a5dd-volume-directive-shadow\") pod \"kube-state-metrics-54fd6bc5c4-c8zff\" (UID: \"1b52309c-162a-413a-a514-801d0da5a5dd\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-c8zff"
May 25 12:15:28.178237 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:28.178153    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-llx28\" (UniqueName: \"kubernetes.io/projected/1b52309c-162a-413a-a514-801d0da5a5dd-kube-api-access-llx28\") pod \"kube-state-metrics-54fd6bc5c4-c8zff\" (UID: \"1b52309c-162a-413a-a514-801d0da5a5dd\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-c8zff"
May 25 12:15:28.178237 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:28.178189    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1b52309c-162a-413a-a514-801d0da5a5dd-metrics-client-ca\") pod \"kube-state-metrics-54fd6bc5c4-c8zff\" (UID: \"1b52309c-162a-413a-a514-801d0da5a5dd\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-c8zff"
May 25 12:15:28.178352 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:28.178233    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8ddcb91f-25a2-4a72-b02a-19b0f85f855d-node-exporter-accelerators-collector-config\") pod \"node-exporter-w4csw\" (UID: \"8ddcb91f-25a2-4a72-b02a-19b0f85f855d\") " pod="openshift-monitoring/node-exporter-w4csw"
May 25 12:15:28.178352 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:28.178281    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8ddcb91f-25a2-4a72-b02a-19b0f85f855d-root\") pod \"node-exporter-w4csw\" (UID: \"8ddcb91f-25a2-4a72-b02a-19b0f85f855d\") " pod="openshift-monitoring/node-exporter-w4csw"
May 25 12:15:28.178352 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:28.178307    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/1b52309c-162a-413a-a514-801d0da5a5dd-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-54fd6bc5c4-c8zff\" (UID: \"1b52309c-162a-413a-a514-801d0da5a5dd\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-c8zff"
May 25 12:15:28.178352 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:28.178333    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/42ecc201-f5ef-489e-8eb3-775f91385c17-openshift-state-metrics-tls\") pod \"openshift-state-metrics-9c959559f-2s6h2\" (UID: \"42ecc201-f5ef-489e-8eb3-775f91385c17\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-2s6h2"
May 25 12:15:28.178352 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:28.178356    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8ddcb91f-25a2-4a72-b02a-19b0f85f855d-sys\") pod \"node-exporter-w4csw\" (UID: \"8ddcb91f-25a2-4a72-b02a-19b0f85f855d\") " pod="openshift-monitoring/node-exporter-w4csw"
May 25 12:15:28.279163 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:28.279074    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/1b52309c-162a-413a-a514-801d0da5a5dd-kube-state-metrics-tls\") pod \"kube-state-metrics-54fd6bc5c4-c8zff\" (UID: \"1b52309c-162a-413a-a514-801d0da5a5dd\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-c8zff"
May 25 12:15:28.279163 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:28.279118    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-npmp8\" (UniqueName: \"kubernetes.io/projected/42ecc201-f5ef-489e-8eb3-775f91385c17-kube-api-access-npmp8\") pod \"openshift-state-metrics-9c959559f-2s6h2\" (UID: \"42ecc201-f5ef-489e-8eb3-775f91385c17\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-2s6h2"
May 25 12:15:28.279163 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:28.279146    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/42ecc201-f5ef-489e-8eb3-775f91385c17-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-9c959559f-2s6h2\" (UID: \"42ecc201-f5ef-489e-8eb3-775f91385c17\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-2s6h2"
May 25 12:15:28.279459 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:28.279174    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/42ecc201-f5ef-489e-8eb3-775f91385c17-metrics-client-ca\") pod \"openshift-state-metrics-9c959559f-2s6h2\" (UID: \"42ecc201-f5ef-489e-8eb3-775f91385c17\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-2s6h2"
May 25 12:15:28.279459 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:15:28.279233    2549 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found
May 25 12:15:28.279459 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:15:28.279299    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1b52309c-162a-413a-a514-801d0da5a5dd-kube-state-metrics-tls podName:1b52309c-162a-413a-a514-801d0da5a5dd nodeName:}" failed. No retries permitted until 2026-05-25 12:15:28.779283407 +0000 UTC m=+132.495098672 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/1b52309c-162a-413a-a514-801d0da5a5dd-kube-state-metrics-tls") pod "kube-state-metrics-54fd6bc5c4-c8zff" (UID: "1b52309c-162a-413a-a514-801d0da5a5dd") : secret "kube-state-metrics-tls" not found
May 25 12:15:28.279459 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:28.279340    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8ddcb91f-25a2-4a72-b02a-19b0f85f855d-node-exporter-wtmp\") pod \"node-exporter-w4csw\" (UID: \"8ddcb91f-25a2-4a72-b02a-19b0f85f855d\") " pod="openshift-monitoring/node-exporter-w4csw"
May 25 12:15:28.279459 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:28.279372    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-2g6r6\" (UniqueName: \"kubernetes.io/projected/8ddcb91f-25a2-4a72-b02a-19b0f85f855d-kube-api-access-2g6r6\") pod \"node-exporter-w4csw\" (UID: \"8ddcb91f-25a2-4a72-b02a-19b0f85f855d\") " pod="openshift-monitoring/node-exporter-w4csw"
May 25 12:15:28.279459 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:28.279409    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/1b52309c-162a-413a-a514-801d0da5a5dd-volume-directive-shadow\") pod \"kube-state-metrics-54fd6bc5c4-c8zff\" (UID: \"1b52309c-162a-413a-a514-801d0da5a5dd\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-c8zff"
May 25 12:15:28.279459 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:28.279441    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-llx28\" (UniqueName: \"kubernetes.io/projected/1b52309c-162a-413a-a514-801d0da5a5dd-kube-api-access-llx28\") pod \"kube-state-metrics-54fd6bc5c4-c8zff\" (UID: \"1b52309c-162a-413a-a514-801d0da5a5dd\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-c8zff"
May 25 12:15:28.279881 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:28.279490    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1b52309c-162a-413a-a514-801d0da5a5dd-metrics-client-ca\") pod \"kube-state-metrics-54fd6bc5c4-c8zff\" (UID: \"1b52309c-162a-413a-a514-801d0da5a5dd\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-c8zff"
May 25 12:15:28.279881 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:28.279535    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8ddcb91f-25a2-4a72-b02a-19b0f85f855d-node-exporter-accelerators-collector-config\") pod \"node-exporter-w4csw\" (UID: \"8ddcb91f-25a2-4a72-b02a-19b0f85f855d\") " pod="openshift-monitoring/node-exporter-w4csw"
May 25 12:15:28.279881 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:28.279577    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8ddcb91f-25a2-4a72-b02a-19b0f85f855d-root\") pod \"node-exporter-w4csw\" (UID: \"8ddcb91f-25a2-4a72-b02a-19b0f85f855d\") " pod="openshift-monitoring/node-exporter-w4csw"
May 25 12:15:28.279881 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:28.279614    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/1b52309c-162a-413a-a514-801d0da5a5dd-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-54fd6bc5c4-c8zff\" (UID: \"1b52309c-162a-413a-a514-801d0da5a5dd\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-c8zff"
May 25 12:15:28.279881 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:28.279657    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/42ecc201-f5ef-489e-8eb3-775f91385c17-openshift-state-metrics-tls\") pod \"openshift-state-metrics-9c959559f-2s6h2\" (UID: \"42ecc201-f5ef-489e-8eb3-775f91385c17\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-2s6h2"
May 25 12:15:28.279881 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:28.279679    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8ddcb91f-25a2-4a72-b02a-19b0f85f855d-sys\") pod \"node-exporter-w4csw\" (UID: \"8ddcb91f-25a2-4a72-b02a-19b0f85f855d\") " pod="openshift-monitoring/node-exporter-w4csw"
May 25 12:15:28.279881 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:28.279711    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1b52309c-162a-413a-a514-801d0da5a5dd-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-54fd6bc5c4-c8zff\" (UID: \"1b52309c-162a-413a-a514-801d0da5a5dd\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-c8zff"
May 25 12:15:28.279881 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:28.279743    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8ddcb91f-25a2-4a72-b02a-19b0f85f855d-metrics-client-ca\") pod \"node-exporter-w4csw\" (UID: \"8ddcb91f-25a2-4a72-b02a-19b0f85f855d\") " pod="openshift-monitoring/node-exporter-w4csw"
May 25 12:15:28.279881 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:28.279788    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8ddcb91f-25a2-4a72-b02a-19b0f85f855d-node-exporter-textfile\") pod \"node-exporter-w4csw\" (UID: \"8ddcb91f-25a2-4a72-b02a-19b0f85f855d\") " pod="openshift-monitoring/node-exporter-w4csw"
May 25 12:15:28.279881 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:28.279814    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8ddcb91f-25a2-4a72-b02a-19b0f85f855d-node-exporter-tls\") pod \"node-exporter-w4csw\" (UID: \"8ddcb91f-25a2-4a72-b02a-19b0f85f855d\") " pod="openshift-monitoring/node-exporter-w4csw"
May 25 12:15:28.279881 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:28.279839    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8ddcb91f-25a2-4a72-b02a-19b0f85f855d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-w4csw\" (UID: \"8ddcb91f-25a2-4a72-b02a-19b0f85f855d\") " pod="openshift-monitoring/node-exporter-w4csw"
May 25 12:15:28.279881 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:28.279846    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8ddcb91f-25a2-4a72-b02a-19b0f85f855d-sys\") pod \"node-exporter-w4csw\" (UID: \"8ddcb91f-25a2-4a72-b02a-19b0f85f855d\") " pod="openshift-monitoring/node-exporter-w4csw"
May 25 12:15:28.279881 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:28.279531    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8ddcb91f-25a2-4a72-b02a-19b0f85f855d-node-exporter-wtmp\") pod \"node-exporter-w4csw\" (UID: \"8ddcb91f-25a2-4a72-b02a-19b0f85f855d\") " pod="openshift-monitoring/node-exporter-w4csw"
May 25 12:15:28.280524 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:28.279927    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8ddcb91f-25a2-4a72-b02a-19b0f85f855d-root\") pod \"node-exporter-w4csw\" (UID: \"8ddcb91f-25a2-4a72-b02a-19b0f85f855d\") " pod="openshift-monitoring/node-exporter-w4csw"
May 25 12:15:28.280524 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:28.279998    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/42ecc201-f5ef-489e-8eb3-775f91385c17-metrics-client-ca\") pod \"openshift-state-metrics-9c959559f-2s6h2\" (UID: \"42ecc201-f5ef-489e-8eb3-775f91385c17\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-2s6h2"
May 25 12:15:28.280524 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:28.280262    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8ddcb91f-25a2-4a72-b02a-19b0f85f855d-node-exporter-accelerators-collector-config\") pod \"node-exporter-w4csw\" (UID: \"8ddcb91f-25a2-4a72-b02a-19b0f85f855d\") " pod="openshift-monitoring/node-exporter-w4csw"
May 25 12:15:28.280524 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:28.280405    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/1b52309c-162a-413a-a514-801d0da5a5dd-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-54fd6bc5c4-c8zff\" (UID: \"1b52309c-162a-413a-a514-801d0da5a5dd\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-c8zff"
May 25 12:15:28.280524 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:15:28.280489    2549 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found
May 25 12:15:28.280771 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:28.280536    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8ddcb91f-25a2-4a72-b02a-19b0f85f855d-node-exporter-textfile\") pod \"node-exporter-w4csw\" (UID: \"8ddcb91f-25a2-4a72-b02a-19b0f85f855d\") " pod="openshift-monitoring/node-exporter-w4csw"
May 25 12:15:28.280771 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:15:28.280553    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/42ecc201-f5ef-489e-8eb3-775f91385c17-openshift-state-metrics-tls podName:42ecc201-f5ef-489e-8eb3-775f91385c17 nodeName:}" failed. No retries permitted until 2026-05-25 12:15:28.780537866 +0000 UTC m=+132.496353145 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/42ecc201-f5ef-489e-8eb3-775f91385c17-openshift-state-metrics-tls") pod "openshift-state-metrics-9c959559f-2s6h2" (UID: "42ecc201-f5ef-489e-8eb3-775f91385c17") : secret "openshift-state-metrics-tls" not found
May 25 12:15:28.280771 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:28.280566    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8ddcb91f-25a2-4a72-b02a-19b0f85f855d-metrics-client-ca\") pod \"node-exporter-w4csw\" (UID: \"8ddcb91f-25a2-4a72-b02a-19b0f85f855d\") " pod="openshift-monitoring/node-exporter-w4csw"
May 25 12:15:28.280771 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:28.280712    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/1b52309c-162a-413a-a514-801d0da5a5dd-volume-directive-shadow\") pod \"kube-state-metrics-54fd6bc5c4-c8zff\" (UID: \"1b52309c-162a-413a-a514-801d0da5a5dd\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-c8zff"
May 25 12:15:28.281119 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:28.281095    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1b52309c-162a-413a-a514-801d0da5a5dd-metrics-client-ca\") pod \"kube-state-metrics-54fd6bc5c4-c8zff\" (UID: \"1b52309c-162a-413a-a514-801d0da5a5dd\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-c8zff"
May 25 12:15:28.282650 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:28.282622    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8ddcb91f-25a2-4a72-b02a-19b0f85f855d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-w4csw\" (UID: \"8ddcb91f-25a2-4a72-b02a-19b0f85f855d\") " pod="openshift-monitoring/node-exporter-w4csw"
May 25 12:15:28.282789 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:28.282770    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1b52309c-162a-413a-a514-801d0da5a5dd-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-54fd6bc5c4-c8zff\" (UID: \"1b52309c-162a-413a-a514-801d0da5a5dd\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-c8zff"
May 25 12:15:28.282977 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:28.282957    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8ddcb91f-25a2-4a72-b02a-19b0f85f855d-node-exporter-tls\") pod \"node-exporter-w4csw\" (UID: \"8ddcb91f-25a2-4a72-b02a-19b0f85f855d\") " pod="openshift-monitoring/node-exporter-w4csw"
May 25 12:15:28.283154 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:28.283117    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/42ecc201-f5ef-489e-8eb3-775f91385c17-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-9c959559f-2s6h2\" (UID: \"42ecc201-f5ef-489e-8eb3-775f91385c17\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-2s6h2"
May 25 12:15:28.288826 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:28.288800    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-llx28\" (UniqueName: \"kubernetes.io/projected/1b52309c-162a-413a-a514-801d0da5a5dd-kube-api-access-llx28\") pod \"kube-state-metrics-54fd6bc5c4-c8zff\" (UID: \"1b52309c-162a-413a-a514-801d0da5a5dd\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-c8zff"
May 25 12:15:28.288947 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:28.288926    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2g6r6\" (UniqueName: \"kubernetes.io/projected/8ddcb91f-25a2-4a72-b02a-19b0f85f855d-kube-api-access-2g6r6\") pod \"node-exporter-w4csw\" (UID: \"8ddcb91f-25a2-4a72-b02a-19b0f85f855d\") " pod="openshift-monitoring/node-exporter-w4csw"
May 25 12:15:28.289012 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:28.288942    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-npmp8\" (UniqueName: \"kubernetes.io/projected/42ecc201-f5ef-489e-8eb3-775f91385c17-kube-api-access-npmp8\") pod \"openshift-state-metrics-9c959559f-2s6h2\" (UID: \"42ecc201-f5ef-489e-8eb3-775f91385c17\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-2s6h2"
May 25 12:15:28.353784 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:28.353748    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-w4csw"
May 25 12:15:28.363439 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:15:28.363414    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ddcb91f_25a2_4a72_b02a_19b0f85f855d.slice/crio-845c366c91108e740e3404e6970c14905e9fb3b2a9ac2ce05a6301682bd04bdd WatchSource:0}: Error finding container 845c366c91108e740e3404e6970c14905e9fb3b2a9ac2ce05a6301682bd04bdd: Status 404 returned error can't find the container with id 845c366c91108e740e3404e6970c14905e9fb3b2a9ac2ce05a6301682bd04bdd
May 25 12:15:28.784446 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:28.784413    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/42ecc201-f5ef-489e-8eb3-775f91385c17-openshift-state-metrics-tls\") pod \"openshift-state-metrics-9c959559f-2s6h2\" (UID: \"42ecc201-f5ef-489e-8eb3-775f91385c17\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-2s6h2"
May 25 12:15:28.784609 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:28.784472    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/1b52309c-162a-413a-a514-801d0da5a5dd-kube-state-metrics-tls\") pod \"kube-state-metrics-54fd6bc5c4-c8zff\" (UID: \"1b52309c-162a-413a-a514-801d0da5a5dd\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-c8zff"
May 25 12:15:28.786787 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:28.786756    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/1b52309c-162a-413a-a514-801d0da5a5dd-kube-state-metrics-tls\") pod \"kube-state-metrics-54fd6bc5c4-c8zff\" (UID: \"1b52309c-162a-413a-a514-801d0da5a5dd\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-c8zff"
May 25 12:15:28.786954 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:28.786932    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/42ecc201-f5ef-489e-8eb3-775f91385c17-openshift-state-metrics-tls\") pod \"openshift-state-metrics-9c959559f-2s6h2\" (UID: \"42ecc201-f5ef-489e-8eb3-775f91385c17\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-2s6h2"
May 25 12:15:28.940556 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:28.940518    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-9c959559f-2s6h2"
May 25 12:15:28.946267 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:28.946242    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-c8zff"
May 25 12:15:29.099683 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:29.099646    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-9c959559f-2s6h2"]
May 25 12:15:29.124977 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:29.124946    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-54fd6bc5c4-c8zff"]
May 25 12:15:29.292850 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:15:29.292779    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42ecc201_f5ef_489e_8eb3_775f91385c17.slice/crio-be3651a93eaa2631a3889ec0c82416828a44820ef05c48db7bfe22d66e30e325 WatchSource:0}: Error finding container be3651a93eaa2631a3889ec0c82416828a44820ef05c48db7bfe22d66e30e325: Status 404 returned error can't find the container with id be3651a93eaa2631a3889ec0c82416828a44820ef05c48db7bfe22d66e30e325
May 25 12:15:29.293438 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:15:29.293415    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b52309c_162a_413a_a514_801d0da5a5dd.slice/crio-d1d82fb84b2d66e0ab25317ab0998ad86584cd0784a0cdf2ed287d36b2a4def7 WatchSource:0}: Error finding container d1d82fb84b2d66e0ab25317ab0998ad86584cd0784a0cdf2ed287d36b2a4def7: Status 404 returned error can't find the container with id d1d82fb84b2d66e0ab25317ab0998ad86584cd0784a0cdf2ed287d36b2a4def7
May 25 12:15:29.366968 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:29.366926    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-c8zff" event={"ID":"1b52309c-162a-413a-a514-801d0da5a5dd","Type":"ContainerStarted","Data":"d1d82fb84b2d66e0ab25317ab0998ad86584cd0784a0cdf2ed287d36b2a4def7"}
May 25 12:15:29.368523 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:29.368490    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-9c959559f-2s6h2" event={"ID":"42ecc201-f5ef-489e-8eb3-775f91385c17","Type":"ContainerStarted","Data":"2e3c9b610820e78f1acdbe79285a66c28b6ce10031caf95d8dbcd73a9dee39fc"}
May 25 12:15:29.368641 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:29.368528    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-9c959559f-2s6h2" event={"ID":"42ecc201-f5ef-489e-8eb3-775f91385c17","Type":"ContainerStarted","Data":"be3651a93eaa2631a3889ec0c82416828a44820ef05c48db7bfe22d66e30e325"}
May 25 12:15:29.369848 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:29.369785    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-w4csw" event={"ID":"8ddcb91f-25a2-4a72-b02a-19b0f85f855d","Type":"ContainerStarted","Data":"845c366c91108e740e3404e6970c14905e9fb3b2a9ac2ce05a6301682bd04bdd"}
May 25 12:15:30.015726 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:30.015669    2549 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-5b67cb8f7-zkltg"]
May 25 12:15:30.019808 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:30.019729    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5b67cb8f7-zkltg"
May 25 12:15:30.022593 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:30.022572    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
May 25 12:15:30.023389 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:30.022745    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
May 25 12:15:30.023389 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:30.022798    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
May 25 12:15:30.023389 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:30.022625    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
May 25 12:15:30.023389 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:30.022866    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
May 25 12:15:30.023389 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:30.023272    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-tgxb8\""
May 25 12:15:30.023731 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:30.023555    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-eb6q1k8v4n2rj\""
May 25 12:15:30.030250 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:30.030106    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5b67cb8f7-zkltg"]
May 25 12:15:30.094608 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:30.094566    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/83c8cfdf-2cea-45d9-832b-b4172368763f-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5b67cb8f7-zkltg\" (UID: \"83c8cfdf-2cea-45d9-832b-b4172368763f\") " pod="openshift-monitoring/thanos-querier-5b67cb8f7-zkltg"
May 25 12:15:30.094608 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:30.094613    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/83c8cfdf-2cea-45d9-832b-b4172368763f-metrics-client-ca\") pod \"thanos-querier-5b67cb8f7-zkltg\" (UID: \"83c8cfdf-2cea-45d9-832b-b4172368763f\") " pod="openshift-monitoring/thanos-querier-5b67cb8f7-zkltg"
May 25 12:15:30.094845 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:30.094664    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxh9n\" (UniqueName: \"kubernetes.io/projected/83c8cfdf-2cea-45d9-832b-b4172368763f-kube-api-access-cxh9n\") pod \"thanos-querier-5b67cb8f7-zkltg\" (UID: \"83c8cfdf-2cea-45d9-832b-b4172368763f\") " pod="openshift-monitoring/thanos-querier-5b67cb8f7-zkltg"
May 25 12:15:30.094845 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:30.094709    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/83c8cfdf-2cea-45d9-832b-b4172368763f-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5b67cb8f7-zkltg\" (UID: \"83c8cfdf-2cea-45d9-832b-b4172368763f\") " pod="openshift-monitoring/thanos-querier-5b67cb8f7-zkltg"
May 25 12:15:30.094845 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:30.094757    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/83c8cfdf-2cea-45d9-832b-b4172368763f-secret-thanos-querier-tls\") pod \"thanos-querier-5b67cb8f7-zkltg\" (UID: \"83c8cfdf-2cea-45d9-832b-b4172368763f\") " pod="openshift-monitoring/thanos-querier-5b67cb8f7-zkltg"
May 25 12:15:30.095018 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:30.094862    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/83c8cfdf-2cea-45d9-832b-b4172368763f-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5b67cb8f7-zkltg\" (UID: \"83c8cfdf-2cea-45d9-832b-b4172368763f\") " pod="openshift-monitoring/thanos-querier-5b67cb8f7-zkltg"
May 25 12:15:30.095018 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:30.094908    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/83c8cfdf-2cea-45d9-832b-b4172368763f-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5b67cb8f7-zkltg\" (UID: \"83c8cfdf-2cea-45d9-832b-b4172368763f\") " pod="openshift-monitoring/thanos-querier-5b67cb8f7-zkltg"
May 25 12:15:30.095018 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:30.094940    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/83c8cfdf-2cea-45d9-832b-b4172368763f-secret-grpc-tls\") pod \"thanos-querier-5b67cb8f7-zkltg\" (UID: \"83c8cfdf-2cea-45d9-832b-b4172368763f\") " pod="openshift-monitoring/thanos-querier-5b67cb8f7-zkltg"
May 25 12:15:30.196305 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:30.196272    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/83c8cfdf-2cea-45d9-832b-b4172368763f-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5b67cb8f7-zkltg\" (UID: \"83c8cfdf-2cea-45d9-832b-b4172368763f\") " pod="openshift-monitoring/thanos-querier-5b67cb8f7-zkltg"
May 25 12:15:30.196492 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:30.196311    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/83c8cfdf-2cea-45d9-832b-b4172368763f-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5b67cb8f7-zkltg\" (UID: \"83c8cfdf-2cea-45d9-832b-b4172368763f\") " pod="openshift-monitoring/thanos-querier-5b67cb8f7-zkltg"
May 25 12:15:30.196492 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:30.196339    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/83c8cfdf-2cea-45d9-832b-b4172368763f-secret-grpc-tls\") pod \"thanos-querier-5b67cb8f7-zkltg\" (UID: \"83c8cfdf-2cea-45d9-832b-b4172368763f\") " pod="openshift-monitoring/thanos-querier-5b67cb8f7-zkltg"
May 25 12:15:30.196492 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:30.196394    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/83c8cfdf-2cea-45d9-832b-b4172368763f-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5b67cb8f7-zkltg\" (UID: \"83c8cfdf-2cea-45d9-832b-b4172368763f\") " pod="openshift-monitoring/thanos-querier-5b67cb8f7-zkltg"
May 25 12:15:30.196492 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:30.196426    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/83c8cfdf-2cea-45d9-832b-b4172368763f-metrics-client-ca\") pod \"thanos-querier-5b67cb8f7-zkltg\" (UID: \"83c8cfdf-2cea-45d9-832b-b4172368763f\") " pod="openshift-monitoring/thanos-querier-5b67cb8f7-zkltg"
May 25 12:15:30.196492 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:30.196451    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-cxh9n\" (UniqueName: \"kubernetes.io/projected/83c8cfdf-2cea-45d9-832b-b4172368763f-kube-api-access-cxh9n\") pod \"thanos-querier-5b67cb8f7-zkltg\" (UID: \"83c8cfdf-2cea-45d9-832b-b4172368763f\") " pod="openshift-monitoring/thanos-querier-5b67cb8f7-zkltg"
May 25 12:15:30.196492 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:30.196484    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/83c8cfdf-2cea-45d9-832b-b4172368763f-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5b67cb8f7-zkltg\" (UID: \"83c8cfdf-2cea-45d9-832b-b4172368763f\") " pod="openshift-monitoring/thanos-querier-5b67cb8f7-zkltg"
May 25 12:15:30.196811 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:30.196535    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/83c8cfdf-2cea-45d9-832b-b4172368763f-secret-thanos-querier-tls\") pod \"thanos-querier-5b67cb8f7-zkltg\" (UID: \"83c8cfdf-2cea-45d9-832b-b4172368763f\") " pod="openshift-monitoring/thanos-querier-5b67cb8f7-zkltg"
May 25 12:15:30.197369 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:30.197317    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/83c8cfdf-2cea-45d9-832b-b4172368763f-metrics-client-ca\") pod \"thanos-querier-5b67cb8f7-zkltg\" (UID: \"83c8cfdf-2cea-45d9-832b-b4172368763f\") " pod="openshift-monitoring/thanos-querier-5b67cb8f7-zkltg"
May 25 12:15:30.199352 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:30.199326    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/83c8cfdf-2cea-45d9-832b-b4172368763f-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5b67cb8f7-zkltg\" (UID: \"83c8cfdf-2cea-45d9-832b-b4172368763f\") " pod="openshift-monitoring/thanos-querier-5b67cb8f7-zkltg"
May 25 12:15:30.199472 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:30.199424    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/83c8cfdf-2cea-45d9-832b-b4172368763f-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5b67cb8f7-zkltg\" (UID: \"83c8cfdf-2cea-45d9-832b-b4172368763f\") " pod="openshift-monitoring/thanos-querier-5b67cb8f7-zkltg"
May 25 12:15:30.199612 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:30.199570    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/83c8cfdf-2cea-45d9-832b-b4172368763f-secret-thanos-querier-tls\") pod \"thanos-querier-5b67cb8f7-zkltg\" (UID: \"83c8cfdf-2cea-45d9-832b-b4172368763f\") " pod="openshift-monitoring/thanos-querier-5b67cb8f7-zkltg"
May 25 12:15:30.199612 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:30.199594    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/83c8cfdf-2cea-45d9-832b-b4172368763f-secret-grpc-tls\") pod \"thanos-querier-5b67cb8f7-zkltg\" (UID: \"83c8cfdf-2cea-45d9-832b-b4172368763f\") " pod="openshift-monitoring/thanos-querier-5b67cb8f7-zkltg"
May 25 12:15:30.199785 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:30.199761    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/83c8cfdf-2cea-45d9-832b-b4172368763f-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5b67cb8f7-zkltg\" (UID: \"83c8cfdf-2cea-45d9-832b-b4172368763f\") " pod="openshift-monitoring/thanos-querier-5b67cb8f7-zkltg"
May 25 12:15:30.199863 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:30.199848    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/83c8cfdf-2cea-45d9-832b-b4172368763f-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5b67cb8f7-zkltg\" (UID: \"83c8cfdf-2cea-45d9-832b-b4172368763f\") " pod="openshift-monitoring/thanos-querier-5b67cb8f7-zkltg"
May 25 12:15:30.206001 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:30.205979    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxh9n\" (UniqueName: \"kubernetes.io/projected/83c8cfdf-2cea-45d9-832b-b4172368763f-kube-api-access-cxh9n\") pod \"thanos-querier-5b67cb8f7-zkltg\" (UID: \"83c8cfdf-2cea-45d9-832b-b4172368763f\") " pod="openshift-monitoring/thanos-querier-5b67cb8f7-zkltg"
May 25 12:15:30.332633 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:30.332603    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5b67cb8f7-zkltg"
May 25 12:15:30.376269 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:30.375347    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-9c959559f-2s6h2" event={"ID":"42ecc201-f5ef-489e-8eb3-775f91385c17","Type":"ContainerStarted","Data":"789e6853343d0b54031088e4c74b88340a6b4eea380c1f1a651eb495db2cae35"}
May 25 12:15:30.377106 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:30.377075    2549 generic.go:358] "Generic (PLEG): container finished" podID="8ddcb91f-25a2-4a72-b02a-19b0f85f855d" containerID="c5afab60823b5d997dea65624fc2544daaaef598e7ef30a6a8dedc7b2c63a077" exitCode=0
May 25 12:15:30.377240 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:30.377173    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-w4csw" event={"ID":"8ddcb91f-25a2-4a72-b02a-19b0f85f855d","Type":"ContainerDied","Data":"c5afab60823b5d997dea65624fc2544daaaef598e7ef30a6a8dedc7b2c63a077"}
May 25 12:15:30.838425 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:30.838159    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5b67cb8f7-zkltg"]
May 25 12:15:30.841314 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:15:30.841281    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83c8cfdf_2cea_45d9_832b_b4172368763f.slice/crio-440e819b33d24744440fe7d633a9da55f26fb105d663c4d37841735b095a4a33 WatchSource:0}: Error finding container 440e819b33d24744440fe7d633a9da55f26fb105d663c4d37841735b095a4a33: Status 404 returned error can't find the container with id 440e819b33d24744440fe7d633a9da55f26fb105d663c4d37841735b095a4a33
May 25 12:15:31.384369 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:31.384317    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-c8zff" event={"ID":"1b52309c-162a-413a-a514-801d0da5a5dd","Type":"ContainerStarted","Data":"6f07bf47439bea1b367c25a4a3be26e0e22811ffb50d87d32102e2a88bc2b4c2"}
May 25 12:15:31.384369 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:31.384373    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-c8zff" event={"ID":"1b52309c-162a-413a-a514-801d0da5a5dd","Type":"ContainerStarted","Data":"5f5bf9b18dd8077619d98ebceab2f848fa1f28c15ca68943c57de4cee2cdf60a"}
May 25 12:15:31.384873 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:31.384387    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-c8zff" event={"ID":"1b52309c-162a-413a-a514-801d0da5a5dd","Type":"ContainerStarted","Data":"429559d11372e3a0a7ea8d6887b600ab16f8c10b4d149ecec0025ea226b3f394"}
May 25 12:15:31.386626 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:31.386594    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-9c959559f-2s6h2" event={"ID":"42ecc201-f5ef-489e-8eb3-775f91385c17","Type":"ContainerStarted","Data":"4d8023d7009f9dd158e7e77da50a050921b7521fb778b0787d9f10903862699c"}
May 25 12:15:31.388984 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:31.388958    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-w4csw" event={"ID":"8ddcb91f-25a2-4a72-b02a-19b0f85f855d","Type":"ContainerStarted","Data":"ad47c6495214462543791c8c4d38848cac4c8634af7bb5d7fe8b2b69a0d5cddf"}
May 25 12:15:31.389094 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:31.388991    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-w4csw" event={"ID":"8ddcb91f-25a2-4a72-b02a-19b0f85f855d","Type":"ContainerStarted","Data":"cfa677d82295589cd6e514cc08f5d0e82056287d2886f8dcd712ab4fe8361025"}
May 25 12:15:31.390061 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:31.390039    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5b67cb8f7-zkltg" event={"ID":"83c8cfdf-2cea-45d9-832b-b4172368763f","Type":"ContainerStarted","Data":"440e819b33d24744440fe7d633a9da55f26fb105d663c4d37841735b095a4a33"}
May 25 12:15:31.410029 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:31.409975    2549 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-c8zff" podStartSLOduration=2.01254907 podStartE2EDuration="3.40996064s" podCreationTimestamp="2026-05-25 12:15:28 +0000 UTC" firstStartedPulling="2026-05-25 12:15:29.295393527 +0000 UTC m=+133.011208810" lastFinishedPulling="2026-05-25 12:15:30.6928051 +0000 UTC m=+134.408620380" observedRunningTime="2026-05-25 12:15:31.408637203 +0000 UTC m=+135.124452491" watchObservedRunningTime="2026-05-25 12:15:31.40996064 +0000 UTC m=+135.125775929"
May 25 12:15:31.427513 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:31.427467    2549 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-9c959559f-2s6h2" podStartSLOduration=2.150405399 podStartE2EDuration="3.427452265s" podCreationTimestamp="2026-05-25 12:15:28 +0000 UTC" firstStartedPulling="2026-05-25 12:15:29.412801728 +0000 UTC m=+133.128616994" lastFinishedPulling="2026-05-25 12:15:30.689848579 +0000 UTC m=+134.405663860" observedRunningTime="2026-05-25 12:15:31.426605125 +0000 UTC m=+135.142420414" watchObservedRunningTime="2026-05-25 12:15:31.427452265 +0000 UTC m=+135.143267553"
May 25 12:15:31.451243 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:31.451177    2549 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-w4csw" podStartSLOduration=2.474139184 podStartE2EDuration="3.451163812s" podCreationTimestamp="2026-05-25 12:15:28 +0000 UTC" firstStartedPulling="2026-05-25 12:15:28.365035419 +0000 UTC m=+132.080850698" lastFinishedPulling="2026-05-25 12:15:29.342060045 +0000 UTC m=+133.057875326" observedRunningTime="2026-05-25 12:15:31.449764727 +0000 UTC m=+135.165580015" watchObservedRunningTime="2026-05-25 12:15:31.451163812 +0000 UTC m=+135.166979098"
May 25 12:15:32.474145 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:32.474111    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7654684cf-5vvhl"
May 25 12:15:32.474581 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:32.474160    2549 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7654684cf-5vvhl"
May 25 12:15:32.475585 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:32.475561    2549 patch_prober.go:28] interesting pod/console-7654684cf-5vvhl container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.134.0.16:8443/health\": dial tcp 10.134.0.16:8443: connect: connection refused" start-of-body=
May 25 12:15:32.475692 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:32.475619    2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-console/console-7654684cf-5vvhl" podUID="620aafc8-f648-4809-8abe-094c0f66f3de" containerName="console" probeResult="failure" output="Get \"https://10.134.0.16:8443/health\": dial tcp 10.134.0.16:8443: connect: connection refused"
May 25 12:15:32.808708 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:32.808563    2549 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-665c9c8584-dpgd2"]
May 25 12:15:32.813009 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:32.812988    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-665c9c8584-dpgd2"
May 25 12:15:32.815567 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:32.815547    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
May 25 12:15:32.815645 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:32.815551    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"default-dockercfg-fg7d9\""
May 25 12:15:32.819646 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:32.819604    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-665c9c8584-dpgd2"]
May 25 12:15:32.922010 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:32.921991    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/891d3439-8489-46f4-9612-dbce761a67c6-monitoring-plugin-cert\") pod \"monitoring-plugin-665c9c8584-dpgd2\" (UID: \"891d3439-8489-46f4-9612-dbce761a67c6\") " pod="openshift-monitoring/monitoring-plugin-665c9c8584-dpgd2"
May 25 12:15:33.022623 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:33.022523    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/891d3439-8489-46f4-9612-dbce761a67c6-monitoring-plugin-cert\") pod \"monitoring-plugin-665c9c8584-dpgd2\" (UID: \"891d3439-8489-46f4-9612-dbce761a67c6\") " pod="openshift-monitoring/monitoring-plugin-665c9c8584-dpgd2"
May 25 12:15:33.024914 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:33.024889    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/891d3439-8489-46f4-9612-dbce761a67c6-monitoring-plugin-cert\") pod \"monitoring-plugin-665c9c8584-dpgd2\" (UID: \"891d3439-8489-46f4-9612-dbce761a67c6\") " pod="openshift-monitoring/monitoring-plugin-665c9c8584-dpgd2"
May 25 12:15:33.143246 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:33.143212    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-665c9c8584-dpgd2"
May 25 12:15:33.253614 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:33.253582    2549 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-99b5fcf8-8wb4v"]
May 25 12:15:33.258171 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:33.258147    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-99b5fcf8-8wb4v"
May 25 12:15:33.260784 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:33.260765    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
May 25 12:15:33.260908 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:33.260881    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
May 25 12:15:33.261055 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:33.261036    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
May 25 12:15:33.261172 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:33.261043    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
May 25 12:15:33.261172 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:33.261114    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-dg9gx\""
May 25 12:15:33.261343 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:33.261325    2549 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
May 25 12:15:33.266596 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:33.266463    2549 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
May 25 12:15:33.269504 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:33.269482    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-99b5fcf8-8wb4v"]
May 25 12:15:33.276422 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:33.276403    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-665c9c8584-dpgd2"]
May 25 12:15:33.278867 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:15:33.278844    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod891d3439_8489_46f4_9612_dbce761a67c6.slice/crio-68d13df8927e1f94d239718d004867203a92d134290dd1ecfca97f22a1a223ba WatchSource:0}: Error finding container 68d13df8927e1f94d239718d004867203a92d134290dd1ecfca97f22a1a223ba: Status 404 returned error can't find the container with id 68d13df8927e1f94d239718d004867203a92d134290dd1ecfca97f22a1a223ba
May 25 12:15:33.398055 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:33.398024    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5b67cb8f7-zkltg" event={"ID":"83c8cfdf-2cea-45d9-832b-b4172368763f","Type":"ContainerStarted","Data":"ec32003ef09c3217758a5a61416e7aa3072936ce29708ea77eaf500447cd5d72"}
May 25 12:15:33.398237 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:33.398059    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5b67cb8f7-zkltg" event={"ID":"83c8cfdf-2cea-45d9-832b-b4172368763f","Type":"ContainerStarted","Data":"5956c736ae8587bd43873f41344df707968bf96b122003c49a08a65352711d3f"}
May 25 12:15:33.398237 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:33.398076    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5b67cb8f7-zkltg" event={"ID":"83c8cfdf-2cea-45d9-832b-b4172368763f","Type":"ContainerStarted","Data":"061cf4dbbde3faddbb3b6c2c4793699ee298a526cef342bb93463ea4b6947be1"}
May 25 12:15:33.398970 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:33.398950    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-665c9c8584-dpgd2" event={"ID":"891d3439-8489-46f4-9612-dbce761a67c6","Type":"ContainerStarted","Data":"68d13df8927e1f94d239718d004867203a92d134290dd1ecfca97f22a1a223ba"}
May 25 12:15:33.425945 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:33.425894    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/048a5c2a-b40b-48c8-b889-58593849775f-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-99b5fcf8-8wb4v\" (UID: \"048a5c2a-b40b-48c8-b889-58593849775f\") " pod="openshift-monitoring/telemeter-client-99b5fcf8-8wb4v"
May 25 12:15:33.426088 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:33.426011    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/048a5c2a-b40b-48c8-b889-58593849775f-telemeter-trusted-ca-bundle\") pod \"telemeter-client-99b5fcf8-8wb4v\" (UID: \"048a5c2a-b40b-48c8-b889-58593849775f\") " pod="openshift-monitoring/telemeter-client-99b5fcf8-8wb4v"
May 25 12:15:33.426088 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:33.426048    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/048a5c2a-b40b-48c8-b889-58593849775f-metrics-client-ca\") pod \"telemeter-client-99b5fcf8-8wb4v\" (UID: \"048a5c2a-b40b-48c8-b889-58593849775f\") " pod="openshift-monitoring/telemeter-client-99b5fcf8-8wb4v"
May 25 12:15:33.426237 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:33.426090    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/048a5c2a-b40b-48c8-b889-58593849775f-serving-certs-ca-bundle\") pod \"telemeter-client-99b5fcf8-8wb4v\" (UID: \"048a5c2a-b40b-48c8-b889-58593849775f\") " pod="openshift-monitoring/telemeter-client-99b5fcf8-8wb4v"
May 25 12:15:33.426237 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:33.426153    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/048a5c2a-b40b-48c8-b889-58593849775f-secret-telemeter-client\") pod \"telemeter-client-99b5fcf8-8wb4v\" (UID: \"048a5c2a-b40b-48c8-b889-58593849775f\") " pod="openshift-monitoring/telemeter-client-99b5fcf8-8wb4v"
May 25 12:15:33.426347 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:33.426223    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/048a5c2a-b40b-48c8-b889-58593849775f-telemeter-client-tls\") pod \"telemeter-client-99b5fcf8-8wb4v\" (UID: \"048a5c2a-b40b-48c8-b889-58593849775f\") " pod="openshift-monitoring/telemeter-client-99b5fcf8-8wb4v"
May 25 12:15:33.426347 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:33.426266    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/048a5c2a-b40b-48c8-b889-58593849775f-federate-client-tls\") pod \"telemeter-client-99b5fcf8-8wb4v\" (UID: \"048a5c2a-b40b-48c8-b889-58593849775f\") " pod="openshift-monitoring/telemeter-client-99b5fcf8-8wb4v"
May 25 12:15:33.426347 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:33.426299    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhl4t\" (UniqueName: \"kubernetes.io/projected/048a5c2a-b40b-48c8-b889-58593849775f-kube-api-access-xhl4t\") pod \"telemeter-client-99b5fcf8-8wb4v\" (UID: \"048a5c2a-b40b-48c8-b889-58593849775f\") " pod="openshift-monitoring/telemeter-client-99b5fcf8-8wb4v"
May 25 12:15:33.527619 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:33.527541    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-xhl4t\" (UniqueName: \"kubernetes.io/projected/048a5c2a-b40b-48c8-b889-58593849775f-kube-api-access-xhl4t\") pod \"telemeter-client-99b5fcf8-8wb4v\" (UID: \"048a5c2a-b40b-48c8-b889-58593849775f\") " pod="openshift-monitoring/telemeter-client-99b5fcf8-8wb4v"
May 25 12:15:33.527997 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:33.527630    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/048a5c2a-b40b-48c8-b889-58593849775f-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-99b5fcf8-8wb4v\" (UID: \"048a5c2a-b40b-48c8-b889-58593849775f\") " pod="openshift-monitoring/telemeter-client-99b5fcf8-8wb4v"
May 25 12:15:33.527997 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:33.527694    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/048a5c2a-b40b-48c8-b889-58593849775f-telemeter-trusted-ca-bundle\") pod \"telemeter-client-99b5fcf8-8wb4v\" (UID: \"048a5c2a-b40b-48c8-b889-58593849775f\") " pod="openshift-monitoring/telemeter-client-99b5fcf8-8wb4v"
May 25 12:15:33.527997 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:33.527725    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/048a5c2a-b40b-48c8-b889-58593849775f-metrics-client-ca\") pod \"telemeter-client-99b5fcf8-8wb4v\" (UID: \"048a5c2a-b40b-48c8-b889-58593849775f\") " pod="openshift-monitoring/telemeter-client-99b5fcf8-8wb4v"
May 25 12:15:33.527997 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:33.527763    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/048a5c2a-b40b-48c8-b889-58593849775f-serving-certs-ca-bundle\") pod \"telemeter-client-99b5fcf8-8wb4v\" (UID: \"048a5c2a-b40b-48c8-b889-58593849775f\") " pod="openshift-monitoring/telemeter-client-99b5fcf8-8wb4v"
May 25 12:15:33.527997 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:33.527784    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/048a5c2a-b40b-48c8-b889-58593849775f-secret-telemeter-client\") pod \"telemeter-client-99b5fcf8-8wb4v\" (UID: \"048a5c2a-b40b-48c8-b889-58593849775f\") " pod="openshift-monitoring/telemeter-client-99b5fcf8-8wb4v"
May 25 12:15:33.527997 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:33.527818    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/048a5c2a-b40b-48c8-b889-58593849775f-telemeter-client-tls\") pod \"telemeter-client-99b5fcf8-8wb4v\" (UID: \"048a5c2a-b40b-48c8-b889-58593849775f\") " pod="openshift-monitoring/telemeter-client-99b5fcf8-8wb4v"
May 25 12:15:33.527997 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:33.527863    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/048a5c2a-b40b-48c8-b889-58593849775f-federate-client-tls\") pod \"telemeter-client-99b5fcf8-8wb4v\" (UID: \"048a5c2a-b40b-48c8-b889-58593849775f\") " pod="openshift-monitoring/telemeter-client-99b5fcf8-8wb4v"
May 25 12:15:33.529230 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:33.529097    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/048a5c2a-b40b-48c8-b889-58593849775f-serving-certs-ca-bundle\") pod \"telemeter-client-99b5fcf8-8wb4v\" (UID: \"048a5c2a-b40b-48c8-b889-58593849775f\") " pod="openshift-monitoring/telemeter-client-99b5fcf8-8wb4v"
May 25 12:15:33.534532 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:33.530103    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/048a5c2a-b40b-48c8-b889-58593849775f-metrics-client-ca\") pod \"telemeter-client-99b5fcf8-8wb4v\" (UID: \"048a5c2a-b40b-48c8-b889-58593849775f\") " pod="openshift-monitoring/telemeter-client-99b5fcf8-8wb4v"
May 25 12:15:33.534532 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:33.531627    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/048a5c2a-b40b-48c8-b889-58593849775f-telemeter-client-tls\") pod \"telemeter-client-99b5fcf8-8wb4v\" (UID: \"048a5c2a-b40b-48c8-b889-58593849775f\") " pod="openshift-monitoring/telemeter-client-99b5fcf8-8wb4v"
May 25 12:15:33.534532 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:33.531746    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/048a5c2a-b40b-48c8-b889-58593849775f-telemeter-trusted-ca-bundle\") pod \"telemeter-client-99b5fcf8-8wb4v\" (UID: \"048a5c2a-b40b-48c8-b889-58593849775f\") " pod="openshift-monitoring/telemeter-client-99b5fcf8-8wb4v"
May 25 12:15:33.534532 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:33.532420    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/048a5c2a-b40b-48c8-b889-58593849775f-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-99b5fcf8-8wb4v\" (UID: \"048a5c2a-b40b-48c8-b889-58593849775f\") " pod="openshift-monitoring/telemeter-client-99b5fcf8-8wb4v"
May 25 12:15:33.534532 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:33.532820    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/048a5c2a-b40b-48c8-b889-58593849775f-secret-telemeter-client\") pod \"telemeter-client-99b5fcf8-8wb4v\" (UID: \"048a5c2a-b40b-48c8-b889-58593849775f\") " pod="openshift-monitoring/telemeter-client-99b5fcf8-8wb4v"
May 25 12:15:33.535267 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:33.535243    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/048a5c2a-b40b-48c8-b889-58593849775f-federate-client-tls\") pod \"telemeter-client-99b5fcf8-8wb4v\" (UID: \"048a5c2a-b40b-48c8-b889-58593849775f\") " pod="openshift-monitoring/telemeter-client-99b5fcf8-8wb4v"
May 25 12:15:33.537441 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:33.537418    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhl4t\" (UniqueName: \"kubernetes.io/projected/048a5c2a-b40b-48c8-b889-58593849775f-kube-api-access-xhl4t\") pod \"telemeter-client-99b5fcf8-8wb4v\" (UID: \"048a5c2a-b40b-48c8-b889-58593849775f\") " pod="openshift-monitoring/telemeter-client-99b5fcf8-8wb4v"
May 25 12:15:33.570255 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:33.570229    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-99b5fcf8-8wb4v"
May 25 12:15:33.708962 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:33.708930    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-99b5fcf8-8wb4v"]
May 25 12:15:33.787616 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:15:33.787539    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod048a5c2a_b40b_48c8_b889_58593849775f.slice/crio-bdf9c68185d1cc6564c17b5cbebd99661014ddfae36bfdc56d0b3018a5fc9742 WatchSource:0}: Error finding container bdf9c68185d1cc6564c17b5cbebd99661014ddfae36bfdc56d0b3018a5fc9742: Status 404 returned error can't find the container with id bdf9c68185d1cc6564c17b5cbebd99661014ddfae36bfdc56d0b3018a5fc9742
May 25 12:15:34.406435 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:34.406398    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5b67cb8f7-zkltg" event={"ID":"83c8cfdf-2cea-45d9-832b-b4172368763f","Type":"ContainerStarted","Data":"971194c239f7f0597e7bafa9dfb9770e0fab6d82f596f9822f739ba162e5def1"}
May 25 12:15:34.406435 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:34.406442    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5b67cb8f7-zkltg" event={"ID":"83c8cfdf-2cea-45d9-832b-b4172368763f","Type":"ContainerStarted","Data":"c17b9d3bea7430357158dc4d0ca1da0a1966d7e948b37cee962eda3ca01a6f7e"}
May 25 12:15:34.406697 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:34.406457    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5b67cb8f7-zkltg" event={"ID":"83c8cfdf-2cea-45d9-832b-b4172368763f","Type":"ContainerStarted","Data":"15769334fe9a3c653e24665b1a1e62041b8e4df1c34069a50c5ba1868055435b"}
May 25 12:15:34.406697 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:34.406557    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-5b67cb8f7-zkltg"
May 25 12:15:34.408011 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:34.407987    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-99b5fcf8-8wb4v" event={"ID":"048a5c2a-b40b-48c8-b889-58593849775f","Type":"ContainerStarted","Data":"bdf9c68185d1cc6564c17b5cbebd99661014ddfae36bfdc56d0b3018a5fc9742"}
May 25 12:15:34.431610 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:34.431550    2549 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-5b67cb8f7-zkltg" podStartSLOduration=2.41737516 podStartE2EDuration="5.431532774s" podCreationTimestamp="2026-05-25 12:15:29 +0000 UTC" firstStartedPulling="2026-05-25 12:15:30.84362673 +0000 UTC m=+134.559442006" lastFinishedPulling="2026-05-25 12:15:33.857784339 +0000 UTC m=+137.573599620" observedRunningTime="2026-05-25 12:15:34.429838839 +0000 UTC m=+138.145654150" watchObservedRunningTime="2026-05-25 12:15:34.431532774 +0000 UTC m=+138.147348065"
May 25 12:15:35.412531 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:35.412487    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-665c9c8584-dpgd2" event={"ID":"891d3439-8489-46f4-9612-dbce761a67c6","Type":"ContainerStarted","Data":"ed6cdb992067a6a9e72d21c17793da5b8ea51aa0e753cfef1c6d8889c4d46df3"}
May 25 12:15:35.413009 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:35.412825    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-665c9c8584-dpgd2"
May 25 12:15:35.417536 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:35.417511    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-665c9c8584-dpgd2"
May 25 12:15:35.446484 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:35.446430    2549 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-665c9c8584-dpgd2" podStartSLOduration=1.760503814 podStartE2EDuration="3.446414624s" podCreationTimestamp="2026-05-25 12:15:32 +0000 UTC" firstStartedPulling="2026-05-25 12:15:33.280711534 +0000 UTC m=+136.996526799" lastFinishedPulling="2026-05-25 12:15:34.966622344 +0000 UTC m=+138.682437609" observedRunningTime="2026-05-25 12:15:35.445948157 +0000 UTC m=+139.161763445" watchObservedRunningTime="2026-05-25 12:15:35.446414624 +0000 UTC m=+139.162229910"
May 25 12:15:36.417495 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:36.417461    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-99b5fcf8-8wb4v" event={"ID":"048a5c2a-b40b-48c8-b889-58593849775f","Type":"ContainerStarted","Data":"a2d08dcf953e92da6024f795589ad50aeb0f044336a403968c8da3956662d11e"}
May 25 12:15:37.422018 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:37.421980    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-99b5fcf8-8wb4v" event={"ID":"048a5c2a-b40b-48c8-b889-58593849775f","Type":"ContainerStarted","Data":"29d69a91fa5057d9c1c54ca5938ecb2038ec9c94a74bdf2e98344d417d5923af"}
May 25 12:15:37.422018 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:37.422020    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-99b5fcf8-8wb4v" event={"ID":"048a5c2a-b40b-48c8-b889-58593849775f","Type":"ContainerStarted","Data":"bc12c70b35aa5dcb862aae3d1c231ca39e5d1a48d99ba070cf81b995f9bfa624"}
May 25 12:15:37.447192 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:37.447135    2549 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-99b5fcf8-8wb4v" podStartSLOduration=1.562637 podStartE2EDuration="4.44712292s" podCreationTimestamp="2026-05-25 12:15:33 +0000 UTC" firstStartedPulling="2026-05-25 12:15:33.789383501 +0000 UTC m=+137.505198767" lastFinishedPulling="2026-05-25 12:15:36.673869421 +0000 UTC m=+140.389684687" observedRunningTime="2026-05-25 12:15:37.446713945 +0000 UTC m=+141.162529232" watchObservedRunningTime="2026-05-25 12:15:37.44712292 +0000 UTC m=+141.162938207"
May 25 12:15:37.764998 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:37.764909    2549 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7654684cf-5vvhl"]
May 25 12:15:37.802939 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:37.802909    2549 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console/console-54c67d8478-vzxxl"]
May 25 12:15:37.806443 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:37.806428    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-54c67d8478-vzxxl"
May 25 12:15:37.814450 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:37.814160    2549 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
May 25 12:15:37.815651 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:37.815627    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-54c67d8478-vzxxl"]
May 25 12:15:37.975656 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:37.975606    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p25dc\" (UniqueName: \"kubernetes.io/projected/f1fc499f-2c5a-4c4d-849e-f8e3453a2026-kube-api-access-p25dc\") pod \"console-54c67d8478-vzxxl\" (UID: \"f1fc499f-2c5a-4c4d-849e-f8e3453a2026\") " pod="openshift-console/console-54c67d8478-vzxxl"
May 25 12:15:37.975656 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:37.975663    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f1fc499f-2c5a-4c4d-849e-f8e3453a2026-console-oauth-config\") pod \"console-54c67d8478-vzxxl\" (UID: \"f1fc499f-2c5a-4c4d-849e-f8e3453a2026\") " pod="openshift-console/console-54c67d8478-vzxxl"
May 25 12:15:37.975861 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:37.975716    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f1fc499f-2c5a-4c4d-849e-f8e3453a2026-trusted-ca-bundle\") pod \"console-54c67d8478-vzxxl\" (UID: \"f1fc499f-2c5a-4c4d-849e-f8e3453a2026\") " pod="openshift-console/console-54c67d8478-vzxxl"
May 25 12:15:37.975861 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:37.975826    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f1fc499f-2c5a-4c4d-849e-f8e3453a2026-console-config\") pod \"console-54c67d8478-vzxxl\" (UID: \"f1fc499f-2c5a-4c4d-849e-f8e3453a2026\") " pod="openshift-console/console-54c67d8478-vzxxl"
May 25 12:15:37.975861 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:37.975854    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f1fc499f-2c5a-4c4d-849e-f8e3453a2026-service-ca\") pod \"console-54c67d8478-vzxxl\" (UID: \"f1fc499f-2c5a-4c4d-849e-f8e3453a2026\") " pod="openshift-console/console-54c67d8478-vzxxl"
May 25 12:15:37.975958 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:37.975877    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f1fc499f-2c5a-4c4d-849e-f8e3453a2026-console-serving-cert\") pod \"console-54c67d8478-vzxxl\" (UID: \"f1fc499f-2c5a-4c4d-849e-f8e3453a2026\") " pod="openshift-console/console-54c67d8478-vzxxl"
May 25 12:15:37.975958 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:37.975893    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f1fc499f-2c5a-4c4d-849e-f8e3453a2026-oauth-serving-cert\") pod \"console-54c67d8478-vzxxl\" (UID: \"f1fc499f-2c5a-4c4d-849e-f8e3453a2026\") " pod="openshift-console/console-54c67d8478-vzxxl"
May 25 12:15:38.076715 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:38.076614    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f1fc499f-2c5a-4c4d-849e-f8e3453a2026-service-ca\") pod \"console-54c67d8478-vzxxl\" (UID: \"f1fc499f-2c5a-4c4d-849e-f8e3453a2026\") " pod="openshift-console/console-54c67d8478-vzxxl"
May 25 12:15:38.076715 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:38.076671    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f1fc499f-2c5a-4c4d-849e-f8e3453a2026-console-serving-cert\") pod \"console-54c67d8478-vzxxl\" (UID: \"f1fc499f-2c5a-4c4d-849e-f8e3453a2026\") " pod="openshift-console/console-54c67d8478-vzxxl"
May 25 12:15:38.076715 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:38.076693    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f1fc499f-2c5a-4c4d-849e-f8e3453a2026-oauth-serving-cert\") pod \"console-54c67d8478-vzxxl\" (UID: \"f1fc499f-2c5a-4c4d-849e-f8e3453a2026\") " pod="openshift-console/console-54c67d8478-vzxxl"
May 25 12:15:38.076979 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:38.076731    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-p25dc\" (UniqueName: \"kubernetes.io/projected/f1fc499f-2c5a-4c4d-849e-f8e3453a2026-kube-api-access-p25dc\") pod \"console-54c67d8478-vzxxl\" (UID: \"f1fc499f-2c5a-4c4d-849e-f8e3453a2026\") " pod="openshift-console/console-54c67d8478-vzxxl"
May 25 12:15:38.076979 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:38.076759    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f1fc499f-2c5a-4c4d-849e-f8e3453a2026-console-oauth-config\") pod \"console-54c67d8478-vzxxl\" (UID: \"f1fc499f-2c5a-4c4d-849e-f8e3453a2026\") " pod="openshift-console/console-54c67d8478-vzxxl"
May 25 12:15:38.076979 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:38.076780    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f1fc499f-2c5a-4c4d-849e-f8e3453a2026-trusted-ca-bundle\") pod \"console-54c67d8478-vzxxl\" (UID: \"f1fc499f-2c5a-4c4d-849e-f8e3453a2026\") " pod="openshift-console/console-54c67d8478-vzxxl"
May 25 12:15:38.076979 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:38.076879    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f1fc499f-2c5a-4c4d-849e-f8e3453a2026-console-config\") pod \"console-54c67d8478-vzxxl\" (UID: \"f1fc499f-2c5a-4c4d-849e-f8e3453a2026\") " pod="openshift-console/console-54c67d8478-vzxxl"
May 25 12:15:38.077455 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:38.077430    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f1fc499f-2c5a-4c4d-849e-f8e3453a2026-oauth-serving-cert\") pod \"console-54c67d8478-vzxxl\" (UID: \"f1fc499f-2c5a-4c4d-849e-f8e3453a2026\") " pod="openshift-console/console-54c67d8478-vzxxl"
May 25 12:15:38.077562 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:38.077449    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f1fc499f-2c5a-4c4d-849e-f8e3453a2026-service-ca\") pod \"console-54c67d8478-vzxxl\" (UID: \"f1fc499f-2c5a-4c4d-849e-f8e3453a2026\") " pod="openshift-console/console-54c67d8478-vzxxl"
May 25 12:15:38.077619 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:38.077578    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f1fc499f-2c5a-4c4d-849e-f8e3453a2026-console-config\") pod \"console-54c67d8478-vzxxl\" (UID: \"f1fc499f-2c5a-4c4d-849e-f8e3453a2026\") " pod="openshift-console/console-54c67d8478-vzxxl"
May 25 12:15:38.077676 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:38.077657    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f1fc499f-2c5a-4c4d-849e-f8e3453a2026-trusted-ca-bundle\") pod \"console-54c67d8478-vzxxl\" (UID: \"f1fc499f-2c5a-4c4d-849e-f8e3453a2026\") " pod="openshift-console/console-54c67d8478-vzxxl"
May 25 12:15:38.079739 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:38.079717    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f1fc499f-2c5a-4c4d-849e-f8e3453a2026-console-oauth-config\") pod \"console-54c67d8478-vzxxl\" (UID: \"f1fc499f-2c5a-4c4d-849e-f8e3453a2026\") " pod="openshift-console/console-54c67d8478-vzxxl"
May 25 12:15:38.079819 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:38.079805    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f1fc499f-2c5a-4c4d-849e-f8e3453a2026-console-serving-cert\") pod \"console-54c67d8478-vzxxl\" (UID: \"f1fc499f-2c5a-4c4d-849e-f8e3453a2026\") " pod="openshift-console/console-54c67d8478-vzxxl"
May 25 12:15:38.085986 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:38.085960    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p25dc\" (UniqueName: \"kubernetes.io/projected/f1fc499f-2c5a-4c4d-849e-f8e3453a2026-kube-api-access-p25dc\") pod \"console-54c67d8478-vzxxl\" (UID: \"f1fc499f-2c5a-4c4d-849e-f8e3453a2026\") " pod="openshift-console/console-54c67d8478-vzxxl"
May 25 12:15:38.115848 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:38.115811    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-54c67d8478-vzxxl"
May 25 12:15:38.238155 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:38.238098    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-54c67d8478-vzxxl"]
May 25 12:15:38.240745 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:15:38.240715    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1fc499f_2c5a_4c4d_849e_f8e3453a2026.slice/crio-3474d600c859fc326d7f5c62f5298545c75ac19c9f762cdbe2cbe9db617a14a0 WatchSource:0}: Error finding container 3474d600c859fc326d7f5c62f5298545c75ac19c9f762cdbe2cbe9db617a14a0: Status 404 returned error can't find the container with id 3474d600c859fc326d7f5c62f5298545c75ac19c9f762cdbe2cbe9db617a14a0
May 25 12:15:38.426926 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:38.426886    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-54c67d8478-vzxxl" event={"ID":"f1fc499f-2c5a-4c4d-849e-f8e3453a2026","Type":"ContainerStarted","Data":"21b61279d43a74ffcf3b02fcc78580e8a5b65dabe2ef3b4cfa314d6e10987fe9"}
May 25 12:15:38.427299 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:38.426931    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-54c67d8478-vzxxl" event={"ID":"f1fc499f-2c5a-4c4d-849e-f8e3453a2026","Type":"ContainerStarted","Data":"3474d600c859fc326d7f5c62f5298545c75ac19c9f762cdbe2cbe9db617a14a0"}
May 25 12:15:38.447329 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:38.447271    2549 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console/console-54c67d8478-vzxxl" podStartSLOduration=1.44725204 podStartE2EDuration="1.44725204s" podCreationTimestamp="2026-05-25 12:15:37 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 12:15:38.446352378 +0000 UTC m=+142.162167665" watchObservedRunningTime="2026-05-25 12:15:38.44725204 +0000 UTC m=+142.163067327"
May 25 12:15:40.418618 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:40.418589    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-5b67cb8f7-zkltg"
May 25 12:15:48.116275 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:48.116239    2549 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-54c67d8478-vzxxl"
May 25 12:15:48.116275 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:48.116276    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-54c67d8478-vzxxl"
May 25 12:15:48.117546 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:48.117522    2549 patch_prober.go:28] interesting pod/console-54c67d8478-vzxxl container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.134.0.23:8443/health\": dial tcp 10.134.0.23:8443: connect: connection refused" start-of-body=
May 25 12:15:48.117666 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:48.117568    2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-console/console-54c67d8478-vzxxl" podUID="f1fc499f-2c5a-4c4d-849e-f8e3453a2026" containerName="console" probeResult="failure" output="Get \"https://10.134.0.23:8443/health\": dial tcp 10.134.0.23:8443: connect: connection refused"
May 25 12:15:54.176643 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:15:54.176595    2549 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-ml56g" podUID="acccf0fc-6d0f-4793-acb0-69227950e520"
May 25 12:15:54.196895 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:15:54.196867    2549 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-ingress-canary/ingress-canary-fxlzt" podUID="0ea389da-cc7e-4b86-846d-e95f5a2dae7c"
May 25 12:15:54.470388 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:54.470305    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-fxlzt"
May 25 12:15:54.470534 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:54.470310    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-ml56g"
May 25 12:15:58.116601 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:58.116564    2549 patch_prober.go:28] interesting pod/console-54c67d8478-vzxxl container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.134.0.23:8443/health\": dial tcp 10.134.0.23:8443: connect: connection refused" start-of-body=
May 25 12:15:58.116975 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:58.116621    2549 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-console/console-54c67d8478-vzxxl" podUID="f1fc499f-2c5a-4c4d-849e-f8e3453a2026" containerName="console" probeResult="failure" output="Get \"https://10.134.0.23:8443/health\": dial tcp 10.134.0.23:8443: connect: connection refused"
May 25 12:15:59.059810 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:59.059766    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/acccf0fc-6d0f-4793-acb0-69227950e520-metrics-tls\") pod \"dns-default-ml56g\" (UID: \"acccf0fc-6d0f-4793-acb0-69227950e520\") " pod="openshift-dns/dns-default-ml56g"
May 25 12:15:59.060290 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:59.060270    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0ea389da-cc7e-4b86-846d-e95f5a2dae7c-cert\") pod \"ingress-canary-fxlzt\" (UID: \"0ea389da-cc7e-4b86-846d-e95f5a2dae7c\") " pod="openshift-ingress-canary/ingress-canary-fxlzt"
May 25 12:15:59.063787 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:59.063523    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0ea389da-cc7e-4b86-846d-e95f5a2dae7c-cert\") pod \"ingress-canary-fxlzt\" (UID: \"0ea389da-cc7e-4b86-846d-e95f5a2dae7c\") " pod="openshift-ingress-canary/ingress-canary-fxlzt"
May 25 12:15:59.065896 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:59.065867    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/acccf0fc-6d0f-4793-acb0-69227950e520-metrics-tls\") pod \"dns-default-ml56g\" (UID: \"acccf0fc-6d0f-4793-acb0-69227950e520\") " pod="openshift-dns/dns-default-ml56g"
May 25 12:15:59.274138 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:59.274106    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-sw9bq\""
May 25 12:15:59.275103 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:59.275087    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-c8dc6\""
May 25 12:15:59.282152 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:59.282125    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-fxlzt"
May 25 12:15:59.282282 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:59.282152    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-ml56g"
May 25 12:15:59.416650 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:59.416624    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-fxlzt"]
May 25 12:15:59.421959 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:15:59.421911    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ea389da_cc7e_4b86_846d_e95f5a2dae7c.slice/crio-571d34fbca84fd24af5a14e14ed77da5c3d31495053445d27e354fe0f519ce07 WatchSource:0}: Error finding container 571d34fbca84fd24af5a14e14ed77da5c3d31495053445d27e354fe0f519ce07: Status 404 returned error can't find the container with id 571d34fbca84fd24af5a14e14ed77da5c3d31495053445d27e354fe0f519ce07
May 25 12:15:59.431126 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:59.431102    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-ml56g"]
May 25 12:15:59.435090 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:15:59.435067    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacccf0fc_6d0f_4793_acb0_69227950e520.slice/crio-73a30fd1ca0ff677717f825c54c8ffe455d4091a0aa6912f8e6a6ee207751d67 WatchSource:0}: Error finding container 73a30fd1ca0ff677717f825c54c8ffe455d4091a0aa6912f8e6a6ee207751d67: Status 404 returned error can't find the container with id 73a30fd1ca0ff677717f825c54c8ffe455d4091a0aa6912f8e6a6ee207751d67
May 25 12:15:59.485257 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:59.485224    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-ml56g" event={"ID":"acccf0fc-6d0f-4793-acb0-69227950e520","Type":"ContainerStarted","Data":"73a30fd1ca0ff677717f825c54c8ffe455d4091a0aa6912f8e6a6ee207751d67"}
May 25 12:15:59.486134 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:15:59.486109    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-fxlzt" event={"ID":"0ea389da-cc7e-4b86-846d-e95f5a2dae7c","Type":"ContainerStarted","Data":"571d34fbca84fd24af5a14e14ed77da5c3d31495053445d27e354fe0f519ce07"}
May 25 12:16:00.491003 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:16:00.490910    2549 generic.go:358] "Generic (PLEG): container finished" podID="bf795e4e-8902-4aa8-be41-2671139da816" containerID="32e4d277473ee027b7d875c1cb90246b2ab52d06be2cdc907133393062c94259" exitCode=0
May 25 12:16:00.491634 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:16:00.490990    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-78bdfb888c-nxz4w" event={"ID":"bf795e4e-8902-4aa8-be41-2671139da816","Type":"ContainerDied","Data":"32e4d277473ee027b7d875c1cb90246b2ab52d06be2cdc907133393062c94259"}
May 25 12:16:00.491634 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:16:00.491435    2549 scope.go:117] "RemoveContainer" containerID="32e4d277473ee027b7d875c1cb90246b2ab52d06be2cdc907133393062c94259"
May 25 12:16:01.497645 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:16:01.497599    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-78bdfb888c-nxz4w" event={"ID":"bf795e4e-8902-4aa8-be41-2671139da816","Type":"ContainerStarted","Data":"3149bba81101bebf0c6159b577ddfc648b33b0f20fbb365750294861652f7f00"}
May 25 12:16:01.499542 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:16:01.499515    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-ml56g" event={"ID":"acccf0fc-6d0f-4793-acb0-69227950e520","Type":"ContainerStarted","Data":"301c6ca66b134ef619a00dd44f081daaf5d6a950414873332776d0bb04de386b"}
May 25 12:16:01.499542 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:16:01.499546    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-ml56g" event={"ID":"acccf0fc-6d0f-4793-acb0-69227950e520","Type":"ContainerStarted","Data":"54719559b4f54c5cd085dc610af3ddbede6b261e26d439dd169c5fe9adec3bcf"}
May 25 12:16:01.499714 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:16:01.499649    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-ml56g"
May 25 12:16:01.503143 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:16:01.503122    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-fxlzt" event={"ID":"0ea389da-cc7e-4b86-846d-e95f5a2dae7c","Type":"ContainerStarted","Data":"34553acc60383599ff7eb15c3dd04243bddae66213ac45997bf1971a8839e23d"}
May 25 12:16:01.536435 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:16:01.536366    2549 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-dns/dns-default-ml56g" podStartSLOduration=128.713022489 podStartE2EDuration="2m10.536346582s" podCreationTimestamp="2026-05-25 12:13:51 +0000 UTC" firstStartedPulling="2026-05-25 12:15:59.43691493 +0000 UTC m=+163.152730195" lastFinishedPulling="2026-05-25 12:16:01.260239023 +0000 UTC m=+164.976054288" observedRunningTime="2026-05-25 12:16:01.535062763 +0000 UTC m=+165.250878065" watchObservedRunningTime="2026-05-25 12:16:01.536346582 +0000 UTC m=+165.252161869"
May 25 12:16:01.557623 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:16:01.557558    2549 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-fxlzt" podStartSLOduration=128.719961829 podStartE2EDuration="2m10.55754191s" podCreationTimestamp="2026-05-25 12:13:51 +0000 UTC" firstStartedPulling="2026-05-25 12:15:59.423321744 +0000 UTC m=+163.139137009" lastFinishedPulling="2026-05-25 12:16:01.260901814 +0000 UTC m=+164.976717090" observedRunningTime="2026-05-25 12:16:01.556666874 +0000 UTC m=+165.272482184" watchObservedRunningTime="2026-05-25 12:16:01.55754191 +0000 UTC m=+165.273357197"
May 25 12:16:02.784183 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:16:02.784135    2549 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-console/console-7654684cf-5vvhl" podUID="620aafc8-f648-4809-8abe-094c0f66f3de" containerName="console" containerID="cri-o://056f1b05ec15a68360a22883a381de0431191d5cf5e58b08d76d06ef0a2b56ed" gracePeriod=15
May 25 12:16:03.034110 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:16:03.034086    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-7654684cf-5vvhl_620aafc8-f648-4809-8abe-094c0f66f3de/console/0.log"
May 25 12:16:03.034301 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:16:03.034146    2549 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7654684cf-5vvhl"
May 25 12:16:03.096837 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:16:03.096799    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/620aafc8-f648-4809-8abe-094c0f66f3de-console-config\") pod \"620aafc8-f648-4809-8abe-094c0f66f3de\" (UID: \"620aafc8-f648-4809-8abe-094c0f66f3de\") "
May 25 12:16:03.097011 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:16:03.096849    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ctrrg\" (UniqueName: \"kubernetes.io/projected/620aafc8-f648-4809-8abe-094c0f66f3de-kube-api-access-ctrrg\") pod \"620aafc8-f648-4809-8abe-094c0f66f3de\" (UID: \"620aafc8-f648-4809-8abe-094c0f66f3de\") "
May 25 12:16:03.097011 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:16:03.096886    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/620aafc8-f648-4809-8abe-094c0f66f3de-console-serving-cert\") pod \"620aafc8-f648-4809-8abe-094c0f66f3de\" (UID: \"620aafc8-f648-4809-8abe-094c0f66f3de\") "
May 25 12:16:03.097011 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:16:03.096923    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/620aafc8-f648-4809-8abe-094c0f66f3de-oauth-serving-cert\") pod \"620aafc8-f648-4809-8abe-094c0f66f3de\" (UID: \"620aafc8-f648-4809-8abe-094c0f66f3de\") "
May 25 12:16:03.097011 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:16:03.096948    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/620aafc8-f648-4809-8abe-094c0f66f3de-console-oauth-config\") pod \"620aafc8-f648-4809-8abe-094c0f66f3de\" (UID: \"620aafc8-f648-4809-8abe-094c0f66f3de\") "
May 25 12:16:03.097011 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:16:03.096980    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/620aafc8-f648-4809-8abe-094c0f66f3de-service-ca\") pod \"620aafc8-f648-4809-8abe-094c0f66f3de\" (UID: \"620aafc8-f648-4809-8abe-094c0f66f3de\") "
May 25 12:16:03.097382 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:16:03.097354    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/620aafc8-f648-4809-8abe-094c0f66f3de-console-config" (OuterVolumeSpecName: "console-config") pod "620aafc8-f648-4809-8abe-094c0f66f3de" (UID: "620aafc8-f648-4809-8abe-094c0f66f3de"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 12:16:03.097492 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:16:03.097386    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/620aafc8-f648-4809-8abe-094c0f66f3de-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "620aafc8-f648-4809-8abe-094c0f66f3de" (UID: "620aafc8-f648-4809-8abe-094c0f66f3de"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 12:16:03.097492 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:16:03.097409    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/620aafc8-f648-4809-8abe-094c0f66f3de-service-ca" (OuterVolumeSpecName: "service-ca") pod "620aafc8-f648-4809-8abe-094c0f66f3de" (UID: "620aafc8-f648-4809-8abe-094c0f66f3de"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 12:16:03.099314 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:16:03.099288    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/620aafc8-f648-4809-8abe-094c0f66f3de-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "620aafc8-f648-4809-8abe-094c0f66f3de" (UID: "620aafc8-f648-4809-8abe-094c0f66f3de"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 12:16:03.099314 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:16:03.099287    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/620aafc8-f648-4809-8abe-094c0f66f3de-kube-api-access-ctrrg" (OuterVolumeSpecName: "kube-api-access-ctrrg") pod "620aafc8-f648-4809-8abe-094c0f66f3de" (UID: "620aafc8-f648-4809-8abe-094c0f66f3de"). InnerVolumeSpecName "kube-api-access-ctrrg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 12:16:03.099466 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:16:03.099337    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/620aafc8-f648-4809-8abe-094c0f66f3de-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "620aafc8-f648-4809-8abe-094c0f66f3de" (UID: "620aafc8-f648-4809-8abe-094c0f66f3de"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 12:16:03.198188 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:16:03.198156    2549 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/620aafc8-f648-4809-8abe-094c0f66f3de-oauth-serving-cert\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
May 25 12:16:03.198188 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:16:03.198184    2549 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/620aafc8-f648-4809-8abe-094c0f66f3de-console-oauth-config\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
May 25 12:16:03.198384 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:16:03.198215    2549 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/620aafc8-f648-4809-8abe-094c0f66f3de-service-ca\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
May 25 12:16:03.198384 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:16:03.198224    2549 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/620aafc8-f648-4809-8abe-094c0f66f3de-console-config\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
May 25 12:16:03.198384 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:16:03.198232    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ctrrg\" (UniqueName: \"kubernetes.io/projected/620aafc8-f648-4809-8abe-094c0f66f3de-kube-api-access-ctrrg\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
May 25 12:16:03.198384 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:16:03.198242    2549 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/620aafc8-f648-4809-8abe-094c0f66f3de-console-serving-cert\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
May 25 12:16:03.513864 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:16:03.513835    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-7654684cf-5vvhl_620aafc8-f648-4809-8abe-094c0f66f3de/console/0.log"
May 25 12:16:03.514033 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:16:03.513874    2549 generic.go:358] "Generic (PLEG): container finished" podID="620aafc8-f648-4809-8abe-094c0f66f3de" containerID="056f1b05ec15a68360a22883a381de0431191d5cf5e58b08d76d06ef0a2b56ed" exitCode=2
May 25 12:16:03.514033 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:16:03.513917    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7654684cf-5vvhl" event={"ID":"620aafc8-f648-4809-8abe-094c0f66f3de","Type":"ContainerDied","Data":"056f1b05ec15a68360a22883a381de0431191d5cf5e58b08d76d06ef0a2b56ed"}
May 25 12:16:03.514033 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:16:03.513941    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7654684cf-5vvhl" event={"ID":"620aafc8-f648-4809-8abe-094c0f66f3de","Type":"ContainerDied","Data":"cc68baf0fe0338572716deb27207a2bbfce1c1ac6f4b5e652c40830da36c19ec"}
May 25 12:16:03.514033 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:16:03.513937    2549 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7654684cf-5vvhl"
May 25 12:16:03.514033 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:16:03.513961    2549 scope.go:117] "RemoveContainer" containerID="056f1b05ec15a68360a22883a381de0431191d5cf5e58b08d76d06ef0a2b56ed"
May 25 12:16:03.522775 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:16:03.522756    2549 scope.go:117] "RemoveContainer" containerID="056f1b05ec15a68360a22883a381de0431191d5cf5e58b08d76d06ef0a2b56ed"
May 25 12:16:03.523028 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:16:03.523007    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"056f1b05ec15a68360a22883a381de0431191d5cf5e58b08d76d06ef0a2b56ed\": container with ID starting with 056f1b05ec15a68360a22883a381de0431191d5cf5e58b08d76d06ef0a2b56ed not found: ID does not exist" containerID="056f1b05ec15a68360a22883a381de0431191d5cf5e58b08d76d06ef0a2b56ed"
May 25 12:16:03.523075 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:16:03.523037    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"056f1b05ec15a68360a22883a381de0431191d5cf5e58b08d76d06ef0a2b56ed"} err="failed to get container status \"056f1b05ec15a68360a22883a381de0431191d5cf5e58b08d76d06ef0a2b56ed\": rpc error: code = NotFound desc = could not find container \"056f1b05ec15a68360a22883a381de0431191d5cf5e58b08d76d06ef0a2b56ed\": container with ID starting with 056f1b05ec15a68360a22883a381de0431191d5cf5e58b08d76d06ef0a2b56ed not found: ID does not exist"
May 25 12:16:03.537974 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:16:03.537947    2549 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7654684cf-5vvhl"]
May 25 12:16:03.542612 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:16:03.542590    2549 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7654684cf-5vvhl"]
May 25 12:16:04.848097 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:16:04.848068    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="620aafc8-f648-4809-8abe-094c0f66f3de" path="/var/lib/kubelet/pods/620aafc8-f648-4809-8abe-094c0f66f3de/volumes"
May 25 12:16:05.521744 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:16:05.521707    2549 generic.go:358] "Generic (PLEG): container finished" podID="1dc72bef-ba17-4ede-88bb-0e1a18c43bbf" containerID="41d86f35477064ceb561aa9f63448747e6adb2b73da01bf99719192959b9f80e" exitCode=0
May 25 12:16:05.521935 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:16:05.521786    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-7t27f" event={"ID":"1dc72bef-ba17-4ede-88bb-0e1a18c43bbf","Type":"ContainerDied","Data":"41d86f35477064ceb561aa9f63448747e6adb2b73da01bf99719192959b9f80e"}
May 25 12:16:05.522154 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:16:05.522137    2549 scope.go:117] "RemoveContainer" containerID="41d86f35477064ceb561aa9f63448747e6adb2b73da01bf99719192959b9f80e"
May 25 12:16:06.527477 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:16:06.527443    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-7t27f" event={"ID":"1dc72bef-ba17-4ede-88bb-0e1a18c43bbf","Type":"ContainerStarted","Data":"5036294b58b23962813a682da27cea654ce94c581a9ff4f82816ada12f0e4e5e"}
May 25 12:16:08.119937 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:16:08.119906    2549 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-54c67d8478-vzxxl"
May 25 12:16:08.123699 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:16:08.123680    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-54c67d8478-vzxxl"
May 25 12:16:11.511641 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:16:11.511604    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-ml56g"
May 25 12:16:50.258126 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:16:50.258092    2549 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
May 25 12:16:50.258615 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:16:50.258431    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="620aafc8-f648-4809-8abe-094c0f66f3de" containerName="console"
May 25 12:16:50.258615 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:16:50.258442    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="620aafc8-f648-4809-8abe-094c0f66f3de" containerName="console"
May 25 12:16:50.258615 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:16:50.258505    2549 memory_manager.go:356] "RemoveStaleState removing state" podUID="620aafc8-f648-4809-8abe-094c0f66f3de" containerName="console"
May 25 12:16:50.263334 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:16:50.263313    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
May 25 12:16:50.266848 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:16:50.266745    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-8lp65\""
May 25 12:16:50.266848 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:16:50.266796    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
May 25 12:16:50.266848 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:16:50.266746    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
May 25 12:16:50.266848 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:16:50.266751    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
May 25 12:16:50.267141 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:16:50.266801    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
May 25 12:16:50.267141 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:16:50.266745    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
May 25 12:16:50.267141 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:16:50.266810    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
May 25 12:16:50.267141 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:16:50.266813    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
May 25 12:16:50.267320 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:16:50.267190    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
May 25 12:16:50.272871 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:16:50.272852    2549 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
May 25 12:16:50.273351 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:16:50.273326    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0e0aaee5-af52-40f2-b0be-923d0f8467a5-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"0e0aaee5-af52-40f2-b0be-923d0f8467a5\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 12:16:50.273451 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:16:50.273372    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0e0aaee5-af52-40f2-b0be-923d0f8467a5-web-config\") pod \"alertmanager-main-0\" (UID: \"0e0aaee5-af52-40f2-b0be-923d0f8467a5\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 12:16:50.273451 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:16:50.273430    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0e0aaee5-af52-40f2-b0be-923d0f8467a5-config-out\") pod \"alertmanager-main-0\" (UID: \"0e0aaee5-af52-40f2-b0be-923d0f8467a5\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 12:16:50.273558 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:16:50.273453    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrrbs\" (UniqueName: \"kubernetes.io/projected/0e0aaee5-af52-40f2-b0be-923d0f8467a5-kube-api-access-nrrbs\") pod \"alertmanager-main-0\" (UID: \"0e0aaee5-af52-40f2-b0be-923d0f8467a5\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 12:16:50.273558 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:16:50.273488    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/0e0aaee5-af52-40f2-b0be-923d0f8467a5-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"0e0aaee5-af52-40f2-b0be-923d0f8467a5\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 12:16:50.273558 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:16:50.273515    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/0e0aaee5-af52-40f2-b0be-923d0f8467a5-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"0e0aaee5-af52-40f2-b0be-923d0f8467a5\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 12:16:50.273558 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:16:50.273549    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0e0aaee5-af52-40f2-b0be-923d0f8467a5-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"0e0aaee5-af52-40f2-b0be-923d0f8467a5\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 12:16:50.273747 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:16:50.273573    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0e0aaee5-af52-40f2-b0be-923d0f8467a5-tls-assets\") pod \"alertmanager-main-0\" (UID: \"0e0aaee5-af52-40f2-b0be-923d0f8467a5\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 12:16:50.273747 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:16:50.273600    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/0e0aaee5-af52-40f2-b0be-923d0f8467a5-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"0e0aaee5-af52-40f2-b0be-923d0f8467a5\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 12:16:50.273747 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:16:50.273617    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0e0aaee5-af52-40f2-b0be-923d0f8467a5-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"0e0aaee5-af52-40f2-b0be-923d0f8467a5\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 12:16:50.273747 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:16:50.273646    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0e0aaee5-af52-40f2-b0be-923d0f8467a5-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"0e0aaee5-af52-40f2-b0be-923d0f8467a5\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 12:16:50.273747 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:16:50.273673    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/0e0aaee5-af52-40f2-b0be-923d0f8467a5-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"0e0aaee5-af52-40f2-b0be-923d0f8467a5\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 12:16:50.273747 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:16:50.273694    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/0e0aaee5-af52-40f2-b0be-923d0f8467a5-config-volume\") pod \"alertmanager-main-0\" (UID: \"0e0aaee5-af52-40f2-b0be-923d0f8467a5\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 12:16:50.276775 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:16:50.276754    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
May 25 12:16:50.375121 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:16:50.375079    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0e0aaee5-af52-40f2-b0be-923d0f8467a5-tls-assets\") pod \"alertmanager-main-0\" (UID: \"0e0aaee5-af52-40f2-b0be-923d0f8467a5\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 12:16:50.375121 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:16:50.375123    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/0e0aaee5-af52-40f2-b0be-923d0f8467a5-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"0e0aaee5-af52-40f2-b0be-923d0f8467a5\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 12:16:50.375390 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:16:50.375141    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0e0aaee5-af52-40f2-b0be-923d0f8467a5-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"0e0aaee5-af52-40f2-b0be-923d0f8467a5\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 12:16:50.375390 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:16:50.375172    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0e0aaee5-af52-40f2-b0be-923d0f8467a5-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"0e0aaee5-af52-40f2-b0be-923d0f8467a5\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 12:16:50.375390 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:16:50.375226    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/0e0aaee5-af52-40f2-b0be-923d0f8467a5-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"0e0aaee5-af52-40f2-b0be-923d0f8467a5\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 12:16:50.375390 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:16:50.375249    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/0e0aaee5-af52-40f2-b0be-923d0f8467a5-config-volume\") pod \"alertmanager-main-0\" (UID: \"0e0aaee5-af52-40f2-b0be-923d0f8467a5\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 12:16:50.375390 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:16:50.375286    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0e0aaee5-af52-40f2-b0be-923d0f8467a5-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"0e0aaee5-af52-40f2-b0be-923d0f8467a5\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 12:16:50.375390 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:16:50.375322    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0e0aaee5-af52-40f2-b0be-923d0f8467a5-web-config\") pod \"alertmanager-main-0\" (UID: \"0e0aaee5-af52-40f2-b0be-923d0f8467a5\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 12:16:50.375390 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:16:50.375365    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0e0aaee5-af52-40f2-b0be-923d0f8467a5-config-out\") pod \"alertmanager-main-0\" (UID: \"0e0aaee5-af52-40f2-b0be-923d0f8467a5\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 12:16:50.375740 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:16:50.375401    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-nrrbs\" (UniqueName: \"kubernetes.io/projected/0e0aaee5-af52-40f2-b0be-923d0f8467a5-kube-api-access-nrrbs\") pod \"alertmanager-main-0\" (UID: \"0e0aaee5-af52-40f2-b0be-923d0f8467a5\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 12:16:50.375740 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:16:50.375575    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/0e0aaee5-af52-40f2-b0be-923d0f8467a5-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"0e0aaee5-af52-40f2-b0be-923d0f8467a5\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 12:16:50.376227 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:16:50.375943    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0e0aaee5-af52-40f2-b0be-923d0f8467a5-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"0e0aaee5-af52-40f2-b0be-923d0f8467a5\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 12:16:50.376227 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:16:50.375945    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/0e0aaee5-af52-40f2-b0be-923d0f8467a5-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"0e0aaee5-af52-40f2-b0be-923d0f8467a5\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 12:16:50.376421 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:16:50.376241    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/0e0aaee5-af52-40f2-b0be-923d0f8467a5-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"0e0aaee5-af52-40f2-b0be-923d0f8467a5\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 12:16:50.376421 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:16:50.376260    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0e0aaee5-af52-40f2-b0be-923d0f8467a5-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"0e0aaee5-af52-40f2-b0be-923d0f8467a5\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 12:16:50.376421 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:16:50.376288    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0e0aaee5-af52-40f2-b0be-923d0f8467a5-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"0e0aaee5-af52-40f2-b0be-923d0f8467a5\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 12:16:50.378688 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:16:50.378659    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0e0aaee5-af52-40f2-b0be-923d0f8467a5-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"0e0aaee5-af52-40f2-b0be-923d0f8467a5\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 12:16:50.378802 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:16:50.378781    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0e0aaee5-af52-40f2-b0be-923d0f8467a5-web-config\") pod \"alertmanager-main-0\" (UID: \"0e0aaee5-af52-40f2-b0be-923d0f8467a5\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 12:16:50.379052 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:16:50.379004    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0e0aaee5-af52-40f2-b0be-923d0f8467a5-config-out\") pod \"alertmanager-main-0\" (UID: \"0e0aaee5-af52-40f2-b0be-923d0f8467a5\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 12:16:50.379311 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:16:50.379290    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0e0aaee5-af52-40f2-b0be-923d0f8467a5-tls-assets\") pod \"alertmanager-main-0\" (UID: \"0e0aaee5-af52-40f2-b0be-923d0f8467a5\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 12:16:50.379422 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:16:50.379322    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/0e0aaee5-af52-40f2-b0be-923d0f8467a5-config-volume\") pod \"alertmanager-main-0\" (UID: \"0e0aaee5-af52-40f2-b0be-923d0f8467a5\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 12:16:50.379477 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:16:50.379445    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/0e0aaee5-af52-40f2-b0be-923d0f8467a5-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"0e0aaee5-af52-40f2-b0be-923d0f8467a5\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 12:16:50.379734 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:16:50.379709    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/0e0aaee5-af52-40f2-b0be-923d0f8467a5-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"0e0aaee5-af52-40f2-b0be-923d0f8467a5\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 12:16:50.379869 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:16:50.379851    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0e0aaee5-af52-40f2-b0be-923d0f8467a5-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"0e0aaee5-af52-40f2-b0be-923d0f8467a5\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 12:16:50.380389 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:16:50.380368    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/0e0aaee5-af52-40f2-b0be-923d0f8467a5-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"0e0aaee5-af52-40f2-b0be-923d0f8467a5\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 12:16:50.389467 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:16:50.389451    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrrbs\" (UniqueName: \"kubernetes.io/projected/0e0aaee5-af52-40f2-b0be-923d0f8467a5-kube-api-access-nrrbs\") pod \"alertmanager-main-0\" (UID: \"0e0aaee5-af52-40f2-b0be-923d0f8467a5\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 12:16:50.573653 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:16:50.573569    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
May 25 12:16:50.719114 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:16:50.719081    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
May 25 12:16:50.722967 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:16:50.722936    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e0aaee5_af52_40f2_b0be_923d0f8467a5.slice/crio-4e16099fcd25d96854a1f9a55d63836372f35f060ff89430fd8f911c71ebae4e WatchSource:0}: Error finding container 4e16099fcd25d96854a1f9a55d63836372f35f060ff89430fd8f911c71ebae4e: Status 404 returned error can't find the container with id 4e16099fcd25d96854a1f9a55d63836372f35f060ff89430fd8f911c71ebae4e
May 25 12:16:51.511148 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:16:51.511110    2549 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console/console-65fff857b4-rbmpj"]
May 25 12:16:51.514900 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:16:51.514881    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65fff857b4-rbmpj"
May 25 12:16:51.530058 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:16:51.530033    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-65fff857b4-rbmpj"]
May 25 12:16:51.586854 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:16:51.586814    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/89a53512-040a-4621-b9ec-386b21177fc5-service-ca\") pod \"console-65fff857b4-rbmpj\" (UID: \"89a53512-040a-4621-b9ec-386b21177fc5\") " pod="openshift-console/console-65fff857b4-rbmpj"
May 25 12:16:51.587029 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:16:51.586861    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/89a53512-040a-4621-b9ec-386b21177fc5-console-oauth-config\") pod \"console-65fff857b4-rbmpj\" (UID: \"89a53512-040a-4621-b9ec-386b21177fc5\") " pod="openshift-console/console-65fff857b4-rbmpj"
May 25 12:16:51.587029 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:16:51.586888    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/89a53512-040a-4621-b9ec-386b21177fc5-trusted-ca-bundle\") pod \"console-65fff857b4-rbmpj\" (UID: \"89a53512-040a-4621-b9ec-386b21177fc5\") " pod="openshift-console/console-65fff857b4-rbmpj"
May 25 12:16:51.587029 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:16:51.586906    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9vx2\" (UniqueName: \"kubernetes.io/projected/89a53512-040a-4621-b9ec-386b21177fc5-kube-api-access-b9vx2\") pod \"console-65fff857b4-rbmpj\" (UID: \"89a53512-040a-4621-b9ec-386b21177fc5\") " pod="openshift-console/console-65fff857b4-rbmpj"
May 25 12:16:51.587029 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:16:51.586942    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/89a53512-040a-4621-b9ec-386b21177fc5-console-serving-cert\") pod \"console-65fff857b4-rbmpj\" (UID: \"89a53512-040a-4621-b9ec-386b21177fc5\") " pod="openshift-console/console-65fff857b4-rbmpj"
May 25 12:16:51.587029 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:16:51.587006    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/89a53512-040a-4621-b9ec-386b21177fc5-console-config\") pod \"console-65fff857b4-rbmpj\" (UID: \"89a53512-040a-4621-b9ec-386b21177fc5\") " pod="openshift-console/console-65fff857b4-rbmpj"
May 25 12:16:51.587185 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:16:51.587059    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/89a53512-040a-4621-b9ec-386b21177fc5-oauth-serving-cert\") pod \"console-65fff857b4-rbmpj\" (UID: \"89a53512-040a-4621-b9ec-386b21177fc5\") " pod="openshift-console/console-65fff857b4-rbmpj"
May 25 12:16:51.669005 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:16:51.668972    2549 generic.go:358] "Generic (PLEG): container finished" podID="0e0aaee5-af52-40f2-b0be-923d0f8467a5" containerID="e0cbd572e00345081b912b289d67d450618bbd2791463dfb3e1f75cf8d16a270" exitCode=0
May 25 12:16:51.669170 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:16:51.669065    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0e0aaee5-af52-40f2-b0be-923d0f8467a5","Type":"ContainerDied","Data":"e0cbd572e00345081b912b289d67d450618bbd2791463dfb3e1f75cf8d16a270"}
May 25 12:16:51.669170 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:16:51.669102    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0e0aaee5-af52-40f2-b0be-923d0f8467a5","Type":"ContainerStarted","Data":"4e16099fcd25d96854a1f9a55d63836372f35f060ff89430fd8f911c71ebae4e"}
May 25 12:16:51.687444 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:16:51.687418    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/89a53512-040a-4621-b9ec-386b21177fc5-console-serving-cert\") pod \"console-65fff857b4-rbmpj\" (UID: \"89a53512-040a-4621-b9ec-386b21177fc5\") " pod="openshift-console/console-65fff857b4-rbmpj"
May 25 12:16:51.687555 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:16:51.687475    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/89a53512-040a-4621-b9ec-386b21177fc5-console-config\") pod \"console-65fff857b4-rbmpj\" (UID: \"89a53512-040a-4621-b9ec-386b21177fc5\") " pod="openshift-console/console-65fff857b4-rbmpj"
May 25 12:16:51.687737 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:16:51.687691    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/89a53512-040a-4621-b9ec-386b21177fc5-oauth-serving-cert\") pod \"console-65fff857b4-rbmpj\" (UID: \"89a53512-040a-4621-b9ec-386b21177fc5\") " pod="openshift-console/console-65fff857b4-rbmpj"
May 25 12:16:51.687865 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:16:51.687788    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/89a53512-040a-4621-b9ec-386b21177fc5-service-ca\") pod \"console-65fff857b4-rbmpj\" (UID: \"89a53512-040a-4621-b9ec-386b21177fc5\") " pod="openshift-console/console-65fff857b4-rbmpj"
May 25 12:16:51.688475 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:16:51.688010    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/89a53512-040a-4621-b9ec-386b21177fc5-console-oauth-config\") pod \"console-65fff857b4-rbmpj\" (UID: \"89a53512-040a-4621-b9ec-386b21177fc5\") " pod="openshift-console/console-65fff857b4-rbmpj"
May 25 12:16:51.688475 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:16:51.688065    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/89a53512-040a-4621-b9ec-386b21177fc5-trusted-ca-bundle\") pod \"console-65fff857b4-rbmpj\" (UID: \"89a53512-040a-4621-b9ec-386b21177fc5\") " pod="openshift-console/console-65fff857b4-rbmpj"
May 25 12:16:51.688475 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:16:51.688091    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-b9vx2\" (UniqueName: \"kubernetes.io/projected/89a53512-040a-4621-b9ec-386b21177fc5-kube-api-access-b9vx2\") pod \"console-65fff857b4-rbmpj\" (UID: \"89a53512-040a-4621-b9ec-386b21177fc5\") " pod="openshift-console/console-65fff857b4-rbmpj"
May 25 12:16:51.688475 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:16:51.688283    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/89a53512-040a-4621-b9ec-386b21177fc5-console-config\") pod \"console-65fff857b4-rbmpj\" (UID: \"89a53512-040a-4621-b9ec-386b21177fc5\") " pod="openshift-console/console-65fff857b4-rbmpj"
May 25 12:16:51.688475 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:16:51.688398    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/89a53512-040a-4621-b9ec-386b21177fc5-oauth-serving-cert\") pod \"console-65fff857b4-rbmpj\" (UID: \"89a53512-040a-4621-b9ec-386b21177fc5\") " pod="openshift-console/console-65fff857b4-rbmpj"
May 25 12:16:51.688475 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:16:51.688471    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/89a53512-040a-4621-b9ec-386b21177fc5-service-ca\") pod \"console-65fff857b4-rbmpj\" (UID: \"89a53512-040a-4621-b9ec-386b21177fc5\") " pod="openshift-console/console-65fff857b4-rbmpj"
May 25 12:16:51.689038 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:16:51.689016    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/89a53512-040a-4621-b9ec-386b21177fc5-trusted-ca-bundle\") pod \"console-65fff857b4-rbmpj\" (UID: \"89a53512-040a-4621-b9ec-386b21177fc5\") " pod="openshift-console/console-65fff857b4-rbmpj"
May 25 12:16:51.690039 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:16:51.690024    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/89a53512-040a-4621-b9ec-386b21177fc5-console-serving-cert\") pod \"console-65fff857b4-rbmpj\" (UID: \"89a53512-040a-4621-b9ec-386b21177fc5\") " pod="openshift-console/console-65fff857b4-rbmpj"
May 25 12:16:51.690554 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:16:51.690532    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/89a53512-040a-4621-b9ec-386b21177fc5-console-oauth-config\") pod \"console-65fff857b4-rbmpj\" (UID: \"89a53512-040a-4621-b9ec-386b21177fc5\") " pod="openshift-console/console-65fff857b4-rbmpj"
May 25 12:16:51.698881 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:16:51.698856    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9vx2\" (UniqueName: \"kubernetes.io/projected/89a53512-040a-4621-b9ec-386b21177fc5-kube-api-access-b9vx2\") pod \"console-65fff857b4-rbmpj\" (UID: \"89a53512-040a-4621-b9ec-386b21177fc5\") " pod="openshift-console/console-65fff857b4-rbmpj"
May 25 12:16:51.823658 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:16:51.823572    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65fff857b4-rbmpj"
May 25 12:16:51.958262 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:16:51.958232    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-65fff857b4-rbmpj"]
May 25 12:16:51.961113 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:16:51.961085    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89a53512_040a_4621_b9ec_386b21177fc5.slice/crio-b4737e8c59598ec23e7244eb1efb741174421a247bab0d88970bac6b8eaa6972 WatchSource:0}: Error finding container b4737e8c59598ec23e7244eb1efb741174421a247bab0d88970bac6b8eaa6972: Status 404 returned error can't find the container with id b4737e8c59598ec23e7244eb1efb741174421a247bab0d88970bac6b8eaa6972
May 25 12:16:52.673359 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:16:52.673324    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65fff857b4-rbmpj" event={"ID":"89a53512-040a-4621-b9ec-386b21177fc5","Type":"ContainerStarted","Data":"005c340cfac08932427133540638fc4d94f7c4d41a4659e65c5cb87943182c6f"}
May 25 12:16:52.673359 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:16:52.673364    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65fff857b4-rbmpj" event={"ID":"89a53512-040a-4621-b9ec-386b21177fc5","Type":"ContainerStarted","Data":"b4737e8c59598ec23e7244eb1efb741174421a247bab0d88970bac6b8eaa6972"}
May 25 12:16:52.712993 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:16:52.712909    2549 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console/console-65fff857b4-rbmpj" podStartSLOduration=1.712888064 podStartE2EDuration="1.712888064s" podCreationTimestamp="2026-05-25 12:16:51 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 12:16:52.712236366 +0000 UTC m=+216.428051654" watchObservedRunningTime="2026-05-25 12:16:52.712888064 +0000 UTC m=+216.428703352"
May 25 12:16:53.679265 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:16:53.679238    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0e0aaee5-af52-40f2-b0be-923d0f8467a5","Type":"ContainerStarted","Data":"39484c86861684c6ab59a8cb435dd07996e6ca1cc7aa20adcb3fed740a93d9cf"}
May 25 12:16:53.679625 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:16:53.679270    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0e0aaee5-af52-40f2-b0be-923d0f8467a5","Type":"ContainerStarted","Data":"6a742d7d33cc91bd4970e5f7ac1cfba337b796d9c7192ade20fa508e48965ae7"}
May 25 12:16:53.679625 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:16:53.679280    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0e0aaee5-af52-40f2-b0be-923d0f8467a5","Type":"ContainerStarted","Data":"245d504398a9b4d65cc07c7bf1dfa6165479ca002a934f7492ea2c8e6a1fee45"}
May 25 12:16:53.679625 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:16:53.679288    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0e0aaee5-af52-40f2-b0be-923d0f8467a5","Type":"ContainerStarted","Data":"cc22518a40ee36035350d28d2b0173e25900ecd3fd849eb442992e796d19d53e"}
May 25 12:16:54.685289 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:16:54.685251    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0e0aaee5-af52-40f2-b0be-923d0f8467a5","Type":"ContainerStarted","Data":"ad630d8d0a908fe1e06ac97e0cdbeba09c2ead3898b7167220df4d38ce5f58b2"}
May 25 12:16:54.685289 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:16:54.685289    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0e0aaee5-af52-40f2-b0be-923d0f8467a5","Type":"ContainerStarted","Data":"531cc622dd84f89ad665e2e5cc6f7f2c4efca1d62cec5d93a6acc8845354dd46"}
May 25 12:16:54.723893 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:16:54.723836    2549 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.011507895 podStartE2EDuration="4.723819945s" podCreationTimestamp="2026-05-25 12:16:50 +0000 UTC" firstStartedPulling="2026-05-25 12:16:51.670274687 +0000 UTC m=+215.386089952" lastFinishedPulling="2026-05-25 12:16:53.38258672 +0000 UTC m=+217.098402002" observedRunningTime="2026-05-25 12:16:54.722013061 +0000 UTC m=+218.437828348" watchObservedRunningTime="2026-05-25 12:16:54.723819945 +0000 UTC m=+218.439635231"
May 25 12:17:01.824454 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:17:01.824406    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-65fff857b4-rbmpj"
May 25 12:17:01.824912 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:17:01.824468    2549 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-65fff857b4-rbmpj"
May 25 12:17:01.829069 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:17:01.829046    2549 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-65fff857b4-rbmpj"
May 25 12:17:02.713298 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:17:02.713269    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-65fff857b4-rbmpj"
May 25 12:17:02.766684 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:17:02.766647    2549 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-54c67d8478-vzxxl"]
May 25 12:17:14.556905 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:17:14.556869    2549 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-cxn5k"]
May 25 12:17:14.559144 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:17:14.559127    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cxn5k"
May 25 12:17:14.561918 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:17:14.561887    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
May 25 12:17:14.567485 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:17:14.567456    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-cxn5k"]
May 25 12:17:14.691729 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:17:14.691674    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/271c3650-d1f0-4e2c-93f7-a2a9fd04f3b8-original-pull-secret\") pod \"global-pull-secret-syncer-cxn5k\" (UID: \"271c3650-d1f0-4e2c-93f7-a2a9fd04f3b8\") " pod="kube-system/global-pull-secret-syncer-cxn5k"
May 25 12:17:14.691910 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:17:14.691753    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/271c3650-d1f0-4e2c-93f7-a2a9fd04f3b8-kubelet-config\") pod \"global-pull-secret-syncer-cxn5k\" (UID: \"271c3650-d1f0-4e2c-93f7-a2a9fd04f3b8\") " pod="kube-system/global-pull-secret-syncer-cxn5k"
May 25 12:17:14.691910 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:17:14.691786    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/271c3650-d1f0-4e2c-93f7-a2a9fd04f3b8-dbus\") pod \"global-pull-secret-syncer-cxn5k\" (UID: \"271c3650-d1f0-4e2c-93f7-a2a9fd04f3b8\") " pod="kube-system/global-pull-secret-syncer-cxn5k"
May 25 12:17:14.792701 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:17:14.792668    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/271c3650-d1f0-4e2c-93f7-a2a9fd04f3b8-kubelet-config\") pod \"global-pull-secret-syncer-cxn5k\" (UID: \"271c3650-d1f0-4e2c-93f7-a2a9fd04f3b8\") " pod="kube-system/global-pull-secret-syncer-cxn5k"
May 25 12:17:14.792701 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:17:14.792704    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/271c3650-d1f0-4e2c-93f7-a2a9fd04f3b8-dbus\") pod \"global-pull-secret-syncer-cxn5k\" (UID: \"271c3650-d1f0-4e2c-93f7-a2a9fd04f3b8\") " pod="kube-system/global-pull-secret-syncer-cxn5k"
May 25 12:17:14.792901 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:17:14.792760    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/271c3650-d1f0-4e2c-93f7-a2a9fd04f3b8-original-pull-secret\") pod \"global-pull-secret-syncer-cxn5k\" (UID: \"271c3650-d1f0-4e2c-93f7-a2a9fd04f3b8\") " pod="kube-system/global-pull-secret-syncer-cxn5k"
May 25 12:17:14.792901 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:17:14.792805    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/271c3650-d1f0-4e2c-93f7-a2a9fd04f3b8-kubelet-config\") pod \"global-pull-secret-syncer-cxn5k\" (UID: \"271c3650-d1f0-4e2c-93f7-a2a9fd04f3b8\") " pod="kube-system/global-pull-secret-syncer-cxn5k"
May 25 12:17:14.792966 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:17:14.792925    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/271c3650-d1f0-4e2c-93f7-a2a9fd04f3b8-dbus\") pod \"global-pull-secret-syncer-cxn5k\" (UID: \"271c3650-d1f0-4e2c-93f7-a2a9fd04f3b8\") " pod="kube-system/global-pull-secret-syncer-cxn5k"
May 25 12:17:14.794983 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:17:14.794958    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/271c3650-d1f0-4e2c-93f7-a2a9fd04f3b8-original-pull-secret\") pod \"global-pull-secret-syncer-cxn5k\" (UID: \"271c3650-d1f0-4e2c-93f7-a2a9fd04f3b8\") " pod="kube-system/global-pull-secret-syncer-cxn5k"
May 25 12:17:14.868541 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:17:14.868517    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cxn5k"
May 25 12:17:14.994658 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:17:14.994635    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-cxn5k"]
May 25 12:17:14.996936 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:17:14.996904    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod271c3650_d1f0_4e2c_93f7_a2a9fd04f3b8.slice/crio-565553e7aace8c21ad55d764ee243219fb566e41e594ca2987cb4702e74003a1 WatchSource:0}: Error finding container 565553e7aace8c21ad55d764ee243219fb566e41e594ca2987cb4702e74003a1: Status 404 returned error can't find the container with id 565553e7aace8c21ad55d764ee243219fb566e41e594ca2987cb4702e74003a1
May 25 12:17:15.752258 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:17:15.752215    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-cxn5k" event={"ID":"271c3650-d1f0-4e2c-93f7-a2a9fd04f3b8","Type":"ContainerStarted","Data":"565553e7aace8c21ad55d764ee243219fb566e41e594ca2987cb4702e74003a1"}
May 25 12:17:19.771322 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:17:19.771284    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-cxn5k" event={"ID":"271c3650-d1f0-4e2c-93f7-a2a9fd04f3b8","Type":"ContainerStarted","Data":"61af8319d26ae0e0adaeb98cde44e5226b1bcfda81dd223e1ea67f8f6cf9b1df"}
May 25 12:17:19.793555 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:17:19.793508    2549 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-cxn5k" podStartSLOduration=1.698910711 podStartE2EDuration="5.793495229s" podCreationTimestamp="2026-05-25 12:17:14 +0000 UTC" firstStartedPulling="2026-05-25 12:17:14.99870698 +0000 UTC m=+238.714522250" lastFinishedPulling="2026-05-25 12:17:19.093291503 +0000 UTC m=+242.809106768" observedRunningTime="2026-05-25 12:17:19.791062046 +0000 UTC m=+243.506877347" watchObservedRunningTime="2026-05-25 12:17:19.793495229 +0000 UTC m=+243.509310515"
May 25 12:17:27.787239 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:17:27.787159    2549 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-console/console-54c67d8478-vzxxl" podUID="f1fc499f-2c5a-4c4d-849e-f8e3453a2026" containerName="console" containerID="cri-o://21b61279d43a74ffcf3b02fcc78580e8a5b65dabe2ef3b4cfa314d6e10987fe9" gracePeriod=15
May 25 12:17:28.034756 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:17:28.034730    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-54c67d8478-vzxxl_f1fc499f-2c5a-4c4d-849e-f8e3453a2026/console/0.log"
May 25 12:17:28.034876 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:17:28.034794    2549 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-54c67d8478-vzxxl"
May 25 12:17:28.214010 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:17:28.213960    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f1fc499f-2c5a-4c4d-849e-f8e3453a2026-trusted-ca-bundle\") pod \"f1fc499f-2c5a-4c4d-849e-f8e3453a2026\" (UID: \"f1fc499f-2c5a-4c4d-849e-f8e3453a2026\") "
May 25 12:17:28.214187 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:17:28.214032    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f1fc499f-2c5a-4c4d-849e-f8e3453a2026-console-serving-cert\") pod \"f1fc499f-2c5a-4c4d-849e-f8e3453a2026\" (UID: \"f1fc499f-2c5a-4c4d-849e-f8e3453a2026\") "
May 25 12:17:28.214187 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:17:28.214051    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f1fc499f-2c5a-4c4d-849e-f8e3453a2026-console-config\") pod \"f1fc499f-2c5a-4c4d-849e-f8e3453a2026\" (UID: \"f1fc499f-2c5a-4c4d-849e-f8e3453a2026\") "
May 25 12:17:28.214187 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:17:28.214075    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f1fc499f-2c5a-4c4d-849e-f8e3453a2026-oauth-serving-cert\") pod \"f1fc499f-2c5a-4c4d-849e-f8e3453a2026\" (UID: \"f1fc499f-2c5a-4c4d-849e-f8e3453a2026\") "
May 25 12:17:28.214187 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:17:28.214104    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f1fc499f-2c5a-4c4d-849e-f8e3453a2026-service-ca\") pod \"f1fc499f-2c5a-4c4d-849e-f8e3453a2026\" (UID: \"f1fc499f-2c5a-4c4d-849e-f8e3453a2026\") "
May 25 12:17:28.214187 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:17:28.214146    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f1fc499f-2c5a-4c4d-849e-f8e3453a2026-console-oauth-config\") pod \"f1fc499f-2c5a-4c4d-849e-f8e3453a2026\" (UID: \"f1fc499f-2c5a-4c4d-849e-f8e3453a2026\") "
May 25 12:17:28.214187 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:17:28.214176    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p25dc\" (UniqueName: \"kubernetes.io/projected/f1fc499f-2c5a-4c4d-849e-f8e3453a2026-kube-api-access-p25dc\") pod \"f1fc499f-2c5a-4c4d-849e-f8e3453a2026\" (UID: \"f1fc499f-2c5a-4c4d-849e-f8e3453a2026\") "
May 25 12:17:28.214542 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:17:28.214513    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1fc499f-2c5a-4c4d-849e-f8e3453a2026-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "f1fc499f-2c5a-4c4d-849e-f8e3453a2026" (UID: "f1fc499f-2c5a-4c4d-849e-f8e3453a2026"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 12:17:28.214661 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:17:28.214508    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1fc499f-2c5a-4c4d-849e-f8e3453a2026-console-config" (OuterVolumeSpecName: "console-config") pod "f1fc499f-2c5a-4c4d-849e-f8e3453a2026" (UID: "f1fc499f-2c5a-4c4d-849e-f8e3453a2026"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 12:17:28.214661 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:17:28.214577    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1fc499f-2c5a-4c4d-849e-f8e3453a2026-service-ca" (OuterVolumeSpecName: "service-ca") pod "f1fc499f-2c5a-4c4d-849e-f8e3453a2026" (UID: "f1fc499f-2c5a-4c4d-849e-f8e3453a2026"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 12:17:28.214661 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:17:28.214576    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1fc499f-2c5a-4c4d-849e-f8e3453a2026-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "f1fc499f-2c5a-4c4d-849e-f8e3453a2026" (UID: "f1fc499f-2c5a-4c4d-849e-f8e3453a2026"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 12:17:28.216410 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:17:28.216376    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1fc499f-2c5a-4c4d-849e-f8e3453a2026-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "f1fc499f-2c5a-4c4d-849e-f8e3453a2026" (UID: "f1fc499f-2c5a-4c4d-849e-f8e3453a2026"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 12:17:28.216561 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:17:28.216538    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1fc499f-2c5a-4c4d-849e-f8e3453a2026-kube-api-access-p25dc" (OuterVolumeSpecName: "kube-api-access-p25dc") pod "f1fc499f-2c5a-4c4d-849e-f8e3453a2026" (UID: "f1fc499f-2c5a-4c4d-849e-f8e3453a2026"). InnerVolumeSpecName "kube-api-access-p25dc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 12:17:28.216603 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:17:28.216539    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1fc499f-2c5a-4c4d-849e-f8e3453a2026-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "f1fc499f-2c5a-4c4d-849e-f8e3453a2026" (UID: "f1fc499f-2c5a-4c4d-849e-f8e3453a2026"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 12:17:28.315252 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:17:28.315182    2549 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f1fc499f-2c5a-4c4d-849e-f8e3453a2026-oauth-serving-cert\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
May 25 12:17:28.315252 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:17:28.315251    2549 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f1fc499f-2c5a-4c4d-849e-f8e3453a2026-service-ca\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
May 25 12:17:28.315457 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:17:28.315266    2549 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f1fc499f-2c5a-4c4d-849e-f8e3453a2026-console-oauth-config\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
May 25 12:17:28.315457 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:17:28.315279    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p25dc\" (UniqueName: \"kubernetes.io/projected/f1fc499f-2c5a-4c4d-849e-f8e3453a2026-kube-api-access-p25dc\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
May 25 12:17:28.315457 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:17:28.315293    2549 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f1fc499f-2c5a-4c4d-849e-f8e3453a2026-trusted-ca-bundle\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
May 25 12:17:28.315457 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:17:28.315305    2549 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f1fc499f-2c5a-4c4d-849e-f8e3453a2026-console-serving-cert\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
May 25 12:17:28.315457 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:17:28.315316    2549 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f1fc499f-2c5a-4c4d-849e-f8e3453a2026-console-config\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
May 25 12:17:28.800107 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:17:28.800079    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-54c67d8478-vzxxl_f1fc499f-2c5a-4c4d-849e-f8e3453a2026/console/0.log"
May 25 12:17:28.800581 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:17:28.800119    2549 generic.go:358] "Generic (PLEG): container finished" podID="f1fc499f-2c5a-4c4d-849e-f8e3453a2026" containerID="21b61279d43a74ffcf3b02fcc78580e8a5b65dabe2ef3b4cfa314d6e10987fe9" exitCode=2
May 25 12:17:28.800581 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:17:28.800216    2549 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-54c67d8478-vzxxl"
May 25 12:17:28.800581 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:17:28.800224    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-54c67d8478-vzxxl" event={"ID":"f1fc499f-2c5a-4c4d-849e-f8e3453a2026","Type":"ContainerDied","Data":"21b61279d43a74ffcf3b02fcc78580e8a5b65dabe2ef3b4cfa314d6e10987fe9"}
May 25 12:17:28.800581 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:17:28.800259    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-54c67d8478-vzxxl" event={"ID":"f1fc499f-2c5a-4c4d-849e-f8e3453a2026","Type":"ContainerDied","Data":"3474d600c859fc326d7f5c62f5298545c75ac19c9f762cdbe2cbe9db617a14a0"}
May 25 12:17:28.800581 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:17:28.800277    2549 scope.go:117] "RemoveContainer" containerID="21b61279d43a74ffcf3b02fcc78580e8a5b65dabe2ef3b4cfa314d6e10987fe9"
May 25 12:17:28.808629 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:17:28.808614    2549 scope.go:117] "RemoveContainer" containerID="21b61279d43a74ffcf3b02fcc78580e8a5b65dabe2ef3b4cfa314d6e10987fe9"
May 25 12:17:28.808844 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:17:28.808827    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21b61279d43a74ffcf3b02fcc78580e8a5b65dabe2ef3b4cfa314d6e10987fe9\": container with ID starting with 21b61279d43a74ffcf3b02fcc78580e8a5b65dabe2ef3b4cfa314d6e10987fe9 not found: ID does not exist" containerID="21b61279d43a74ffcf3b02fcc78580e8a5b65dabe2ef3b4cfa314d6e10987fe9"
May 25 12:17:28.808901 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:17:28.808857    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21b61279d43a74ffcf3b02fcc78580e8a5b65dabe2ef3b4cfa314d6e10987fe9"} err="failed to get container status \"21b61279d43a74ffcf3b02fcc78580e8a5b65dabe2ef3b4cfa314d6e10987fe9\": rpc error: code = NotFound desc = could not find container \"21b61279d43a74ffcf3b02fcc78580e8a5b65dabe2ef3b4cfa314d6e10987fe9\": container with ID starting with 21b61279d43a74ffcf3b02fcc78580e8a5b65dabe2ef3b4cfa314d6e10987fe9 not found: ID does not exist"
May 25 12:17:28.834991 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:17:28.834967    2549 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-54c67d8478-vzxxl"]
May 25 12:17:28.847022 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:17:28.847000    2549 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-54c67d8478-vzxxl"]
May 25 12:17:30.846705 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:17:30.846671    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f1fc499f-2c5a-4c4d-849e-f8e3453a2026" path="/var/lib/kubelet/pods/f1fc499f-2c5a-4c4d-849e-f8e3453a2026/volumes"
May 25 12:17:35.711954 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:17:35.711865    2549 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c2gcx4"]
May 25 12:17:35.712522 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:17:35.712420    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f1fc499f-2c5a-4c4d-849e-f8e3453a2026" containerName="console"
May 25 12:17:35.712522 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:17:35.712440    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="f1fc499f-2c5a-4c4d-849e-f8e3453a2026" containerName="console"
May 25 12:17:35.712641 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:17:35.712533    2549 memory_manager.go:356] "RemoveStaleState removing state" podUID="f1fc499f-2c5a-4c4d-849e-f8e3453a2026" containerName="console"
May 25 12:17:35.715736 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:17:35.715714    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c2gcx4"
May 25 12:17:35.718780 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:17:35.718753    2549 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
May 25 12:17:35.718891 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:17:35.718796    2549 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
May 25 12:17:35.720023 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:17:35.720006    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-wbdd2\""
May 25 12:17:35.724955 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:17:35.724930    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c2gcx4"]
May 25 12:17:35.779658 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:17:35.779628    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8972\" (UniqueName: \"kubernetes.io/projected/0c6d3a0d-2cd7-492f-bba4-c4b9ce4957b6-kube-api-access-r8972\") pod \"59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c2gcx4\" (UID: \"0c6d3a0d-2cd7-492f-bba4-c4b9ce4957b6\") " pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c2gcx4"
May 25 12:17:35.779798 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:17:35.779670    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0c6d3a0d-2cd7-492f-bba4-c4b9ce4957b6-bundle\") pod \"59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c2gcx4\" (UID: \"0c6d3a0d-2cd7-492f-bba4-c4b9ce4957b6\") " pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c2gcx4"
May 25 12:17:35.779798 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:17:35.779738    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0c6d3a0d-2cd7-492f-bba4-c4b9ce4957b6-util\") pod \"59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c2gcx4\" (UID: \"0c6d3a0d-2cd7-492f-bba4-c4b9ce4957b6\") " pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c2gcx4"
May 25 12:17:35.880377 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:17:35.880342    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0c6d3a0d-2cd7-492f-bba4-c4b9ce4957b6-bundle\") pod \"59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c2gcx4\" (UID: \"0c6d3a0d-2cd7-492f-bba4-c4b9ce4957b6\") " pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c2gcx4"
May 25 12:17:35.880526 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:17:35.880400    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0c6d3a0d-2cd7-492f-bba4-c4b9ce4957b6-util\") pod \"59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c2gcx4\" (UID: \"0c6d3a0d-2cd7-492f-bba4-c4b9ce4957b6\") " pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c2gcx4"
May 25 12:17:35.880526 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:17:35.880454    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-r8972\" (UniqueName: \"kubernetes.io/projected/0c6d3a0d-2cd7-492f-bba4-c4b9ce4957b6-kube-api-access-r8972\") pod \"59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c2gcx4\" (UID: \"0c6d3a0d-2cd7-492f-bba4-c4b9ce4957b6\") " pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c2gcx4"
May 25 12:17:35.880768 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:17:35.880749    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0c6d3a0d-2cd7-492f-bba4-c4b9ce4957b6-bundle\") pod \"59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c2gcx4\" (UID: \"0c6d3a0d-2cd7-492f-bba4-c4b9ce4957b6\") " pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c2gcx4"
May 25 12:17:35.880805 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:17:35.880786    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0c6d3a0d-2cd7-492f-bba4-c4b9ce4957b6-util\") pod \"59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c2gcx4\" (UID: \"0c6d3a0d-2cd7-492f-bba4-c4b9ce4957b6\") " pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c2gcx4"
May 25 12:17:35.890610 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:17:35.890586    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8972\" (UniqueName: \"kubernetes.io/projected/0c6d3a0d-2cd7-492f-bba4-c4b9ce4957b6-kube-api-access-r8972\") pod \"59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c2gcx4\" (UID: \"0c6d3a0d-2cd7-492f-bba4-c4b9ce4957b6\") " pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c2gcx4"
May 25 12:17:36.026046 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:17:36.025971    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c2gcx4"
May 25 12:17:36.158165 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:17:36.158143    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c2gcx4"]
May 25 12:17:36.160520 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:17:36.160492    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c6d3a0d_2cd7_492f_bba4_c4b9ce4957b6.slice/crio-cb1e831bfe33c627529569a756662345420b7d6cbebab2087c911379effc23fb WatchSource:0}: Error finding container cb1e831bfe33c627529569a756662345420b7d6cbebab2087c911379effc23fb: Status 404 returned error can't find the container with id cb1e831bfe33c627529569a756662345420b7d6cbebab2087c911379effc23fb
May 25 12:17:36.826825 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:17:36.826788    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c2gcx4" event={"ID":"0c6d3a0d-2cd7-492f-bba4-c4b9ce4957b6","Type":"ContainerStarted","Data":"cb1e831bfe33c627529569a756662345420b7d6cbebab2087c911379effc23fb"}
May 25 12:17:42.848223 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:17:42.848168    2549 generic.go:358] "Generic (PLEG): container finished" podID="0c6d3a0d-2cd7-492f-bba4-c4b9ce4957b6" containerID="640c33847867958f3ad3bdd773a2fdda4513495ef59351e57b1ec29de93cfdc7" exitCode=0
May 25 12:17:42.848690 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:17:42.848225    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c2gcx4" event={"ID":"0c6d3a0d-2cd7-492f-bba4-c4b9ce4957b6","Type":"ContainerDied","Data":"640c33847867958f3ad3bdd773a2fdda4513495ef59351e57b1ec29de93cfdc7"}
May 25 12:17:45.866806 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:17:45.866772    2549 generic.go:358] "Generic (PLEG): container finished" podID="0c6d3a0d-2cd7-492f-bba4-c4b9ce4957b6" containerID="208cb01ba2e8a9cc4a49fb3524a5d3cabc6193454633af737658baa0e608185b" exitCode=0
May 25 12:17:45.867341 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:17:45.866862    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c2gcx4" event={"ID":"0c6d3a0d-2cd7-492f-bba4-c4b9ce4957b6","Type":"ContainerDied","Data":"208cb01ba2e8a9cc4a49fb3524a5d3cabc6193454633af737658baa0e608185b"}
May 25 12:17:52.893556 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:17:52.893521    2549 generic.go:358] "Generic (PLEG): container finished" podID="0c6d3a0d-2cd7-492f-bba4-c4b9ce4957b6" containerID="b14932f47bc7e7b5ed93d9ba26edb6ff29f0fb6982377e8e7ea072b582641465" exitCode=0
May 25 12:17:52.893556 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:17:52.893561    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c2gcx4" event={"ID":"0c6d3a0d-2cd7-492f-bba4-c4b9ce4957b6","Type":"ContainerDied","Data":"b14932f47bc7e7b5ed93d9ba26edb6ff29f0fb6982377e8e7ea072b582641465"}
May 25 12:17:54.021033 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:17:54.021005    2549 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c2gcx4"
May 25 12:17:54.143228 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:17:54.143177    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r8972\" (UniqueName: \"kubernetes.io/projected/0c6d3a0d-2cd7-492f-bba4-c4b9ce4957b6-kube-api-access-r8972\") pod \"0c6d3a0d-2cd7-492f-bba4-c4b9ce4957b6\" (UID: \"0c6d3a0d-2cd7-492f-bba4-c4b9ce4957b6\") "
May 25 12:17:54.143392 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:17:54.143284    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0c6d3a0d-2cd7-492f-bba4-c4b9ce4957b6-bundle\") pod \"0c6d3a0d-2cd7-492f-bba4-c4b9ce4957b6\" (UID: \"0c6d3a0d-2cd7-492f-bba4-c4b9ce4957b6\") "
May 25 12:17:54.143392 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:17:54.143303    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0c6d3a0d-2cd7-492f-bba4-c4b9ce4957b6-util\") pod \"0c6d3a0d-2cd7-492f-bba4-c4b9ce4957b6\" (UID: \"0c6d3a0d-2cd7-492f-bba4-c4b9ce4957b6\") "
May 25 12:17:54.143781 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:17:54.143751    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c6d3a0d-2cd7-492f-bba4-c4b9ce4957b6-bundle" (OuterVolumeSpecName: "bundle") pod "0c6d3a0d-2cd7-492f-bba4-c4b9ce4957b6" (UID: "0c6d3a0d-2cd7-492f-bba4-c4b9ce4957b6"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 12:17:54.145382 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:17:54.145352    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c6d3a0d-2cd7-492f-bba4-c4b9ce4957b6-kube-api-access-r8972" (OuterVolumeSpecName: "kube-api-access-r8972") pod "0c6d3a0d-2cd7-492f-bba4-c4b9ce4957b6" (UID: "0c6d3a0d-2cd7-492f-bba4-c4b9ce4957b6"). InnerVolumeSpecName "kube-api-access-r8972". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 12:17:54.147617 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:17:54.147565    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c6d3a0d-2cd7-492f-bba4-c4b9ce4957b6-util" (OuterVolumeSpecName: "util") pod "0c6d3a0d-2cd7-492f-bba4-c4b9ce4957b6" (UID: "0c6d3a0d-2cd7-492f-bba4-c4b9ce4957b6"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 12:17:54.244284 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:17:54.244248    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r8972\" (UniqueName: \"kubernetes.io/projected/0c6d3a0d-2cd7-492f-bba4-c4b9ce4957b6-kube-api-access-r8972\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
May 25 12:17:54.244284 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:17:54.244279    2549 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0c6d3a0d-2cd7-492f-bba4-c4b9ce4957b6-bundle\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
May 25 12:17:54.244284 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:17:54.244293    2549 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0c6d3a0d-2cd7-492f-bba4-c4b9ce4957b6-util\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
May 25 12:17:54.901478 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:17:54.901441    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c2gcx4" event={"ID":"0c6d3a0d-2cd7-492f-bba4-c4b9ce4957b6","Type":"ContainerDied","Data":"cb1e831bfe33c627529569a756662345420b7d6cbebab2087c911379effc23fb"}
May 25 12:17:54.901640 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:17:54.901488    2549 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cb1e831bfe33c627529569a756662345420b7d6cbebab2087c911379effc23fb"
May 25 12:17:54.901640 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:17:54.901457    2549 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c2gcx4"
May 25 12:18:06.742227 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:18:06.742181    2549 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-metrics-apiserver-7c9f485588-pwslt"]
May 25 12:18:06.742589 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:18:06.742551    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c6d3a0d-2cd7-492f-bba4-c4b9ce4957b6" containerName="util"
May 25 12:18:06.742589 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:18:06.742563    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c6d3a0d-2cd7-492f-bba4-c4b9ce4957b6" containerName="util"
May 25 12:18:06.742589 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:18:06.742571    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c6d3a0d-2cd7-492f-bba4-c4b9ce4957b6" containerName="extract"
May 25 12:18:06.742589 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:18:06.742577    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c6d3a0d-2cd7-492f-bba4-c4b9ce4957b6" containerName="extract"
May 25 12:18:06.742589 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:18:06.742591    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c6d3a0d-2cd7-492f-bba4-c4b9ce4957b6" containerName="pull"
May 25 12:18:06.742793 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:18:06.742597    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c6d3a0d-2cd7-492f-bba4-c4b9ce4957b6" containerName="pull"
May 25 12:18:06.742793 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:18:06.742649    2549 memory_manager.go:356] "RemoveStaleState removing state" podUID="0c6d3a0d-2cd7-492f-bba4-c4b9ce4957b6" containerName="extract"
May 25 12:18:06.744586 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:18:06.744569    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-7c9f485588-pwslt"
May 25 12:18:06.747800 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:18:06.747777    2549 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
May 25 12:18:06.747918 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:18:06.747789    2549 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
May 25 12:18:06.749103 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:18:06.749078    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-xj8ll\""
May 25 12:18:06.749237 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:18:06.749082    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
May 25 12:18:06.749237 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:18:06.749082    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-metrics-apiserver-certs\""
May 25 12:18:06.749424 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:18:06.749339    2549 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
May 25 12:18:06.758574 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:18:06.758549    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-7c9f485588-pwslt"]
May 25 12:18:06.857013 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:18:06.856985    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/37494496-75de-453c-9c44-4d9662395d6d-certificates\") pod \"keda-metrics-apiserver-7c9f485588-pwslt\" (UID: \"37494496-75de-453c-9c44-4d9662395d6d\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-pwslt"
May 25 12:18:06.857280 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:18:06.857019    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/37494496-75de-453c-9c44-4d9662395d6d-temp-vol\") pod \"keda-metrics-apiserver-7c9f485588-pwslt\" (UID: \"37494496-75de-453c-9c44-4d9662395d6d\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-pwslt"
May 25 12:18:06.857280 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:18:06.857057    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dvlm\" (UniqueName: \"kubernetes.io/projected/37494496-75de-453c-9c44-4d9662395d6d-kube-api-access-4dvlm\") pod \"keda-metrics-apiserver-7c9f485588-pwslt\" (UID: \"37494496-75de-453c-9c44-4d9662395d6d\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-pwslt"
May 25 12:18:06.957967 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:18:06.957935    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/37494496-75de-453c-9c44-4d9662395d6d-certificates\") pod \"keda-metrics-apiserver-7c9f485588-pwslt\" (UID: \"37494496-75de-453c-9c44-4d9662395d6d\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-pwslt"
May 25 12:18:06.958148 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:18:06.957974    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/37494496-75de-453c-9c44-4d9662395d6d-temp-vol\") pod \"keda-metrics-apiserver-7c9f485588-pwslt\" (UID: \"37494496-75de-453c-9c44-4d9662395d6d\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-pwslt"
May 25 12:18:06.958148 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:18:06.958001    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-4dvlm\" (UniqueName: \"kubernetes.io/projected/37494496-75de-453c-9c44-4d9662395d6d-kube-api-access-4dvlm\") pod \"keda-metrics-apiserver-7c9f485588-pwslt\" (UID: \"37494496-75de-453c-9c44-4d9662395d6d\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-pwslt"
May 25 12:18:06.958148 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:18:06.958098    2549 secret.go:281] references non-existent secret key: tls.crt
May 25 12:18:06.958148 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:18:06.958121    2549 projected.go:277] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: tls.crt
May 25 12:18:06.958148 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:18:06.958142    2549 projected.go:264] Couldn't get secret openshift-keda/keda-metrics-apiserver-certs: secret "keda-metrics-apiserver-certs" not found
May 25 12:18:06.958425 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:18:06.958164    2549 projected.go:194] Error preparing data for projected volume certificates for pod openshift-keda/keda-metrics-apiserver-7c9f485588-pwslt: [references non-existent secret key: tls.crt, secret "keda-metrics-apiserver-certs" not found]
May 25 12:18:06.958425 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:18:06.958254    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/37494496-75de-453c-9c44-4d9662395d6d-certificates podName:37494496-75de-453c-9c44-4d9662395d6d nodeName:}" failed. No retries permitted until 2026-05-25 12:18:07.45823419 +0000 UTC m=+291.174049461 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/37494496-75de-453c-9c44-4d9662395d6d-certificates") pod "keda-metrics-apiserver-7c9f485588-pwslt" (UID: "37494496-75de-453c-9c44-4d9662395d6d") : [references non-existent secret key: tls.crt, secret "keda-metrics-apiserver-certs" not found]
May 25 12:18:06.958425 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:18:06.958323    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/37494496-75de-453c-9c44-4d9662395d6d-temp-vol\") pod \"keda-metrics-apiserver-7c9f485588-pwslt\" (UID: \"37494496-75de-453c-9c44-4d9662395d6d\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-pwslt"
May 25 12:18:06.971783 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:18:06.971756    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dvlm\" (UniqueName: \"kubernetes.io/projected/37494496-75de-453c-9c44-4d9662395d6d-kube-api-access-4dvlm\") pod \"keda-metrics-apiserver-7c9f485588-pwslt\" (UID: \"37494496-75de-453c-9c44-4d9662395d6d\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-pwslt"
May 25 12:18:07.016651 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:18:07.016584    2549 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-admission-cf49989db-vzgb6"]
May 25 12:18:07.019143 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:18:07.019125    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-cf49989db-vzgb6"
May 25 12:18:07.021814 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:18:07.021795    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-admission-webhooks-certs\""
May 25 12:18:07.034489 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:18:07.034457    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-cf49989db-vzgb6"]
May 25 12:18:07.159641 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:18:07.159600    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hb6xg\" (UniqueName: \"kubernetes.io/projected/d9620df4-24a2-4d8c-8bd6-13826770c0a6-kube-api-access-hb6xg\") pod \"keda-admission-cf49989db-vzgb6\" (UID: \"d9620df4-24a2-4d8c-8bd6-13826770c0a6\") " pod="openshift-keda/keda-admission-cf49989db-vzgb6"
May 25 12:18:07.159798 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:18:07.159683    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/d9620df4-24a2-4d8c-8bd6-13826770c0a6-certificates\") pod \"keda-admission-cf49989db-vzgb6\" (UID: \"d9620df4-24a2-4d8c-8bd6-13826770c0a6\") " pod="openshift-keda/keda-admission-cf49989db-vzgb6"
May 25 12:18:07.260384 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:18:07.260342    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-hb6xg\" (UniqueName: \"kubernetes.io/projected/d9620df4-24a2-4d8c-8bd6-13826770c0a6-kube-api-access-hb6xg\") pod \"keda-admission-cf49989db-vzgb6\" (UID: \"d9620df4-24a2-4d8c-8bd6-13826770c0a6\") " pod="openshift-keda/keda-admission-cf49989db-vzgb6"
May 25 12:18:07.260577 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:18:07.260406    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/d9620df4-24a2-4d8c-8bd6-13826770c0a6-certificates\") pod \"keda-admission-cf49989db-vzgb6\" (UID: \"d9620df4-24a2-4d8c-8bd6-13826770c0a6\") " pod="openshift-keda/keda-admission-cf49989db-vzgb6"
May 25 12:18:07.262983 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:18:07.262944    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/d9620df4-24a2-4d8c-8bd6-13826770c0a6-certificates\") pod \"keda-admission-cf49989db-vzgb6\" (UID: \"d9620df4-24a2-4d8c-8bd6-13826770c0a6\") " pod="openshift-keda/keda-admission-cf49989db-vzgb6"
May 25 12:18:07.269591 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:18:07.269526    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hb6xg\" (UniqueName: \"kubernetes.io/projected/d9620df4-24a2-4d8c-8bd6-13826770c0a6-kube-api-access-hb6xg\") pod \"keda-admission-cf49989db-vzgb6\" (UID: \"d9620df4-24a2-4d8c-8bd6-13826770c0a6\") " pod="openshift-keda/keda-admission-cf49989db-vzgb6"
May 25 12:18:07.334178 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:18:07.334142    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-cf49989db-vzgb6"
May 25 12:18:07.461552 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:18:07.461517    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/37494496-75de-453c-9c44-4d9662395d6d-certificates\") pod \"keda-metrics-apiserver-7c9f485588-pwslt\" (UID: \"37494496-75de-453c-9c44-4d9662395d6d\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-pwslt"
May 25 12:18:07.461744 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:18:07.461672    2549 secret.go:281] references non-existent secret key: tls.crt
May 25 12:18:07.461744 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:18:07.461695    2549 projected.go:277] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: tls.crt
May 25 12:18:07.461744 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:18:07.461718    2549 projected.go:194] Error preparing data for projected volume certificates for pod openshift-keda/keda-metrics-apiserver-7c9f485588-pwslt: references non-existent secret key: tls.crt
May 25 12:18:07.461899 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:18:07.461822    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/37494496-75de-453c-9c44-4d9662395d6d-certificates podName:37494496-75de-453c-9c44-4d9662395d6d nodeName:}" failed. No retries permitted until 2026-05-25 12:18:08.461798932 +0000 UTC m=+292.177614213 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/37494496-75de-453c-9c44-4d9662395d6d-certificates") pod "keda-metrics-apiserver-7c9f485588-pwslt" (UID: "37494496-75de-453c-9c44-4d9662395d6d") : references non-existent secret key: tls.crt
May 25 12:18:07.490364 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:18:07.490340    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-cf49989db-vzgb6"]
May 25 12:18:07.492655 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:18:07.492628    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9620df4_24a2_4d8c_8bd6_13826770c0a6.slice/crio-15740d0ec65a9cb8f51d63d4cd4f6fcfd23ad382e84b9cd76e89758fe46664c6 WatchSource:0}: Error finding container 15740d0ec65a9cb8f51d63d4cd4f6fcfd23ad382e84b9cd76e89758fe46664c6: Status 404 returned error can't find the container with id 15740d0ec65a9cb8f51d63d4cd4f6fcfd23ad382e84b9cd76e89758fe46664c6
May 25 12:18:07.942574 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:18:07.942539    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-cf49989db-vzgb6" event={"ID":"d9620df4-24a2-4d8c-8bd6-13826770c0a6","Type":"ContainerStarted","Data":"15740d0ec65a9cb8f51d63d4cd4f6fcfd23ad382e84b9cd76e89758fe46664c6"}
May 25 12:18:08.469580 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:18:08.469542    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/37494496-75de-453c-9c44-4d9662395d6d-certificates\") pod \"keda-metrics-apiserver-7c9f485588-pwslt\" (UID: \"37494496-75de-453c-9c44-4d9662395d6d\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-pwslt"
May 25 12:18:08.469750 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:18:08.469684    2549 secret.go:281] references non-existent secret key: tls.crt
May 25 12:18:08.469750 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:18:08.469706    2549 projected.go:277] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: tls.crt
May 25 12:18:08.469750 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:18:08.469723    2549 projected.go:194] Error preparing data for projected volume certificates for pod openshift-keda/keda-metrics-apiserver-7c9f485588-pwslt: references non-existent secret key: tls.crt
May 25 12:18:08.469855 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:18:08.469773    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/37494496-75de-453c-9c44-4d9662395d6d-certificates podName:37494496-75de-453c-9c44-4d9662395d6d nodeName:}" failed. No retries permitted until 2026-05-25 12:18:10.469759414 +0000 UTC m=+294.185574678 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/37494496-75de-453c-9c44-4d9662395d6d-certificates") pod "keda-metrics-apiserver-7c9f485588-pwslt" (UID: "37494496-75de-453c-9c44-4d9662395d6d") : references non-existent secret key: tls.crt
May 25 12:18:09.950833 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:18:09.950791    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-cf49989db-vzgb6" event={"ID":"d9620df4-24a2-4d8c-8bd6-13826770c0a6","Type":"ContainerStarted","Data":"44c844b5e49a0ef3e5f517b4522e9e42ee312673cd244b817c659a6548ba5bb5"}
May 25 12:18:09.951289 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:18:09.950933    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-admission-cf49989db-vzgb6"
May 25 12:18:09.968409 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:18:09.968358    2549 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-keda/keda-admission-cf49989db-vzgb6" podStartSLOduration=1.291095651 podStartE2EDuration="2.96834442s" podCreationTimestamp="2026-05-25 12:18:07 +0000 UTC" firstStartedPulling="2026-05-25 12:18:07.4940397 +0000 UTC m=+291.209854978" lastFinishedPulling="2026-05-25 12:18:09.171288467 +0000 UTC m=+292.887103747" observedRunningTime="2026-05-25 12:18:09.966222079 +0000 UTC m=+293.682037359" watchObservedRunningTime="2026-05-25 12:18:09.96834442 +0000 UTC m=+293.684159720"
May 25 12:18:10.488428 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:18:10.488396    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/37494496-75de-453c-9c44-4d9662395d6d-certificates\") pod \"keda-metrics-apiserver-7c9f485588-pwslt\" (UID: \"37494496-75de-453c-9c44-4d9662395d6d\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-pwslt"
May 25 12:18:10.490846 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:18:10.490827    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/37494496-75de-453c-9c44-4d9662395d6d-certificates\") pod \"keda-metrics-apiserver-7c9f485588-pwslt\" (UID: \"37494496-75de-453c-9c44-4d9662395d6d\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-pwslt"
May 25 12:18:10.657434 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:18:10.657396    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-7c9f485588-pwslt"
May 25 12:18:10.785179 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:18:10.785152    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-7c9f485588-pwslt"]
May 25 12:18:10.787480 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:18:10.787447    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37494496_75de_453c_9c44_4d9662395d6d.slice/crio-01625b7dc71edb789fcf9367453c6d5c055124be8fcccb25917817e552d7aa45 WatchSource:0}: Error finding container 01625b7dc71edb789fcf9367453c6d5c055124be8fcccb25917817e552d7aa45: Status 404 returned error can't find the container with id 01625b7dc71edb789fcf9367453c6d5c055124be8fcccb25917817e552d7aa45
May 25 12:18:10.955367 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:18:10.955329    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-7c9f485588-pwslt" event={"ID":"37494496-75de-453c-9c44-4d9662395d6d","Type":"ContainerStarted","Data":"01625b7dc71edb789fcf9367453c6d5c055124be8fcccb25917817e552d7aa45"}
May 25 12:18:13.968315 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:18:13.968274    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-7c9f485588-pwslt" event={"ID":"37494496-75de-453c-9c44-4d9662395d6d","Type":"ContainerStarted","Data":"d29bd4462274929b86225c685b136f98a88525b13698388d6f875a65a0a6c3eb"}
May 25 12:18:13.968677 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:18:13.968346    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-metrics-apiserver-7c9f485588-pwslt"
May 25 12:18:13.989403 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:18:13.989354    2549 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-keda/keda-metrics-apiserver-7c9f485588-pwslt" podStartSLOduration=5.647504041 podStartE2EDuration="7.989342208s" podCreationTimestamp="2026-05-25 12:18:06 +0000 UTC" firstStartedPulling="2026-05-25 12:18:10.78881372 +0000 UTC m=+294.504628986" lastFinishedPulling="2026-05-25 12:18:13.130651887 +0000 UTC m=+296.846467153" observedRunningTime="2026-05-25 12:18:13.987351602 +0000 UTC m=+297.703166889" watchObservedRunningTime="2026-05-25 12:18:13.989342208 +0000 UTC m=+297.705157494"
May 25 12:18:16.752798 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:18:16.752773    2549 kubelet.go:1628] "Image garbage collection succeeded"
May 25 12:18:24.976490 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:18:24.976461    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-metrics-apiserver-7c9f485588-pwslt"
May 25 12:18:30.958168 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:18:30.958139    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-admission-cf49989db-vzgb6"
May 25 12:19:15.895163 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:19:15.895067    2549 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-6768985688-fg8bv"]
May 25 12:19:15.897516 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:19:15.897495    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-6768985688-fg8bv"
May 25 12:19:15.904234 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:19:15.904191    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\""
May 25 12:19:15.905572 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:19:15.905551    2549 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
May 25 12:19:15.905634 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:19:15.905617    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-q6zcp\""
May 25 12:19:15.905682 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:19:15.905559    2549 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
May 25 12:19:15.911730 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:19:15.911701    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-6768985688-fg8bv"]
May 25 12:19:15.940120 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:19:15.940072    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kr4hc\" (UniqueName: \"kubernetes.io/projected/254b5a52-d031-44c6-819e-abc99e256a34-kube-api-access-kr4hc\") pod \"kserve-controller-manager-6768985688-fg8bv\" (UID: \"254b5a52-d031-44c6-819e-abc99e256a34\") " pod="kserve/kserve-controller-manager-6768985688-fg8bv"
May 25 12:19:15.940308 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:19:15.940167    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/254b5a52-d031-44c6-819e-abc99e256a34-cert\") pod \"kserve-controller-manager-6768985688-fg8bv\" (UID: \"254b5a52-d031-44c6-819e-abc99e256a34\") " pod="kserve/kserve-controller-manager-6768985688-fg8bv"
May 25 12:19:16.040945 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:19:16.040911    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-kr4hc\" (UniqueName: \"kubernetes.io/projected/254b5a52-d031-44c6-819e-abc99e256a34-kube-api-access-kr4hc\") pod \"kserve-controller-manager-6768985688-fg8bv\" (UID: \"254b5a52-d031-44c6-819e-abc99e256a34\") " pod="kserve/kserve-controller-manager-6768985688-fg8bv"
May 25 12:19:16.041138 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:19:16.040954    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/254b5a52-d031-44c6-819e-abc99e256a34-cert\") pod \"kserve-controller-manager-6768985688-fg8bv\" (UID: \"254b5a52-d031-44c6-819e-abc99e256a34\") " pod="kserve/kserve-controller-manager-6768985688-fg8bv"
May 25 12:19:16.048432 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:19:16.044543    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/254b5a52-d031-44c6-819e-abc99e256a34-cert\") pod \"kserve-controller-manager-6768985688-fg8bv\" (UID: \"254b5a52-d031-44c6-819e-abc99e256a34\") " pod="kserve/kserve-controller-manager-6768985688-fg8bv"
May 25 12:19:16.055866 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:19:16.055839    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kr4hc\" (UniqueName: \"kubernetes.io/projected/254b5a52-d031-44c6-819e-abc99e256a34-kube-api-access-kr4hc\") pod \"kserve-controller-manager-6768985688-fg8bv\" (UID: \"254b5a52-d031-44c6-819e-abc99e256a34\") " pod="kserve/kserve-controller-manager-6768985688-fg8bv"
May 25 12:19:16.210004 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:19:16.209913    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-6768985688-fg8bv"
May 25 12:19:16.338151 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:19:16.338121    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-6768985688-fg8bv"]
May 25 12:19:16.340684 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:19:16.340655    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod254b5a52_d031_44c6_819e_abc99e256a34.slice/crio-d22a52365d7700d76ac55b9a518b55ca154742c2205098be9d3e4002b1ff3f80 WatchSource:0}: Error finding container d22a52365d7700d76ac55b9a518b55ca154742c2205098be9d3e4002b1ff3f80: Status 404 returned error can't find the container with id d22a52365d7700d76ac55b9a518b55ca154742c2205098be9d3e4002b1ff3f80
May 25 12:19:16.341942 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:19:16.341927    2549 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 25 12:19:17.204642 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:19:17.204600    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-6768985688-fg8bv" event={"ID":"254b5a52-d031-44c6-819e-abc99e256a34","Type":"ContainerStarted","Data":"d22a52365d7700d76ac55b9a518b55ca154742c2205098be9d3e4002b1ff3f80"}
May 25 12:19:19.221601 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:19:19.221564    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-6768985688-fg8bv" event={"ID":"254b5a52-d031-44c6-819e-abc99e256a34","Type":"ContainerStarted","Data":"466662fdfd34d2ac06cd2da729b146609030bda8583d76dfa0ffeac0dd19d52f"}
May 25 12:19:19.221952 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:19:19.221640    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-6768985688-fg8bv"
May 25 12:19:19.240116 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:19:19.240069    2549 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve/kserve-controller-manager-6768985688-fg8bv" podStartSLOduration=1.553411181 podStartE2EDuration="4.240055721s" podCreationTimestamp="2026-05-25 12:19:15 +0000 UTC" firstStartedPulling="2026-05-25 12:19:16.342043448 +0000 UTC m=+360.057858714" lastFinishedPulling="2026-05-25 12:19:19.028687971 +0000 UTC m=+362.744503254" observedRunningTime="2026-05-25 12:19:19.239160718 +0000 UTC m=+362.954976006" watchObservedRunningTime="2026-05-25 12:19:19.240055721 +0000 UTC m=+362.955871007"
May 25 12:19:50.229704 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:19:50.229676    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-6768985688-fg8bv"
May 25 12:19:51.346775 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:19:51.346741    2549 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-6768985688-fg8bv"]
May 25 12:19:51.347228 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:19:51.346951    2549 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve/kserve-controller-manager-6768985688-fg8bv" podUID="254b5a52-d031-44c6-819e-abc99e256a34" containerName="manager" containerID="cri-o://466662fdfd34d2ac06cd2da729b146609030bda8583d76dfa0ffeac0dd19d52f" gracePeriod=10
May 25 12:19:51.377271 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:19:51.377240    2549 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-6768985688-z8hzg"]
May 25 12:19:51.379430 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:19:51.379415    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-6768985688-z8hzg"
May 25 12:19:51.389986 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:19:51.389962    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-6768985688-z8hzg"]
May 25 12:19:51.552774 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:19:51.552742    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44zzz\" (UniqueName: \"kubernetes.io/projected/d1e2aa01-8b67-4aeb-b948-a73028a69d04-kube-api-access-44zzz\") pod \"kserve-controller-manager-6768985688-z8hzg\" (UID: \"d1e2aa01-8b67-4aeb-b948-a73028a69d04\") " pod="kserve/kserve-controller-manager-6768985688-z8hzg"
May 25 12:19:51.552916 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:19:51.552786    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d1e2aa01-8b67-4aeb-b948-a73028a69d04-cert\") pod \"kserve-controller-manager-6768985688-z8hzg\" (UID: \"d1e2aa01-8b67-4aeb-b948-a73028a69d04\") " pod="kserve/kserve-controller-manager-6768985688-z8hzg"
May 25 12:19:51.593687 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:19:51.593668    2549 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-6768985688-fg8bv"
May 25 12:19:51.654046 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:19:51.653972    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-44zzz\" (UniqueName: \"kubernetes.io/projected/d1e2aa01-8b67-4aeb-b948-a73028a69d04-kube-api-access-44zzz\") pod \"kserve-controller-manager-6768985688-z8hzg\" (UID: \"d1e2aa01-8b67-4aeb-b948-a73028a69d04\") " pod="kserve/kserve-controller-manager-6768985688-z8hzg"
May 25 12:19:51.654046 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:19:51.654012    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d1e2aa01-8b67-4aeb-b948-a73028a69d04-cert\") pod \"kserve-controller-manager-6768985688-z8hzg\" (UID: \"d1e2aa01-8b67-4aeb-b948-a73028a69d04\") " pod="kserve/kserve-controller-manager-6768985688-z8hzg"
May 25 12:19:51.656334 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:19:51.656312    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d1e2aa01-8b67-4aeb-b948-a73028a69d04-cert\") pod \"kserve-controller-manager-6768985688-z8hzg\" (UID: \"d1e2aa01-8b67-4aeb-b948-a73028a69d04\") " pod="kserve/kserve-controller-manager-6768985688-z8hzg"
May 25 12:19:51.664490 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:19:51.664467    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-44zzz\" (UniqueName: \"kubernetes.io/projected/d1e2aa01-8b67-4aeb-b948-a73028a69d04-kube-api-access-44zzz\") pod \"kserve-controller-manager-6768985688-z8hzg\" (UID: \"d1e2aa01-8b67-4aeb-b948-a73028a69d04\") " pod="kserve/kserve-controller-manager-6768985688-z8hzg"
May 25 12:19:51.722281 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:19:51.722230    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-6768985688-z8hzg"
May 25 12:19:51.755211 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:19:51.755167    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kr4hc\" (UniqueName: \"kubernetes.io/projected/254b5a52-d031-44c6-819e-abc99e256a34-kube-api-access-kr4hc\") pod \"254b5a52-d031-44c6-819e-abc99e256a34\" (UID: \"254b5a52-d031-44c6-819e-abc99e256a34\") "
May 25 12:19:51.755375 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:19:51.755331    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/254b5a52-d031-44c6-819e-abc99e256a34-cert\") pod \"254b5a52-d031-44c6-819e-abc99e256a34\" (UID: \"254b5a52-d031-44c6-819e-abc99e256a34\") "
May 25 12:19:51.757253 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:19:51.757228    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/254b5a52-d031-44c6-819e-abc99e256a34-kube-api-access-kr4hc" (OuterVolumeSpecName: "kube-api-access-kr4hc") pod "254b5a52-d031-44c6-819e-abc99e256a34" (UID: "254b5a52-d031-44c6-819e-abc99e256a34"). InnerVolumeSpecName "kube-api-access-kr4hc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 12:19:51.757361 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:19:51.757305    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/254b5a52-d031-44c6-819e-abc99e256a34-cert" (OuterVolumeSpecName: "cert") pod "254b5a52-d031-44c6-819e-abc99e256a34" (UID: "254b5a52-d031-44c6-819e-abc99e256a34"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 12:19:51.855917 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:19:51.855874    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-6768985688-z8hzg"]
May 25 12:19:51.856004 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:19:51.855976    2549 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/254b5a52-d031-44c6-819e-abc99e256a34-cert\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
May 25 12:19:51.856004 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:19:51.855996    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kr4hc\" (UniqueName: \"kubernetes.io/projected/254b5a52-d031-44c6-819e-abc99e256a34-kube-api-access-kr4hc\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
May 25 12:19:51.858307 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:19:51.858285    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1e2aa01_8b67_4aeb_b948_a73028a69d04.slice/crio-219ba9ac454e58037cafd8d45ee2c1554cc067ed7fff118c14167ab83fffc4d9 WatchSource:0}: Error finding container 219ba9ac454e58037cafd8d45ee2c1554cc067ed7fff118c14167ab83fffc4d9: Status 404 returned error can't find the container with id 219ba9ac454e58037cafd8d45ee2c1554cc067ed7fff118c14167ab83fffc4d9
May 25 12:19:52.335587 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:19:52.335550    2549 generic.go:358] "Generic (PLEG): container finished" podID="254b5a52-d031-44c6-819e-abc99e256a34" containerID="466662fdfd34d2ac06cd2da729b146609030bda8583d76dfa0ffeac0dd19d52f" exitCode=0
May 25 12:19:52.335736 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:19:52.335622    2549 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-6768985688-fg8bv"
May 25 12:19:52.335736 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:19:52.335619    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-6768985688-fg8bv" event={"ID":"254b5a52-d031-44c6-819e-abc99e256a34","Type":"ContainerDied","Data":"466662fdfd34d2ac06cd2da729b146609030bda8583d76dfa0ffeac0dd19d52f"}
May 25 12:19:52.335736 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:19:52.335727    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-6768985688-fg8bv" event={"ID":"254b5a52-d031-44c6-819e-abc99e256a34","Type":"ContainerDied","Data":"d22a52365d7700d76ac55b9a518b55ca154742c2205098be9d3e4002b1ff3f80"}
May 25 12:19:52.335883 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:19:52.335747    2549 scope.go:117] "RemoveContainer" containerID="466662fdfd34d2ac06cd2da729b146609030bda8583d76dfa0ffeac0dd19d52f"
May 25 12:19:52.337032 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:19:52.336995    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-6768985688-z8hzg" event={"ID":"d1e2aa01-8b67-4aeb-b948-a73028a69d04","Type":"ContainerStarted","Data":"47378afa630fd11c5239e22612188d62dc78f1c367c909caa282f0a2767231d2"}
May 25 12:19:52.337032 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:19:52.337031    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-6768985688-z8hzg" event={"ID":"d1e2aa01-8b67-4aeb-b948-a73028a69d04","Type":"ContainerStarted","Data":"219ba9ac454e58037cafd8d45ee2c1554cc067ed7fff118c14167ab83fffc4d9"}
May 25 12:19:52.337190 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:19:52.337082    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-6768985688-z8hzg"
May 25 12:19:52.343783 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:19:52.343767    2549 scope.go:117] "RemoveContainer" containerID="466662fdfd34d2ac06cd2da729b146609030bda8583d76dfa0ffeac0dd19d52f"
May 25 12:19:52.344034 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:19:52.344011    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"466662fdfd34d2ac06cd2da729b146609030bda8583d76dfa0ffeac0dd19d52f\": container with ID starting with 466662fdfd34d2ac06cd2da729b146609030bda8583d76dfa0ffeac0dd19d52f not found: ID does not exist" containerID="466662fdfd34d2ac06cd2da729b146609030bda8583d76dfa0ffeac0dd19d52f"
May 25 12:19:52.344085 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:19:52.344041    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"466662fdfd34d2ac06cd2da729b146609030bda8583d76dfa0ffeac0dd19d52f"} err="failed to get container status \"466662fdfd34d2ac06cd2da729b146609030bda8583d76dfa0ffeac0dd19d52f\": rpc error: code = NotFound desc = could not find container \"466662fdfd34d2ac06cd2da729b146609030bda8583d76dfa0ffeac0dd19d52f\": container with ID starting with 466662fdfd34d2ac06cd2da729b146609030bda8583d76dfa0ffeac0dd19d52f not found: ID does not exist"
May 25 12:19:52.379230 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:19:52.379169    2549 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve/kserve-controller-manager-6768985688-z8hzg" podStartSLOduration=1.087224746 podStartE2EDuration="1.379157086s" podCreationTimestamp="2026-05-25 12:19:51 +0000 UTC" firstStartedPulling="2026-05-25 12:19:51.859488564 +0000 UTC m=+395.575303833" lastFinishedPulling="2026-05-25 12:19:52.151420908 +0000 UTC m=+395.867236173" observedRunningTime="2026-05-25 12:19:52.361882742 +0000 UTC m=+396.077698030" watchObservedRunningTime="2026-05-25 12:19:52.379157086 +0000 UTC m=+396.094972373"
May 25 12:19:52.380126 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:19:52.380111    2549 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-6768985688-fg8bv"]
May 25 12:19:52.386794 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:19:52.386770    2549 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve/kserve-controller-manager-6768985688-fg8bv"]
May 25 12:19:52.848518 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:19:52.848483    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="254b5a52-d031-44c6-819e-abc99e256a34" path="/var/lib/kubelet/pods/254b5a52-d031-44c6-819e-abc99e256a34/volumes"
May 25 12:20:23.347108 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:20:23.347079    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-6768985688-z8hzg"
May 25 12:20:24.315919 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:20:24.315886    2549 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-86f7b4b499-m6k77"]
May 25 12:20:24.316269 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:20:24.316256    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="254b5a52-d031-44c6-819e-abc99e256a34" containerName="manager"
May 25 12:20:24.316269 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:20:24.316270    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="254b5a52-d031-44c6-819e-abc99e256a34" containerName="manager"
May 25 12:20:24.316350 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:20:24.316338    2549 memory_manager.go:356] "RemoveStaleState removing state" podUID="254b5a52-d031-44c6-819e-abc99e256a34" containerName="manager"
May 25 12:20:24.319057 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:20:24.319037    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-86f7b4b499-m6k77"
May 25 12:20:24.325451 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:20:24.325430    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-fh44s\""
May 25 12:20:24.325555 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:20:24.325491    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
May 25 12:20:24.333366 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:20:24.333344    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-86f7b4b499-m6k77"]
May 25 12:20:24.404664 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:20:24.404632    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vslqm\" (UniqueName: \"kubernetes.io/projected/a314e08c-f304-41f9-896b-852c9b8ca7ac-kube-api-access-vslqm\") pod \"model-serving-api-86f7b4b499-m6k77\" (UID: \"a314e08c-f304-41f9-896b-852c9b8ca7ac\") " pod="kserve/model-serving-api-86f7b4b499-m6k77"
May 25 12:20:24.405064 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:20:24.404674    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a314e08c-f304-41f9-896b-852c9b8ca7ac-tls-certs\") pod \"model-serving-api-86f7b4b499-m6k77\" (UID: \"a314e08c-f304-41f9-896b-852c9b8ca7ac\") " pod="kserve/model-serving-api-86f7b4b499-m6k77"
May 25 12:20:24.505836 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:20:24.505799    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-vslqm\" (UniqueName: \"kubernetes.io/projected/a314e08c-f304-41f9-896b-852c9b8ca7ac-kube-api-access-vslqm\") pod \"model-serving-api-86f7b4b499-m6k77\" (UID: \"a314e08c-f304-41f9-896b-852c9b8ca7ac\") " pod="kserve/model-serving-api-86f7b4b499-m6k77"
May 25 12:20:24.505836 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:20:24.505840    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a314e08c-f304-41f9-896b-852c9b8ca7ac-tls-certs\") pod \"model-serving-api-86f7b4b499-m6k77\" (UID: \"a314e08c-f304-41f9-896b-852c9b8ca7ac\") " pod="kserve/model-serving-api-86f7b4b499-m6k77"
May 25 12:20:24.508291 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:20:24.508264    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a314e08c-f304-41f9-896b-852c9b8ca7ac-tls-certs\") pod \"model-serving-api-86f7b4b499-m6k77\" (UID: \"a314e08c-f304-41f9-896b-852c9b8ca7ac\") " pod="kserve/model-serving-api-86f7b4b499-m6k77"
May 25 12:20:24.518251 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:20:24.518225    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vslqm\" (UniqueName: \"kubernetes.io/projected/a314e08c-f304-41f9-896b-852c9b8ca7ac-kube-api-access-vslqm\") pod \"model-serving-api-86f7b4b499-m6k77\" (UID: \"a314e08c-f304-41f9-896b-852c9b8ca7ac\") " pod="kserve/model-serving-api-86f7b4b499-m6k77"
May 25 12:20:24.629510 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:20:24.629477    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-86f7b4b499-m6k77"
May 25 12:20:24.768919 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:20:24.768895    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-86f7b4b499-m6k77"]
May 25 12:20:24.770910 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:20:24.770882    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda314e08c_f304_41f9_896b_852c9b8ca7ac.slice/crio-5011fbd7ab4386cc51a6c8529ce1647708fa0758bfdf889789a4331135dc9251 WatchSource:0}: Error finding container 5011fbd7ab4386cc51a6c8529ce1647708fa0758bfdf889789a4331135dc9251: Status 404 returned error can't find the container with id 5011fbd7ab4386cc51a6c8529ce1647708fa0758bfdf889789a4331135dc9251
May 25 12:20:25.458288 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:20:25.458240    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-86f7b4b499-m6k77" event={"ID":"a314e08c-f304-41f9-896b-852c9b8ca7ac","Type":"ContainerStarted","Data":"5011fbd7ab4386cc51a6c8529ce1647708fa0758bfdf889789a4331135dc9251"}
May 25 12:20:27.468190 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:20:27.468152    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-86f7b4b499-m6k77" event={"ID":"a314e08c-f304-41f9-896b-852c9b8ca7ac","Type":"ContainerStarted","Data":"3392cbc0727303d4612eac73682fe3787dfd2a99d151e6f28b8dfd979af0bf86"}
May 25 12:20:27.468591 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:20:27.468236    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-86f7b4b499-m6k77"
May 25 12:20:27.491174 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:20:27.491123    2549 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve/model-serving-api-86f7b4b499-m6k77" podStartSLOduration=1.261484227 podStartE2EDuration="3.491111705s" podCreationTimestamp="2026-05-25 12:20:24 +0000 UTC" firstStartedPulling="2026-05-25 12:20:24.772778444 +0000 UTC m=+428.488593709" lastFinishedPulling="2026-05-25 12:20:27.002405921 +0000 UTC m=+430.718221187" observedRunningTime="2026-05-25 12:20:27.49009352 +0000 UTC m=+431.205908806" watchObservedRunningTime="2026-05-25 12:20:27.491111705 +0000 UTC m=+431.206926992"
May 25 12:20:38.478507 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:20:38.478430    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-86f7b4b499-m6k77"
May 25 12:20:40.439243 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:20:40.439207    2549 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console/console-7c46bb4bf6-cpj2j"]
May 25 12:20:40.441518 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:20:40.441497    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c46bb4bf6-cpj2j"
May 25 12:20:40.451836 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:20:40.451811    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7c46bb4bf6-cpj2j"]
May 25 12:20:40.540387 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:20:40.540338    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/62d21dab-d5f6-47e9-85b8-f5e8cba9333e-service-ca\") pod \"console-7c46bb4bf6-cpj2j\" (UID: \"62d21dab-d5f6-47e9-85b8-f5e8cba9333e\") " pod="openshift-console/console-7c46bb4bf6-cpj2j"
May 25 12:20:40.540387 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:20:40.540389    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/62d21dab-d5f6-47e9-85b8-f5e8cba9333e-trusted-ca-bundle\") pod \"console-7c46bb4bf6-cpj2j\" (UID: \"62d21dab-d5f6-47e9-85b8-f5e8cba9333e\") " pod="openshift-console/console-7c46bb4bf6-cpj2j"
May 25 12:20:40.540636 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:20:40.540457    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/62d21dab-d5f6-47e9-85b8-f5e8cba9333e-console-oauth-config\") pod \"console-7c46bb4bf6-cpj2j\" (UID: \"62d21dab-d5f6-47e9-85b8-f5e8cba9333e\") " pod="openshift-console/console-7c46bb4bf6-cpj2j"
May 25 12:20:40.540636 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:20:40.540565    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55vh4\" (UniqueName: \"kubernetes.io/projected/62d21dab-d5f6-47e9-85b8-f5e8cba9333e-kube-api-access-55vh4\") pod \"console-7c46bb4bf6-cpj2j\" (UID: \"62d21dab-d5f6-47e9-85b8-f5e8cba9333e\") " pod="openshift-console/console-7c46bb4bf6-cpj2j"
May 25 12:20:40.540636 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:20:40.540617    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/62d21dab-d5f6-47e9-85b8-f5e8cba9333e-oauth-serving-cert\") pod \"console-7c46bb4bf6-cpj2j\" (UID: \"62d21dab-d5f6-47e9-85b8-f5e8cba9333e\") " pod="openshift-console/console-7c46bb4bf6-cpj2j"
May 25 12:20:40.540786 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:20:40.540699    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/62d21dab-d5f6-47e9-85b8-f5e8cba9333e-console-config\") pod \"console-7c46bb4bf6-cpj2j\" (UID: \"62d21dab-d5f6-47e9-85b8-f5e8cba9333e\") " pod="openshift-console/console-7c46bb4bf6-cpj2j"
May 25 12:20:40.540786 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:20:40.540732    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/62d21dab-d5f6-47e9-85b8-f5e8cba9333e-console-serving-cert\") pod \"console-7c46bb4bf6-cpj2j\" (UID: \"62d21dab-d5f6-47e9-85b8-f5e8cba9333e\") " pod="openshift-console/console-7c46bb4bf6-cpj2j"
May 25 12:20:40.641330 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:20:40.641289    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/62d21dab-d5f6-47e9-85b8-f5e8cba9333e-service-ca\") pod \"console-7c46bb4bf6-cpj2j\" (UID: \"62d21dab-d5f6-47e9-85b8-f5e8cba9333e\") " pod="openshift-console/console-7c46bb4bf6-cpj2j"
May 25 12:20:40.641330 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:20:40.641330    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/62d21dab-d5f6-47e9-85b8-f5e8cba9333e-trusted-ca-bundle\") pod \"console-7c46bb4bf6-cpj2j\" (UID: \"62d21dab-d5f6-47e9-85b8-f5e8cba9333e\") " pod="openshift-console/console-7c46bb4bf6-cpj2j"
May 25 12:20:40.641568 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:20:40.641351    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/62d21dab-d5f6-47e9-85b8-f5e8cba9333e-console-oauth-config\") pod \"console-7c46bb4bf6-cpj2j\" (UID: \"62d21dab-d5f6-47e9-85b8-f5e8cba9333e\") " pod="openshift-console/console-7c46bb4bf6-cpj2j"
May 25 12:20:40.641568 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:20:40.641370    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-55vh4\" (UniqueName: \"kubernetes.io/projected/62d21dab-d5f6-47e9-85b8-f5e8cba9333e-kube-api-access-55vh4\") pod \"console-7c46bb4bf6-cpj2j\" (UID: \"62d21dab-d5f6-47e9-85b8-f5e8cba9333e\") " pod="openshift-console/console-7c46bb4bf6-cpj2j"
May 25 12:20:40.641568 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:20:40.641418    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/62d21dab-d5f6-47e9-85b8-f5e8cba9333e-oauth-serving-cert\") pod \"console-7c46bb4bf6-cpj2j\" (UID: \"62d21dab-d5f6-47e9-85b8-f5e8cba9333e\") " pod="openshift-console/console-7c46bb4bf6-cpj2j"
May 25 12:20:40.641568 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:20:40.641479    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/62d21dab-d5f6-47e9-85b8-f5e8cba9333e-console-config\") pod \"console-7c46bb4bf6-cpj2j\" (UID: \"62d21dab-d5f6-47e9-85b8-f5e8cba9333e\") " pod="openshift-console/console-7c46bb4bf6-cpj2j"
May 25 12:20:40.641568 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:20:40.641506    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/62d21dab-d5f6-47e9-85b8-f5e8cba9333e-console-serving-cert\") pod \"console-7c46bb4bf6-cpj2j\" (UID: \"62d21dab-d5f6-47e9-85b8-f5e8cba9333e\") " pod="openshift-console/console-7c46bb4bf6-cpj2j"
May 25 12:20:40.642236 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:20:40.642188    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/62d21dab-d5f6-47e9-85b8-f5e8cba9333e-oauth-serving-cert\") pod \"console-7c46bb4bf6-cpj2j\" (UID: \"62d21dab-d5f6-47e9-85b8-f5e8cba9333e\") " pod="openshift-console/console-7c46bb4bf6-cpj2j"
May 25 12:20:40.642352 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:20:40.642243    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/62d21dab-d5f6-47e9-85b8-f5e8cba9333e-service-ca\") pod \"console-7c46bb4bf6-cpj2j\" (UID: \"62d21dab-d5f6-47e9-85b8-f5e8cba9333e\") " pod="openshift-console/console-7c46bb4bf6-cpj2j"
May 25 12:20:40.642352 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:20:40.642295    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/62d21dab-d5f6-47e9-85b8-f5e8cba9333e-console-config\") pod \"console-7c46bb4bf6-cpj2j\" (UID: \"62d21dab-d5f6-47e9-85b8-f5e8cba9333e\") " pod="openshift-console/console-7c46bb4bf6-cpj2j"
May 25 12:20:40.642352 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:20:40.642298    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/62d21dab-d5f6-47e9-85b8-f5e8cba9333e-trusted-ca-bundle\") pod \"console-7c46bb4bf6-cpj2j\" (UID: \"62d21dab-d5f6-47e9-85b8-f5e8cba9333e\") " pod="openshift-console/console-7c46bb4bf6-cpj2j"
May 25 12:20:40.644467 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:20:40.644442    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/62d21dab-d5f6-47e9-85b8-f5e8cba9333e-console-oauth-config\") pod \"console-7c46bb4bf6-cpj2j\" (UID: \"62d21dab-d5f6-47e9-85b8-f5e8cba9333e\") " pod="openshift-console/console-7c46bb4bf6-cpj2j"
May 25 12:20:40.644562 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:20:40.644449    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/62d21dab-d5f6-47e9-85b8-f5e8cba9333e-console-serving-cert\") pod \"console-7c46bb4bf6-cpj2j\" (UID: \"62d21dab-d5f6-47e9-85b8-f5e8cba9333e\") " pod="openshift-console/console-7c46bb4bf6-cpj2j"
May 25 12:20:40.653894 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:20:40.653870    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-55vh4\" (UniqueName: \"kubernetes.io/projected/62d21dab-d5f6-47e9-85b8-f5e8cba9333e-kube-api-access-55vh4\") pod \"console-7c46bb4bf6-cpj2j\" (UID: \"62d21dab-d5f6-47e9-85b8-f5e8cba9333e\") " pod="openshift-console/console-7c46bb4bf6-cpj2j"
May 25 12:20:40.752303 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:20:40.752217    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c46bb4bf6-cpj2j"
May 25 12:20:40.891117 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:20:40.891092    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7c46bb4bf6-cpj2j"]
May 25 12:20:40.892827 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:20:40.892802    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62d21dab_d5f6_47e9_85b8_f5e8cba9333e.slice/crio-0cdd99c8dbf2557ad5cb5e6ac7180fb8e800c5161f02dcd34770fa4a2d299cd9 WatchSource:0}: Error finding container 0cdd99c8dbf2557ad5cb5e6ac7180fb8e800c5161f02dcd34770fa4a2d299cd9: Status 404 returned error can't find the container with id 0cdd99c8dbf2557ad5cb5e6ac7180fb8e800c5161f02dcd34770fa4a2d299cd9
May 25 12:20:41.518260 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:20:41.518225    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c46bb4bf6-cpj2j" event={"ID":"62d21dab-d5f6-47e9-85b8-f5e8cba9333e","Type":"ContainerStarted","Data":"d7e0db8b01551825e9f728ae7755c6d56c2784a7130400b71cf145014587a2f0"}
May 25 12:20:41.518260 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:20:41.518264    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c46bb4bf6-cpj2j" event={"ID":"62d21dab-d5f6-47e9-85b8-f5e8cba9333e","Type":"ContainerStarted","Data":"0cdd99c8dbf2557ad5cb5e6ac7180fb8e800c5161f02dcd34770fa4a2d299cd9"}
May 25 12:20:41.540916 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:20:41.540864    2549 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console/console-7c46bb4bf6-cpj2j" podStartSLOduration=1.5408473470000001 podStartE2EDuration="1.540847347s" podCreationTimestamp="2026-05-25 12:20:40 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 12:20:41.538316184 +0000 UTC m=+445.254131483" watchObservedRunningTime="2026-05-25 12:20:41.540847347 +0000 UTC m=+445.256662633"
May 25 12:20:50.753010 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:20:50.752969    2549 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7c46bb4bf6-cpj2j"
May 25 12:20:50.753010 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:20:50.753019    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7c46bb4bf6-cpj2j"
May 25 12:20:50.757876 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:20:50.757850    2549 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7c46bb4bf6-cpj2j"
May 25 12:20:51.557885 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:20:51.557855    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7c46bb4bf6-cpj2j"
May 25 12:20:51.634053 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:20:51.634014    2549 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-65fff857b4-rbmpj"]
May 25 12:21:01.065646 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:21:01.065608    2549 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-1742e-predictor-5bf4dfd9dd-8j4zf"]
May 25 12:21:01.073303 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:21:01.073278    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-1742e-predictor-5bf4dfd9dd-8j4zf"
May 25 12:21:01.076093 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:21:01.076065    2549 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-1742e-kube-rbac-proxy-sar-config\""
May 25 12:21:01.076276 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:21:01.076255    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-v7cv4\""
May 25 12:21:01.076441 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:21:01.076180    2549 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
May 25 12:21:01.076571 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:21:01.076259    2549 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
May 25 12:21:01.076666 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:21:01.076165    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-1742e-predictor-serving-cert\""
May 25 12:21:01.080893 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:21:01.080870    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-1742e-predictor-5bf4dfd9dd-8j4zf"]
May 25 12:21:01.111254 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:21:01.111216    2549 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-1742e-predictor-5764c6c5cf-6wk56"]
May 25 12:21:01.114759 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:21:01.114738    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-1742e-predictor-5764c6c5cf-6wk56"
May 25 12:21:01.117881 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:21:01.117848    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-1742e-predictor-serving-cert\""
May 25 12:21:01.117987 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:21:01.117862    2549 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-1742e-kube-rbac-proxy-sar-config\""
May 25 12:21:01.125353 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:21:01.125326    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-1742e-predictor-5764c6c5cf-6wk56"]
May 25 12:21:01.131710 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:21:01.131686    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dbeb1945-c642-49c8-ab81-19aeffc568d3-proxy-tls\") pod \"error-404-isvc-1742e-predictor-5764c6c5cf-6wk56\" (UID: \"dbeb1945-c642-49c8-ab81-19aeffc568d3\") " pod="kserve-ci-e2e-test/error-404-isvc-1742e-predictor-5764c6c5cf-6wk56"
May 25 12:21:01.131846 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:21:01.131727    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9krzc\" (UniqueName: \"kubernetes.io/projected/d1e41943-5b0a-4496-ba74-97cc2c381802-kube-api-access-9krzc\") pod \"success-200-isvc-1742e-predictor-5bf4dfd9dd-8j4zf\" (UID: \"d1e41943-5b0a-4496-ba74-97cc2c381802\") " pod="kserve-ci-e2e-test/success-200-isvc-1742e-predictor-5bf4dfd9dd-8j4zf"
May 25 12:21:01.131846 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:21:01.131785    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d1e41943-5b0a-4496-ba74-97cc2c381802-proxy-tls\") pod \"success-200-isvc-1742e-predictor-5bf4dfd9dd-8j4zf\" (UID: \"d1e41943-5b0a-4496-ba74-97cc2c381802\") " pod="kserve-ci-e2e-test/success-200-isvc-1742e-predictor-5bf4dfd9dd-8j4zf"
May 25 12:21:01.131846 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:21:01.131810    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"error-404-isvc-1742e-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dbeb1945-c642-49c8-ab81-19aeffc568d3-error-404-isvc-1742e-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-1742e-predictor-5764c6c5cf-6wk56\" (UID: \"dbeb1945-c642-49c8-ab81-19aeffc568d3\") " pod="kserve-ci-e2e-test/error-404-isvc-1742e-predictor-5764c6c5cf-6wk56"
May 25 12:21:01.131846 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:21:01.131836    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6pd4\" (UniqueName: \"kubernetes.io/projected/dbeb1945-c642-49c8-ab81-19aeffc568d3-kube-api-access-h6pd4\") pod \"error-404-isvc-1742e-predictor-5764c6c5cf-6wk56\" (UID: \"dbeb1945-c642-49c8-ab81-19aeffc568d3\") " pod="kserve-ci-e2e-test/error-404-isvc-1742e-predictor-5764c6c5cf-6wk56"
May 25 12:21:01.132057 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:21:01.131882    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"success-200-isvc-1742e-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d1e41943-5b0a-4496-ba74-97cc2c381802-success-200-isvc-1742e-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-1742e-predictor-5bf4dfd9dd-8j4zf\" (UID: \"d1e41943-5b0a-4496-ba74-97cc2c381802\") " pod="kserve-ci-e2e-test/success-200-isvc-1742e-predictor-5bf4dfd9dd-8j4zf"
May 25 12:21:01.232372 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:21:01.232339    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dbeb1945-c642-49c8-ab81-19aeffc568d3-proxy-tls\") pod \"error-404-isvc-1742e-predictor-5764c6c5cf-6wk56\" (UID: \"dbeb1945-c642-49c8-ab81-19aeffc568d3\") " pod="kserve-ci-e2e-test/error-404-isvc-1742e-predictor-5764c6c5cf-6wk56"
May 25 12:21:01.232571 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:21:01.232392    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-9krzc\" (UniqueName: \"kubernetes.io/projected/d1e41943-5b0a-4496-ba74-97cc2c381802-kube-api-access-9krzc\") pod \"success-200-isvc-1742e-predictor-5bf4dfd9dd-8j4zf\" (UID: \"d1e41943-5b0a-4496-ba74-97cc2c381802\") " pod="kserve-ci-e2e-test/success-200-isvc-1742e-predictor-5bf4dfd9dd-8j4zf"
May 25 12:21:01.232571 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:21:01.232450    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d1e41943-5b0a-4496-ba74-97cc2c381802-proxy-tls\") pod \"success-200-isvc-1742e-predictor-5bf4dfd9dd-8j4zf\" (UID: \"d1e41943-5b0a-4496-ba74-97cc2c381802\") " pod="kserve-ci-e2e-test/success-200-isvc-1742e-predictor-5bf4dfd9dd-8j4zf"
May 25 12:21:01.232571 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:21:01.232473    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"error-404-isvc-1742e-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dbeb1945-c642-49c8-ab81-19aeffc568d3-error-404-isvc-1742e-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-1742e-predictor-5764c6c5cf-6wk56\" (UID: \"dbeb1945-c642-49c8-ab81-19aeffc568d3\") " pod="kserve-ci-e2e-test/error-404-isvc-1742e-predictor-5764c6c5cf-6wk56"
May 25 12:21:01.232571 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:21:01.232508    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-h6pd4\" (UniqueName: \"kubernetes.io/projected/dbeb1945-c642-49c8-ab81-19aeffc568d3-kube-api-access-h6pd4\") pod \"error-404-isvc-1742e-predictor-5764c6c5cf-6wk56\" (UID: \"dbeb1945-c642-49c8-ab81-19aeffc568d3\") " pod="kserve-ci-e2e-test/error-404-isvc-1742e-predictor-5764c6c5cf-6wk56"
May 25 12:21:01.232808 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:21:01.232554    2549 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-1742e-predictor-serving-cert: secret "error-404-isvc-1742e-predictor-serving-cert" not found
May 25 12:21:01.232808 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:21:01.232579    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"success-200-isvc-1742e-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d1e41943-5b0a-4496-ba74-97cc2c381802-success-200-isvc-1742e-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-1742e-predictor-5bf4dfd9dd-8j4zf\" (UID: \"d1e41943-5b0a-4496-ba74-97cc2c381802\") " pod="kserve-ci-e2e-test/success-200-isvc-1742e-predictor-5bf4dfd9dd-8j4zf"
May 25 12:21:01.232808 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:21:01.232672    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dbeb1945-c642-49c8-ab81-19aeffc568d3-proxy-tls podName:dbeb1945-c642-49c8-ab81-19aeffc568d3 nodeName:}" failed. No retries permitted until 2026-05-25 12:21:01.732633375 +0000 UTC m=+465.448448640 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/dbeb1945-c642-49c8-ab81-19aeffc568d3-proxy-tls") pod "error-404-isvc-1742e-predictor-5764c6c5cf-6wk56" (UID: "dbeb1945-c642-49c8-ab81-19aeffc568d3") : secret "error-404-isvc-1742e-predictor-serving-cert" not found
May 25 12:21:01.232981 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:21:01.232576    2549 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-1742e-predictor-serving-cert: secret "success-200-isvc-1742e-predictor-serving-cert" not found
May 25 12:21:01.233116 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:21:01.233103    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d1e41943-5b0a-4496-ba74-97cc2c381802-proxy-tls podName:d1e41943-5b0a-4496-ba74-97cc2c381802 nodeName:}" failed. No retries permitted until 2026-05-25 12:21:01.733084473 +0000 UTC m=+465.448899753 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/d1e41943-5b0a-4496-ba74-97cc2c381802-proxy-tls") pod "success-200-isvc-1742e-predictor-5bf4dfd9dd-8j4zf" (UID: "d1e41943-5b0a-4496-ba74-97cc2c381802") : secret "success-200-isvc-1742e-predictor-serving-cert" not found
May 25 12:21:01.233382 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:21:01.233355    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"error-404-isvc-1742e-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dbeb1945-c642-49c8-ab81-19aeffc568d3-error-404-isvc-1742e-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-1742e-predictor-5764c6c5cf-6wk56\" (UID: \"dbeb1945-c642-49c8-ab81-19aeffc568d3\") " pod="kserve-ci-e2e-test/error-404-isvc-1742e-predictor-5764c6c5cf-6wk56"
May 25 12:21:01.233486 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:21:01.233355    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"success-200-isvc-1742e-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d1e41943-5b0a-4496-ba74-97cc2c381802-success-200-isvc-1742e-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-1742e-predictor-5bf4dfd9dd-8j4zf\" (UID: \"d1e41943-5b0a-4496-ba74-97cc2c381802\") " pod="kserve-ci-e2e-test/success-200-isvc-1742e-predictor-5bf4dfd9dd-8j4zf"
May 25 12:21:01.245691 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:21:01.245663    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9krzc\" (UniqueName: \"kubernetes.io/projected/d1e41943-5b0a-4496-ba74-97cc2c381802-kube-api-access-9krzc\") pod \"success-200-isvc-1742e-predictor-5bf4dfd9dd-8j4zf\" (UID: \"d1e41943-5b0a-4496-ba74-97cc2c381802\") " pod="kserve-ci-e2e-test/success-200-isvc-1742e-predictor-5bf4dfd9dd-8j4zf"
May 25 12:21:01.246045 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:21:01.246024    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6pd4\" (UniqueName: \"kubernetes.io/projected/dbeb1945-c642-49c8-ab81-19aeffc568d3-kube-api-access-h6pd4\") pod \"error-404-isvc-1742e-predictor-5764c6c5cf-6wk56\" (UID: \"dbeb1945-c642-49c8-ab81-19aeffc568d3\") " pod="kserve-ci-e2e-test/error-404-isvc-1742e-predictor-5764c6c5cf-6wk56"
May 25 12:21:01.737657 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:21:01.737604    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d1e41943-5b0a-4496-ba74-97cc2c381802-proxy-tls\") pod \"success-200-isvc-1742e-predictor-5bf4dfd9dd-8j4zf\" (UID: \"d1e41943-5b0a-4496-ba74-97cc2c381802\") " pod="kserve-ci-e2e-test/success-200-isvc-1742e-predictor-5bf4dfd9dd-8j4zf"
May 25 12:21:01.737877 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:21:01.737700    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dbeb1945-c642-49c8-ab81-19aeffc568d3-proxy-tls\") pod \"error-404-isvc-1742e-predictor-5764c6c5cf-6wk56\" (UID: \"dbeb1945-c642-49c8-ab81-19aeffc568d3\") " pod="kserve-ci-e2e-test/error-404-isvc-1742e-predictor-5764c6c5cf-6wk56"
May 25 12:21:01.737877 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:21:01.737780    2549 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-1742e-predictor-serving-cert: secret "success-200-isvc-1742e-predictor-serving-cert" not found
May 25 12:21:01.737877 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:21:01.737813    2549 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-1742e-predictor-serving-cert: secret "error-404-isvc-1742e-predictor-serving-cert" not found
May 25 12:21:01.737877 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:21:01.737856    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d1e41943-5b0a-4496-ba74-97cc2c381802-proxy-tls podName:d1e41943-5b0a-4496-ba74-97cc2c381802 nodeName:}" failed. No retries permitted until 2026-05-25 12:21:02.737834784 +0000 UTC m=+466.453650069 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/d1e41943-5b0a-4496-ba74-97cc2c381802-proxy-tls") pod "success-200-isvc-1742e-predictor-5bf4dfd9dd-8j4zf" (UID: "d1e41943-5b0a-4496-ba74-97cc2c381802") : secret "success-200-isvc-1742e-predictor-serving-cert" not found
May 25 12:21:01.737877 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:21:01.737875    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dbeb1945-c642-49c8-ab81-19aeffc568d3-proxy-tls podName:dbeb1945-c642-49c8-ab81-19aeffc568d3 nodeName:}" failed. No retries permitted until 2026-05-25 12:21:02.737864701 +0000 UTC m=+466.453679973 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/dbeb1945-c642-49c8-ab81-19aeffc568d3-proxy-tls") pod "error-404-isvc-1742e-predictor-5764c6c5cf-6wk56" (UID: "dbeb1945-c642-49c8-ab81-19aeffc568d3") : secret "error-404-isvc-1742e-predictor-serving-cert" not found
May 25 12:21:02.748245 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:21:02.748160    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d1e41943-5b0a-4496-ba74-97cc2c381802-proxy-tls\") pod \"success-200-isvc-1742e-predictor-5bf4dfd9dd-8j4zf\" (UID: \"d1e41943-5b0a-4496-ba74-97cc2c381802\") " pod="kserve-ci-e2e-test/success-200-isvc-1742e-predictor-5bf4dfd9dd-8j4zf"
May 25 12:21:02.748642 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:21:02.748322    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dbeb1945-c642-49c8-ab81-19aeffc568d3-proxy-tls\") pod \"error-404-isvc-1742e-predictor-5764c6c5cf-6wk56\" (UID: \"dbeb1945-c642-49c8-ab81-19aeffc568d3\") " pod="kserve-ci-e2e-test/error-404-isvc-1742e-predictor-5764c6c5cf-6wk56"
May 25 12:21:02.750780 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:21:02.750755    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dbeb1945-c642-49c8-ab81-19aeffc568d3-proxy-tls\") pod \"error-404-isvc-1742e-predictor-5764c6c5cf-6wk56\" (UID: \"dbeb1945-c642-49c8-ab81-19aeffc568d3\") " pod="kserve-ci-e2e-test/error-404-isvc-1742e-predictor-5764c6c5cf-6wk56"
May 25 12:21:02.750907 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:21:02.750884    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d1e41943-5b0a-4496-ba74-97cc2c381802-proxy-tls\") pod \"success-200-isvc-1742e-predictor-5bf4dfd9dd-8j4zf\" (UID: \"d1e41943-5b0a-4496-ba74-97cc2c381802\") " pod="kserve-ci-e2e-test/success-200-isvc-1742e-predictor-5bf4dfd9dd-8j4zf"
May 25 12:21:02.887870 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:21:02.887827    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-1742e-predictor-5bf4dfd9dd-8j4zf"
May 25 12:21:02.927766 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:21:02.927725    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-1742e-predictor-5764c6c5cf-6wk56"
May 25 12:21:03.038325 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:21:03.038289    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-1742e-predictor-5bf4dfd9dd-8j4zf"]
May 25 12:21:03.042354 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:21:03.042315    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1e41943_5b0a_4496_ba74_97cc2c381802.slice/crio-ab87bf0576f84990e8bf7dbf90acd933702f443eacf3ed3c871c5bab1924d7c9 WatchSource:0}: Error finding container ab87bf0576f84990e8bf7dbf90acd933702f443eacf3ed3c871c5bab1924d7c9: Status 404 returned error can't find the container with id ab87bf0576f84990e8bf7dbf90acd933702f443eacf3ed3c871c5bab1924d7c9
May 25 12:21:03.088585 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:21:03.088559    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-1742e-predictor-5764c6c5cf-6wk56"]
May 25 12:21:03.089909 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:21:03.089879    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbeb1945_c642_49c8_ab81_19aeffc568d3.slice/crio-ed69f8138523886cf2244cf63e0200cad3cfe0bd71e89f98bb9f180889e20bdf WatchSource:0}: Error finding container ed69f8138523886cf2244cf63e0200cad3cfe0bd71e89f98bb9f180889e20bdf: Status 404 returned error can't find the container with id ed69f8138523886cf2244cf63e0200cad3cfe0bd71e89f98bb9f180889e20bdf
May 25 12:21:03.597800 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:21:03.597720    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-1742e-predictor-5bf4dfd9dd-8j4zf" event={"ID":"d1e41943-5b0a-4496-ba74-97cc2c381802","Type":"ContainerStarted","Data":"ab87bf0576f84990e8bf7dbf90acd933702f443eacf3ed3c871c5bab1924d7c9"}
May 25 12:21:03.599731 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:21:03.599685    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-1742e-predictor-5764c6c5cf-6wk56" event={"ID":"dbeb1945-c642-49c8-ab81-19aeffc568d3","Type":"ContainerStarted","Data":"ed69f8138523886cf2244cf63e0200cad3cfe0bd71e89f98bb9f180889e20bdf"}
May 25 12:21:16.654956 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:21:16.654878    2549 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-console/console-65fff857b4-rbmpj" podUID="89a53512-040a-4621-b9ec-386b21177fc5" containerName="console" containerID="cri-o://005c340cfac08932427133540638fc4d94f7c4d41a4659e65c5cb87943182c6f" gracePeriod=15
May 25 12:21:17.412047 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:21:17.412020    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-65fff857b4-rbmpj_89a53512-040a-4621-b9ec-386b21177fc5/console/0.log"
May 25 12:21:17.412159 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:21:17.412102    2549 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65fff857b4-rbmpj"
May 25 12:21:17.495911 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:21:17.495881    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/89a53512-040a-4621-b9ec-386b21177fc5-oauth-serving-cert\") pod \"89a53512-040a-4621-b9ec-386b21177fc5\" (UID: \"89a53512-040a-4621-b9ec-386b21177fc5\") "
May 25 12:21:17.496093 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:21:17.495919    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b9vx2\" (UniqueName: \"kubernetes.io/projected/89a53512-040a-4621-b9ec-386b21177fc5-kube-api-access-b9vx2\") pod \"89a53512-040a-4621-b9ec-386b21177fc5\" (UID: \"89a53512-040a-4621-b9ec-386b21177fc5\") "
May 25 12:21:17.496093 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:21:17.495939    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/89a53512-040a-4621-b9ec-386b21177fc5-console-config\") pod \"89a53512-040a-4621-b9ec-386b21177fc5\" (UID: \"89a53512-040a-4621-b9ec-386b21177fc5\") "
May 25 12:21:17.496093 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:21:17.496069    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/89a53512-040a-4621-b9ec-386b21177fc5-console-serving-cert\") pod \"89a53512-040a-4621-b9ec-386b21177fc5\" (UID: \"89a53512-040a-4621-b9ec-386b21177fc5\") "
May 25 12:21:17.496282 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:21:17.496120    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/89a53512-040a-4621-b9ec-386b21177fc5-service-ca\") pod \"89a53512-040a-4621-b9ec-386b21177fc5\" (UID: \"89a53512-040a-4621-b9ec-386b21177fc5\") "
May 25 12:21:17.496282 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:21:17.496145    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/89a53512-040a-4621-b9ec-386b21177fc5-console-oauth-config\") pod \"89a53512-040a-4621-b9ec-386b21177fc5\" (UID: \"89a53512-040a-4621-b9ec-386b21177fc5\") "
May 25 12:21:17.496282 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:21:17.496221    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/89a53512-040a-4621-b9ec-386b21177fc5-trusted-ca-bundle\") pod \"89a53512-040a-4621-b9ec-386b21177fc5\" (UID: \"89a53512-040a-4621-b9ec-386b21177fc5\") "
May 25 12:21:17.496421 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:21:17.496379    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89a53512-040a-4621-b9ec-386b21177fc5-console-config" (OuterVolumeSpecName: "console-config") pod "89a53512-040a-4621-b9ec-386b21177fc5" (UID: "89a53512-040a-4621-b9ec-386b21177fc5"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 12:21:17.496476 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:21:17.496433    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89a53512-040a-4621-b9ec-386b21177fc5-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "89a53512-040a-4621-b9ec-386b21177fc5" (UID: "89a53512-040a-4621-b9ec-386b21177fc5"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 12:21:17.496578 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:21:17.496562    2549 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/89a53512-040a-4621-b9ec-386b21177fc5-console-config\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
May 25 12:21:17.496631 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:21:17.496584    2549 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/89a53512-040a-4621-b9ec-386b21177fc5-oauth-serving-cert\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
May 25 12:21:17.496631 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:21:17.496589    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89a53512-040a-4621-b9ec-386b21177fc5-service-ca" (OuterVolumeSpecName: "service-ca") pod "89a53512-040a-4621-b9ec-386b21177fc5" (UID: "89a53512-040a-4621-b9ec-386b21177fc5"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 12:21:17.496766 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:21:17.496743    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89a53512-040a-4621-b9ec-386b21177fc5-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "89a53512-040a-4621-b9ec-386b21177fc5" (UID: "89a53512-040a-4621-b9ec-386b21177fc5"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 12:21:17.498289 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:21:17.498268    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89a53512-040a-4621-b9ec-386b21177fc5-kube-api-access-b9vx2" (OuterVolumeSpecName: "kube-api-access-b9vx2") pod "89a53512-040a-4621-b9ec-386b21177fc5" (UID: "89a53512-040a-4621-b9ec-386b21177fc5"). InnerVolumeSpecName "kube-api-access-b9vx2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 12:21:17.498715 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:21:17.498688    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89a53512-040a-4621-b9ec-386b21177fc5-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "89a53512-040a-4621-b9ec-386b21177fc5" (UID: "89a53512-040a-4621-b9ec-386b21177fc5"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 12:21:17.498868 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:21:17.498836    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89a53512-040a-4621-b9ec-386b21177fc5-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "89a53512-040a-4621-b9ec-386b21177fc5" (UID: "89a53512-040a-4621-b9ec-386b21177fc5"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 12:21:17.597886 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:21:17.597848    2549 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/89a53512-040a-4621-b9ec-386b21177fc5-console-serving-cert\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
May 25 12:21:17.597886 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:21:17.597876    2549 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/89a53512-040a-4621-b9ec-386b21177fc5-service-ca\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
May 25 12:21:17.597886 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:21:17.597885    2549 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/89a53512-040a-4621-b9ec-386b21177fc5-console-oauth-config\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
May 25 12:21:17.597886 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:21:17.597894    2549 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/89a53512-040a-4621-b9ec-386b21177fc5-trusted-ca-bundle\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
May 25 12:21:17.598256 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:21:17.597903    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b9vx2\" (UniqueName: \"kubernetes.io/projected/89a53512-040a-4621-b9ec-386b21177fc5-kube-api-access-b9vx2\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
May 25 12:21:17.664329 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:21:17.664290    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-1742e-predictor-5bf4dfd9dd-8j4zf" event={"ID":"d1e41943-5b0a-4496-ba74-97cc2c381802","Type":"ContainerStarted","Data":"31877611702d518dbb0a94a91ed23914c7737e30b00b344990e3cdf622763444"}
May 25 12:21:17.665436 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:21:17.665415    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-65fff857b4-rbmpj_89a53512-040a-4621-b9ec-386b21177fc5/console/0.log"
May 25 12:21:17.665550 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:21:17.665448    2549 generic.go:358] "Generic (PLEG): container finished" podID="89a53512-040a-4621-b9ec-386b21177fc5" containerID="005c340cfac08932427133540638fc4d94f7c4d41a4659e65c5cb87943182c6f" exitCode=2
May 25 12:21:17.665550 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:21:17.665504    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65fff857b4-rbmpj" event={"ID":"89a53512-040a-4621-b9ec-386b21177fc5","Type":"ContainerDied","Data":"005c340cfac08932427133540638fc4d94f7c4d41a4659e65c5cb87943182c6f"}
May 25 12:21:17.665550 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:21:17.665514    2549 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65fff857b4-rbmpj"
May 25 12:21:17.665550 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:21:17.665524    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65fff857b4-rbmpj" event={"ID":"89a53512-040a-4621-b9ec-386b21177fc5","Type":"ContainerDied","Data":"b4737e8c59598ec23e7244eb1efb741174421a247bab0d88970bac6b8eaa6972"}
May 25 12:21:17.665550 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:21:17.665539    2549 scope.go:117] "RemoveContainer" containerID="005c340cfac08932427133540638fc4d94f7c4d41a4659e65c5cb87943182c6f"
May 25 12:21:17.667088 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:21:17.667067    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-1742e-predictor-5764c6c5cf-6wk56" event={"ID":"dbeb1945-c642-49c8-ab81-19aeffc568d3","Type":"ContainerStarted","Data":"df75b6b4a3bd1bf2ccdb664cff7c045b0c1788fa7f32b7b8abf07f1a0ec8fe1e"}
May 25 12:21:17.674142 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:21:17.674126    2549 scope.go:117] "RemoveContainer" containerID="005c340cfac08932427133540638fc4d94f7c4d41a4659e65c5cb87943182c6f"
May 25 12:21:17.674418 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:21:17.674399    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"005c340cfac08932427133540638fc4d94f7c4d41a4659e65c5cb87943182c6f\": container with ID starting with 005c340cfac08932427133540638fc4d94f7c4d41a4659e65c5cb87943182c6f not found: ID does not exist" containerID="005c340cfac08932427133540638fc4d94f7c4d41a4659e65c5cb87943182c6f"
May 25 12:21:17.674485 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:21:17.674427    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"005c340cfac08932427133540638fc4d94f7c4d41a4659e65c5cb87943182c6f"} err="failed to get container status \"005c340cfac08932427133540638fc4d94f7c4d41a4659e65c5cb87943182c6f\": rpc error: code = NotFound desc = could not find container \"005c340cfac08932427133540638fc4d94f7c4d41a4659e65c5cb87943182c6f\": container with ID starting with 005c340cfac08932427133540638fc4d94f7c4d41a4659e65c5cb87943182c6f not found: ID does not exist"
May 25 12:21:17.692158 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:21:17.692125    2549 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-65fff857b4-rbmpj"]
May 25 12:21:17.696266 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:21:17.696243    2549 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-65fff857b4-rbmpj"]
May 25 12:21:18.850331 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:21:18.850294    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="89a53512-040a-4621-b9ec-386b21177fc5" path="/var/lib/kubelet/pods/89a53512-040a-4621-b9ec-386b21177fc5/volumes"
May 25 12:21:20.682555 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:21:20.682514    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-1742e-predictor-5764c6c5cf-6wk56" event={"ID":"dbeb1945-c642-49c8-ab81-19aeffc568d3","Type":"ContainerStarted","Data":"2fc13c2af4e674a2b1cca3810cad7241c299b627565937ff6cb219737fb75b48"}
May 25 12:21:20.683039 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:21:20.682563    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-1742e-predictor-5764c6c5cf-6wk56"
May 25 12:21:20.683039 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:21:20.682670    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-1742e-predictor-5764c6c5cf-6wk56"
May 25 12:21:20.684378 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:21:20.684354    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-1742e-predictor-5bf4dfd9dd-8j4zf" event={"ID":"d1e41943-5b0a-4496-ba74-97cc2c381802","Type":"ContainerStarted","Data":"a496dd35397c877274a093011441e8747e1f833d098fbf071ba91b40cdc0b50a"}
May 25 12:21:20.684508 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:21:20.684378    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-1742e-predictor-5764c6c5cf-6wk56" podUID="dbeb1945-c642-49c8-ab81-19aeffc568d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
May 25 12:21:20.684575 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:21:20.684562    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-1742e-predictor-5bf4dfd9dd-8j4zf"
May 25 12:21:20.705940 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:21:20.705886    2549 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-1742e-predictor-5764c6c5cf-6wk56" podStartSLOduration=2.817493623 podStartE2EDuration="19.70587495s" podCreationTimestamp="2026-05-25 12:21:01 +0000 UTC" firstStartedPulling="2026-05-25 12:21:03.09166142 +0000 UTC m=+466.807476686" lastFinishedPulling="2026-05-25 12:21:19.980042745 +0000 UTC m=+483.695858013" observedRunningTime="2026-05-25 12:21:20.703716501 +0000 UTC m=+484.419531812" watchObservedRunningTime="2026-05-25 12:21:20.70587495 +0000 UTC m=+484.421690288"
May 25 12:21:20.724250 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:21:20.724184    2549 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-1742e-predictor-5bf4dfd9dd-8j4zf" podStartSLOduration=2.776794062 podStartE2EDuration="19.724170747s" podCreationTimestamp="2026-05-25 12:21:01 +0000 UTC" firstStartedPulling="2026-05-25 12:21:03.044327744 +0000 UTC m=+466.760143012" lastFinishedPulling="2026-05-25 12:21:19.991704426 +0000 UTC m=+483.707519697" observedRunningTime="2026-05-25 12:21:20.722921974 +0000 UTC m=+484.438737261" watchObservedRunningTime="2026-05-25 12:21:20.724170747 +0000 UTC m=+484.439986033"
May 25 12:21:21.687993 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:21:21.687955    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-1742e-predictor-5bf4dfd9dd-8j4zf"
May 25 12:21:21.688467 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:21:21.688129    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-1742e-predictor-5764c6c5cf-6wk56" podUID="dbeb1945-c642-49c8-ab81-19aeffc568d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
May 25 12:21:21.689426 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:21:21.689399    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-1742e-predictor-5bf4dfd9dd-8j4zf" podUID="d1e41943-5b0a-4496-ba74-97cc2c381802" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
May 25 12:21:22.691871 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:21:22.691832    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-1742e-predictor-5bf4dfd9dd-8j4zf" podUID="d1e41943-5b0a-4496-ba74-97cc2c381802" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
May 25 12:21:26.692805 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:21:26.692776    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-1742e-predictor-5764c6c5cf-6wk56"
May 25 12:21:26.693266 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:21:26.693238    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-1742e-predictor-5764c6c5cf-6wk56" podUID="dbeb1945-c642-49c8-ab81-19aeffc568d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
May 25 12:21:27.696003 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:21:27.695972    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-1742e-predictor-5bf4dfd9dd-8j4zf"
May 25 12:21:27.696569 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:21:27.696542    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-1742e-predictor-5bf4dfd9dd-8j4zf" podUID="d1e41943-5b0a-4496-ba74-97cc2c381802" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
May 25 12:21:36.694041 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:21:36.693998    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-1742e-predictor-5764c6c5cf-6wk56" podUID="dbeb1945-c642-49c8-ab81-19aeffc568d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
May 25 12:21:37.697097 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:21:37.697059    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-1742e-predictor-5bf4dfd9dd-8j4zf" podUID="d1e41943-5b0a-4496-ba74-97cc2c381802" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
May 25 12:21:46.693621 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:21:46.693579    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-1742e-predictor-5764c6c5cf-6wk56" podUID="dbeb1945-c642-49c8-ab81-19aeffc568d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
May 25 12:21:47.697352 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:21:47.697306    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-1742e-predictor-5bf4dfd9dd-8j4zf" podUID="d1e41943-5b0a-4496-ba74-97cc2c381802" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
May 25 12:21:56.694245 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:21:56.694188    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-1742e-predictor-5764c6c5cf-6wk56" podUID="dbeb1945-c642-49c8-ab81-19aeffc568d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
May 25 12:21:57.696702 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:21:57.696657    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-1742e-predictor-5bf4dfd9dd-8j4zf" podUID="d1e41943-5b0a-4496-ba74-97cc2c381802" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
May 25 12:22:06.694284 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:22:06.694177    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-1742e-predictor-5764c6c5cf-6wk56"
May 25 12:22:07.697910 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:22:07.697880    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-1742e-predictor-5bf4dfd9dd-8j4zf"
May 25 12:22:30.929545 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:22:30.929509    2549 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-1742e-predictor-5764c6c5cf-6wk56"]
May 25 12:22:30.930007 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:22:30.929882    2549 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-1742e-predictor-5764c6c5cf-6wk56" podUID="dbeb1945-c642-49c8-ab81-19aeffc568d3" containerName="kserve-container" containerID="cri-o://df75b6b4a3bd1bf2ccdb664cff7c045b0c1788fa7f32b7b8abf07f1a0ec8fe1e" gracePeriod=30
May 25 12:22:30.930080 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:22:30.929983    2549 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-1742e-predictor-5764c6c5cf-6wk56" podUID="dbeb1945-c642-49c8-ab81-19aeffc568d3" containerName="kube-rbac-proxy" containerID="cri-o://2fc13c2af4e674a2b1cca3810cad7241c299b627565937ff6cb219737fb75b48" gracePeriod=30
May 25 12:22:30.980448 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:22:30.980415    2549 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-1742e-predictor-5bf4dfd9dd-8j4zf"]
May 25 12:22:30.980877 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:22:30.980730    2549 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-1742e-predictor-5bf4dfd9dd-8j4zf" podUID="d1e41943-5b0a-4496-ba74-97cc2c381802" containerName="kserve-container" containerID="cri-o://31877611702d518dbb0a94a91ed23914c7737e30b00b344990e3cdf622763444" gracePeriod=30
May 25 12:22:30.980877 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:22:30.980764    2549 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-1742e-predictor-5bf4dfd9dd-8j4zf" podUID="d1e41943-5b0a-4496-ba74-97cc2c381802" containerName="kube-rbac-proxy" containerID="cri-o://a496dd35397c877274a093011441e8747e1f833d098fbf071ba91b40cdc0b50a" gracePeriod=30
May 25 12:22:31.028935 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:22:31.028904    2549 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-40bda-predictor-776fc95cd4-dr29n"]
May 25 12:22:31.029375 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:22:31.029347    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89a53512-040a-4621-b9ec-386b21177fc5" containerName="console"
May 25 12:22:31.029375 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:22:31.029371    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="89a53512-040a-4621-b9ec-386b21177fc5" containerName="console"
May 25 12:22:31.029541 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:22:31.029496    2549 memory_manager.go:356] "RemoveStaleState removing state" podUID="89a53512-040a-4621-b9ec-386b21177fc5" containerName="console"
May 25 12:22:31.033945 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:22:31.033925    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-40bda-predictor-776fc95cd4-dr29n"
May 25 12:22:31.036687 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:22:31.036666    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-40bda-predictor-serving-cert\""
May 25 12:22:31.036806 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:22:31.036667    2549 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-40bda-kube-rbac-proxy-sar-config\""
May 25 12:22:31.042052 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:22:31.042017    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-40bda-predictor-776fc95cd4-dr29n"]
May 25 12:22:31.068680 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:22:31.068636    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7975c442-27c9-43e0-a7b5-5eba5e770eb3-proxy-tls\") pod \"success-200-isvc-40bda-predictor-776fc95cd4-dr29n\" (UID: \"7975c442-27c9-43e0-a7b5-5eba5e770eb3\") " pod="kserve-ci-e2e-test/success-200-isvc-40bda-predictor-776fc95cd4-dr29n"
May 25 12:22:31.068978 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:22:31.068947    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"success-200-isvc-40bda-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7975c442-27c9-43e0-a7b5-5eba5e770eb3-success-200-isvc-40bda-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-40bda-predictor-776fc95cd4-dr29n\" (UID: \"7975c442-27c9-43e0-a7b5-5eba5e770eb3\") " pod="kserve-ci-e2e-test/success-200-isvc-40bda-predictor-776fc95cd4-dr29n"
May 25 12:22:31.069051 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:22:31.069030    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8vk8\" (UniqueName: \"kubernetes.io/projected/7975c442-27c9-43e0-a7b5-5eba5e770eb3-kube-api-access-c8vk8\") pod \"success-200-isvc-40bda-predictor-776fc95cd4-dr29n\" (UID: \"7975c442-27c9-43e0-a7b5-5eba5e770eb3\") " pod="kserve-ci-e2e-test/success-200-isvc-40bda-predictor-776fc95cd4-dr29n"
May 25 12:22:31.124631 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:22:31.124598    2549 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-40bda-predictor-6664b4979f-g8jxd"]
May 25 12:22:31.128809 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:22:31.128787    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-40bda-predictor-6664b4979f-g8jxd"
May 25 12:22:31.131754 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:22:31.131730    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-40bda-predictor-serving-cert\""
May 25 12:22:31.131754 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:22:31.131743    2549 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-40bda-kube-rbac-proxy-sar-config\""
May 25 12:22:31.139084 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:22:31.139057    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-40bda-predictor-6664b4979f-g8jxd"]
May 25 12:22:31.169543 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:22:31.169506    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7975c442-27c9-43e0-a7b5-5eba5e770eb3-proxy-tls\") pod \"success-200-isvc-40bda-predictor-776fc95cd4-dr29n\" (UID: \"7975c442-27c9-43e0-a7b5-5eba5e770eb3\") " pod="kserve-ci-e2e-test/success-200-isvc-40bda-predictor-776fc95cd4-dr29n"
May 25 12:22:31.169719 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:22:31.169563    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"success-200-isvc-40bda-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7975c442-27c9-43e0-a7b5-5eba5e770eb3-success-200-isvc-40bda-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-40bda-predictor-776fc95cd4-dr29n\" (UID: \"7975c442-27c9-43e0-a7b5-5eba5e770eb3\") " pod="kserve-ci-e2e-test/success-200-isvc-40bda-predictor-776fc95cd4-dr29n"
May 25 12:22:31.169719 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:22:31.169599    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-c8vk8\" (UniqueName: \"kubernetes.io/projected/7975c442-27c9-43e0-a7b5-5eba5e770eb3-kube-api-access-c8vk8\") pod \"success-200-isvc-40bda-predictor-776fc95cd4-dr29n\" (UID: \"7975c442-27c9-43e0-a7b5-5eba5e770eb3\") " pod="kserve-ci-e2e-test/success-200-isvc-40bda-predictor-776fc95cd4-dr29n"
May 25 12:22:31.169719 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:22:31.169632    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"error-404-isvc-40bda-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/771a87fe-641c-40bd-8588-a6241156f7b0-error-404-isvc-40bda-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-40bda-predictor-6664b4979f-g8jxd\" (UID: \"771a87fe-641c-40bd-8588-a6241156f7b0\") " pod="kserve-ci-e2e-test/error-404-isvc-40bda-predictor-6664b4979f-g8jxd"
May 25 12:22:31.169719 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:22:31.169662    2549 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-40bda-predictor-serving-cert: secret "success-200-isvc-40bda-predictor-serving-cert" not found
May 25 12:22:31.169719 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:22:31.169686    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/771a87fe-641c-40bd-8588-a6241156f7b0-proxy-tls\") pod \"error-404-isvc-40bda-predictor-6664b4979f-g8jxd\" (UID: \"771a87fe-641c-40bd-8588-a6241156f7b0\") " pod="kserve-ci-e2e-test/error-404-isvc-40bda-predictor-6664b4979f-g8jxd"
May 25 12:22:31.169719 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:22:31.169717    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2r5mr\" (UniqueName: \"kubernetes.io/projected/771a87fe-641c-40bd-8588-a6241156f7b0-kube-api-access-2r5mr\") pod \"error-404-isvc-40bda-predictor-6664b4979f-g8jxd\" (UID: \"771a87fe-641c-40bd-8588-a6241156f7b0\") " pod="kserve-ci-e2e-test/error-404-isvc-40bda-predictor-6664b4979f-g8jxd"
May 25 12:22:31.170028 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:22:31.169752    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7975c442-27c9-43e0-a7b5-5eba5e770eb3-proxy-tls podName:7975c442-27c9-43e0-a7b5-5eba5e770eb3 nodeName:}" failed. No retries permitted until 2026-05-25 12:22:31.669728307 +0000 UTC m=+555.385543589 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/7975c442-27c9-43e0-a7b5-5eba5e770eb3-proxy-tls") pod "success-200-isvc-40bda-predictor-776fc95cd4-dr29n" (UID: "7975c442-27c9-43e0-a7b5-5eba5e770eb3") : secret "success-200-isvc-40bda-predictor-serving-cert" not found
May 25 12:22:31.170344 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:22:31.170326    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"success-200-isvc-40bda-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7975c442-27c9-43e0-a7b5-5eba5e770eb3-success-200-isvc-40bda-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-40bda-predictor-776fc95cd4-dr29n\" (UID: \"7975c442-27c9-43e0-a7b5-5eba5e770eb3\") " pod="kserve-ci-e2e-test/success-200-isvc-40bda-predictor-776fc95cd4-dr29n"
May 25 12:22:31.182144 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:22:31.182085    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8vk8\" (UniqueName: \"kubernetes.io/projected/7975c442-27c9-43e0-a7b5-5eba5e770eb3-kube-api-access-c8vk8\") pod \"success-200-isvc-40bda-predictor-776fc95cd4-dr29n\" (UID: \"7975c442-27c9-43e0-a7b5-5eba5e770eb3\") " pod="kserve-ci-e2e-test/success-200-isvc-40bda-predictor-776fc95cd4-dr29n"
May 25 12:22:31.270669 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:22:31.270628    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"error-404-isvc-40bda-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/771a87fe-641c-40bd-8588-a6241156f7b0-error-404-isvc-40bda-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-40bda-predictor-6664b4979f-g8jxd\" (UID: \"771a87fe-641c-40bd-8588-a6241156f7b0\") " pod="kserve-ci-e2e-test/error-404-isvc-40bda-predictor-6664b4979f-g8jxd"
May 25 12:22:31.270897 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:22:31.270773    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/771a87fe-641c-40bd-8588-a6241156f7b0-proxy-tls\") pod \"error-404-isvc-40bda-predictor-6664b4979f-g8jxd\" (UID: \"771a87fe-641c-40bd-8588-a6241156f7b0\") " pod="kserve-ci-e2e-test/error-404-isvc-40bda-predictor-6664b4979f-g8jxd"
May 25 12:22:31.270897 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:22:31.270807    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-2r5mr\" (UniqueName: \"kubernetes.io/projected/771a87fe-641c-40bd-8588-a6241156f7b0-kube-api-access-2r5mr\") pod \"error-404-isvc-40bda-predictor-6664b4979f-g8jxd\" (UID: \"771a87fe-641c-40bd-8588-a6241156f7b0\") " pod="kserve-ci-e2e-test/error-404-isvc-40bda-predictor-6664b4979f-g8jxd"
May 25 12:22:31.271009 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:22:31.270916    2549 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-40bda-predictor-serving-cert: secret "error-404-isvc-40bda-predictor-serving-cert" not found
May 25 12:22:31.271009 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:22:31.270999    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/771a87fe-641c-40bd-8588-a6241156f7b0-proxy-tls podName:771a87fe-641c-40bd-8588-a6241156f7b0 nodeName:}" failed. No retries permitted until 2026-05-25 12:22:31.770976811 +0000 UTC m=+555.486792091 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/771a87fe-641c-40bd-8588-a6241156f7b0-proxy-tls") pod "error-404-isvc-40bda-predictor-6664b4979f-g8jxd" (UID: "771a87fe-641c-40bd-8588-a6241156f7b0") : secret "error-404-isvc-40bda-predictor-serving-cert" not found
May 25 12:22:31.271427 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:22:31.271406    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"error-404-isvc-40bda-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/771a87fe-641c-40bd-8588-a6241156f7b0-error-404-isvc-40bda-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-40bda-predictor-6664b4979f-g8jxd\" (UID: \"771a87fe-641c-40bd-8588-a6241156f7b0\") " pod="kserve-ci-e2e-test/error-404-isvc-40bda-predictor-6664b4979f-g8jxd"
May 25 12:22:31.281312 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:22:31.281285    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2r5mr\" (UniqueName: \"kubernetes.io/projected/771a87fe-641c-40bd-8588-a6241156f7b0-kube-api-access-2r5mr\") pod \"error-404-isvc-40bda-predictor-6664b4979f-g8jxd\" (UID: \"771a87fe-641c-40bd-8588-a6241156f7b0\") " pod="kserve-ci-e2e-test/error-404-isvc-40bda-predictor-6664b4979f-g8jxd"
May 25 12:22:31.674572 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:22:31.674535    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7975c442-27c9-43e0-a7b5-5eba5e770eb3-proxy-tls\") pod \"success-200-isvc-40bda-predictor-776fc95cd4-dr29n\" (UID: \"7975c442-27c9-43e0-a7b5-5eba5e770eb3\") " pod="kserve-ci-e2e-test/success-200-isvc-40bda-predictor-776fc95cd4-dr29n"
May 25 12:22:31.677083 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:22:31.677047    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7975c442-27c9-43e0-a7b5-5eba5e770eb3-proxy-tls\") pod \"success-200-isvc-40bda-predictor-776fc95cd4-dr29n\" (UID: \"7975c442-27c9-43e0-a7b5-5eba5e770eb3\") " pod="kserve-ci-e2e-test/success-200-isvc-40bda-predictor-776fc95cd4-dr29n"
May 25 12:22:31.688151 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:22:31.688117    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-1742e-predictor-5764c6c5cf-6wk56" podUID="dbeb1945-c642-49c8-ab81-19aeffc568d3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.35:8643/healthz\": dial tcp 10.134.0.35:8643: connect: connection refused"
May 25 12:22:31.775617 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:22:31.775575    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/771a87fe-641c-40bd-8588-a6241156f7b0-proxy-tls\") pod \"error-404-isvc-40bda-predictor-6664b4979f-g8jxd\" (UID: \"771a87fe-641c-40bd-8588-a6241156f7b0\") " pod="kserve-ci-e2e-test/error-404-isvc-40bda-predictor-6664b4979f-g8jxd"
May 25 12:22:31.777971 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:22:31.777939    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/771a87fe-641c-40bd-8588-a6241156f7b0-proxy-tls\") pod \"error-404-isvc-40bda-predictor-6664b4979f-g8jxd\" (UID: \"771a87fe-641c-40bd-8588-a6241156f7b0\") " pod="kserve-ci-e2e-test/error-404-isvc-40bda-predictor-6664b4979f-g8jxd"
May 25 12:22:31.938615 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:22:31.938529    2549 generic.go:358] "Generic (PLEG): container finished" podID="d1e41943-5b0a-4496-ba74-97cc2c381802" containerID="a496dd35397c877274a093011441e8747e1f833d098fbf071ba91b40cdc0b50a" exitCode=2
May 25 12:22:31.939006 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:22:31.938601    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-1742e-predictor-5bf4dfd9dd-8j4zf" event={"ID":"d1e41943-5b0a-4496-ba74-97cc2c381802","Type":"ContainerDied","Data":"a496dd35397c877274a093011441e8747e1f833d098fbf071ba91b40cdc0b50a"}
May 25 12:22:31.940206 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:22:31.940173    2549 generic.go:358] "Generic (PLEG): container finished" podID="dbeb1945-c642-49c8-ab81-19aeffc568d3" containerID="2fc13c2af4e674a2b1cca3810cad7241c299b627565937ff6cb219737fb75b48" exitCode=2
May 25 12:22:31.940304 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:22:31.940219    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-1742e-predictor-5764c6c5cf-6wk56" event={"ID":"dbeb1945-c642-49c8-ab81-19aeffc568d3","Type":"ContainerDied","Data":"2fc13c2af4e674a2b1cca3810cad7241c299b627565937ff6cb219737fb75b48"}
May 25 12:22:31.947404 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:22:31.947380    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-40bda-predictor-776fc95cd4-dr29n"
May 25 12:22:32.042549 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:22:32.042513    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-40bda-predictor-6664b4979f-g8jxd"
May 25 12:22:32.087303 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:22:32.087259    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-40bda-predictor-776fc95cd4-dr29n"]
May 25 12:22:32.091938 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:22:32.091882    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7975c442_27c9_43e0_a7b5_5eba5e770eb3.slice/crio-36452567b8f9888733a43275332c57b822bfb19628f3f148a2b9249d0108594f WatchSource:0}: Error finding container 36452567b8f9888733a43275332c57b822bfb19628f3f148a2b9249d0108594f: Status 404 returned error can't find the container with id 36452567b8f9888733a43275332c57b822bfb19628f3f148a2b9249d0108594f
May 25 12:22:32.181000 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:22:32.180970    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-40bda-predictor-6664b4979f-g8jxd"]
May 25 12:22:32.186635 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:22:32.186605    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod771a87fe_641c_40bd_8588_a6241156f7b0.slice/crio-17141695b02291d3a4daf9e2aa9d8570feec818030c5ac60d0ba8f650611d317 WatchSource:0}: Error finding container 17141695b02291d3a4daf9e2aa9d8570feec818030c5ac60d0ba8f650611d317: Status 404 returned error can't find the container with id 17141695b02291d3a4daf9e2aa9d8570feec818030c5ac60d0ba8f650611d317
May 25 12:22:32.692943 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:22:32.692895    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-1742e-predictor-5bf4dfd9dd-8j4zf" podUID="d1e41943-5b0a-4496-ba74-97cc2c381802" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.34:8643/healthz\": dial tcp 10.134.0.34:8643: connect: connection refused"
May 25 12:22:32.945827 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:22:32.945725    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-40bda-predictor-6664b4979f-g8jxd" event={"ID":"771a87fe-641c-40bd-8588-a6241156f7b0","Type":"ContainerStarted","Data":"f0d95a203f3797358bfad4055a2dd004f37bc20c03377b430648647864437335"}
May 25 12:22:32.945827 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:22:32.945769    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-40bda-predictor-6664b4979f-g8jxd" event={"ID":"771a87fe-641c-40bd-8588-a6241156f7b0","Type":"ContainerStarted","Data":"f3a663c372c80352fb12e353d2d07a1103246f0ea31ed16abf773d60834a179e"}
May 25 12:22:32.945827 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:22:32.945784    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-40bda-predictor-6664b4979f-g8jxd" event={"ID":"771a87fe-641c-40bd-8588-a6241156f7b0","Type":"ContainerStarted","Data":"17141695b02291d3a4daf9e2aa9d8570feec818030c5ac60d0ba8f650611d317"}
May 25 12:22:32.946328 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:22:32.945889    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-40bda-predictor-6664b4979f-g8jxd"
May 25 12:22:32.947744 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:22:32.947712    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-40bda-predictor-776fc95cd4-dr29n" event={"ID":"7975c442-27c9-43e0-a7b5-5eba5e770eb3","Type":"ContainerStarted","Data":"e78747fa671379bda0badc747bf5aba3acb069b6f548eab73f9f26576427d7f1"}
May 25 12:22:32.947857 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:22:32.947745    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-40bda-predictor-776fc95cd4-dr29n" event={"ID":"7975c442-27c9-43e0-a7b5-5eba5e770eb3","Type":"ContainerStarted","Data":"26988f813bd21746bb89e4aa264d007defcf9fb5128c030a9e6b2a293f450082"}
May 25 12:22:32.947857 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:22:32.947762    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-40bda-predictor-776fc95cd4-dr29n" event={"ID":"7975c442-27c9-43e0-a7b5-5eba5e770eb3","Type":"ContainerStarted","Data":"36452567b8f9888733a43275332c57b822bfb19628f3f148a2b9249d0108594f"}
May 25 12:22:32.947951 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:22:32.947876    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-40bda-predictor-776fc95cd4-dr29n"
May 25 12:22:32.947951 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:22:32.947902    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-40bda-predictor-776fc95cd4-dr29n"
May 25 12:22:32.949320 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:22:32.949296    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-40bda-predictor-776fc95cd4-dr29n" podUID="7975c442-27c9-43e0-a7b5-5eba5e770eb3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
May 25 12:22:32.965818 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:22:32.965766    2549 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-40bda-predictor-6664b4979f-g8jxd" podStartSLOduration=1.965752749 podStartE2EDuration="1.965752749s" podCreationTimestamp="2026-05-25 12:22:31 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 12:22:32.964371839 +0000 UTC m=+556.680187126" watchObservedRunningTime="2026-05-25 12:22:32.965752749 +0000 UTC m=+556.681568040"
May 25 12:22:32.995329 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:22:32.995260    2549 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-40bda-predictor-776fc95cd4-dr29n" podStartSLOduration=1.995239005 podStartE2EDuration="1.995239005s" podCreationTimestamp="2026-05-25 12:22:31 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 12:22:32.985597265 +0000 UTC m=+556.701412553" watchObservedRunningTime="2026-05-25 12:22:32.995239005 +0000 UTC m=+556.711054293"
May 25 12:22:33.951534 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:22:33.951484    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-40bda-predictor-776fc95cd4-dr29n" podUID="7975c442-27c9-43e0-a7b5-5eba5e770eb3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
May 25 12:22:33.951534 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:22:33.951510    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-40bda-predictor-6664b4979f-g8jxd"
May 25 12:22:33.952693 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:22:33.952668    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-40bda-predictor-6664b4979f-g8jxd" podUID="771a87fe-641c-40bd-8588-a6241156f7b0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
May 25 12:22:34.437871 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:22:34.437848    2549 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-1742e-predictor-5bf4dfd9dd-8j4zf"
May 25 12:22:34.497224 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:22:34.497179    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"success-200-isvc-1742e-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d1e41943-5b0a-4496-ba74-97cc2c381802-success-200-isvc-1742e-kube-rbac-proxy-sar-config\") pod \"d1e41943-5b0a-4496-ba74-97cc2c381802\" (UID: \"d1e41943-5b0a-4496-ba74-97cc2c381802\") "
May 25 12:22:34.497356 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:22:34.497296    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d1e41943-5b0a-4496-ba74-97cc2c381802-proxy-tls\") pod \"d1e41943-5b0a-4496-ba74-97cc2c381802\" (UID: \"d1e41943-5b0a-4496-ba74-97cc2c381802\") "
May 25 12:22:34.497356 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:22:34.497334    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9krzc\" (UniqueName: \"kubernetes.io/projected/d1e41943-5b0a-4496-ba74-97cc2c381802-kube-api-access-9krzc\") pod \"d1e41943-5b0a-4496-ba74-97cc2c381802\" (UID: \"d1e41943-5b0a-4496-ba74-97cc2c381802\") "
May 25 12:22:34.497511 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:22:34.497487    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1e41943-5b0a-4496-ba74-97cc2c381802-success-200-isvc-1742e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "success-200-isvc-1742e-kube-rbac-proxy-sar-config") pod "d1e41943-5b0a-4496-ba74-97cc2c381802" (UID: "d1e41943-5b0a-4496-ba74-97cc2c381802"). InnerVolumeSpecName "success-200-isvc-1742e-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 12:22:34.497608 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:22:34.497593    2549 reconciler_common.go:299] "Volume detached for volume \"success-200-isvc-1742e-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d1e41943-5b0a-4496-ba74-97cc2c381802-success-200-isvc-1742e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
May 25 12:22:34.499565 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:22:34.499540    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1e41943-5b0a-4496-ba74-97cc2c381802-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d1e41943-5b0a-4496-ba74-97cc2c381802" (UID: "d1e41943-5b0a-4496-ba74-97cc2c381802"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 12:22:34.499655 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:22:34.499612    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1e41943-5b0a-4496-ba74-97cc2c381802-kube-api-access-9krzc" (OuterVolumeSpecName: "kube-api-access-9krzc") pod "d1e41943-5b0a-4496-ba74-97cc2c381802" (UID: "d1e41943-5b0a-4496-ba74-97cc2c381802"). InnerVolumeSpecName "kube-api-access-9krzc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 12:22:34.598405 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:22:34.598372    2549 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d1e41943-5b0a-4496-ba74-97cc2c381802-proxy-tls\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
May 25 12:22:34.598405 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:22:34.598402    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9krzc\" (UniqueName: \"kubernetes.io/projected/d1e41943-5b0a-4496-ba74-97cc2c381802-kube-api-access-9krzc\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
May 25 12:22:34.670396 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:22:34.670369    2549 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-1742e-predictor-5764c6c5cf-6wk56"
May 25 12:22:34.699469 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:22:34.699437    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dbeb1945-c642-49c8-ab81-19aeffc568d3-proxy-tls\") pod \"dbeb1945-c642-49c8-ab81-19aeffc568d3\" (UID: \"dbeb1945-c642-49c8-ab81-19aeffc568d3\") "
May 25 12:22:34.699611 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:22:34.699486    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"error-404-isvc-1742e-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dbeb1945-c642-49c8-ab81-19aeffc568d3-error-404-isvc-1742e-kube-rbac-proxy-sar-config\") pod \"dbeb1945-c642-49c8-ab81-19aeffc568d3\" (UID: \"dbeb1945-c642-49c8-ab81-19aeffc568d3\") "
May 25 12:22:34.699611 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:22:34.699533    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h6pd4\" (UniqueName: \"kubernetes.io/projected/dbeb1945-c642-49c8-ab81-19aeffc568d3-kube-api-access-h6pd4\") pod \"dbeb1945-c642-49c8-ab81-19aeffc568d3\" (UID: \"dbeb1945-c642-49c8-ab81-19aeffc568d3\") "
May 25 12:22:34.699925 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:22:34.699895    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbeb1945-c642-49c8-ab81-19aeffc568d3-error-404-isvc-1742e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "error-404-isvc-1742e-kube-rbac-proxy-sar-config") pod "dbeb1945-c642-49c8-ab81-19aeffc568d3" (UID: "dbeb1945-c642-49c8-ab81-19aeffc568d3"). InnerVolumeSpecName "error-404-isvc-1742e-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 12:22:34.701541 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:22:34.701519    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dbeb1945-c642-49c8-ab81-19aeffc568d3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "dbeb1945-c642-49c8-ab81-19aeffc568d3" (UID: "dbeb1945-c642-49c8-ab81-19aeffc568d3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 12:22:34.701632 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:22:34.701556    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbeb1945-c642-49c8-ab81-19aeffc568d3-kube-api-access-h6pd4" (OuterVolumeSpecName: "kube-api-access-h6pd4") pod "dbeb1945-c642-49c8-ab81-19aeffc568d3" (UID: "dbeb1945-c642-49c8-ab81-19aeffc568d3"). InnerVolumeSpecName "kube-api-access-h6pd4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 12:22:34.800903 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:22:34.800859    2549 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dbeb1945-c642-49c8-ab81-19aeffc568d3-proxy-tls\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
May 25 12:22:34.800903 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:22:34.800896    2549 reconciler_common.go:299] "Volume detached for volume \"error-404-isvc-1742e-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dbeb1945-c642-49c8-ab81-19aeffc568d3-error-404-isvc-1742e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
May 25 12:22:34.800903 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:22:34.800906    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h6pd4\" (UniqueName: \"kubernetes.io/projected/dbeb1945-c642-49c8-ab81-19aeffc568d3-kube-api-access-h6pd4\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
May 25 12:22:34.955714 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:22:34.955679    2549 generic.go:358] "Generic (PLEG): container finished" podID="d1e41943-5b0a-4496-ba74-97cc2c381802" containerID="31877611702d518dbb0a94a91ed23914c7737e30b00b344990e3cdf622763444" exitCode=0
May 25 12:22:34.956135 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:22:34.955748    2549 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-1742e-predictor-5bf4dfd9dd-8j4zf"
May 25 12:22:34.956135 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:22:34.955763    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-1742e-predictor-5bf4dfd9dd-8j4zf" event={"ID":"d1e41943-5b0a-4496-ba74-97cc2c381802","Type":"ContainerDied","Data":"31877611702d518dbb0a94a91ed23914c7737e30b00b344990e3cdf622763444"}
May 25 12:22:34.956135 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:22:34.955802    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-1742e-predictor-5bf4dfd9dd-8j4zf" event={"ID":"d1e41943-5b0a-4496-ba74-97cc2c381802","Type":"ContainerDied","Data":"ab87bf0576f84990e8bf7dbf90acd933702f443eacf3ed3c871c5bab1924d7c9"}
May 25 12:22:34.956135 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:22:34.955822    2549 scope.go:117] "RemoveContainer" containerID="a496dd35397c877274a093011441e8747e1f833d098fbf071ba91b40cdc0b50a"
May 25 12:22:34.957364 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:22:34.957342    2549 generic.go:358] "Generic (PLEG): container finished" podID="dbeb1945-c642-49c8-ab81-19aeffc568d3" containerID="df75b6b4a3bd1bf2ccdb664cff7c045b0c1788fa7f32b7b8abf07f1a0ec8fe1e" exitCode=0
May 25 12:22:34.957482 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:22:34.957457    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-1742e-predictor-5764c6c5cf-6wk56" event={"ID":"dbeb1945-c642-49c8-ab81-19aeffc568d3","Type":"ContainerDied","Data":"df75b6b4a3bd1bf2ccdb664cff7c045b0c1788fa7f32b7b8abf07f1a0ec8fe1e"}
May 25 12:22:34.957543 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:22:34.957502    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-1742e-predictor-5764c6c5cf-6wk56" event={"ID":"dbeb1945-c642-49c8-ab81-19aeffc568d3","Type":"ContainerDied","Data":"ed69f8138523886cf2244cf63e0200cad3cfe0bd71e89f98bb9f180889e20bdf"}
May 25 12:22:34.957543 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:22:34.957533    2549 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-1742e-predictor-5764c6c5cf-6wk56"
May 25 12:22:34.958106 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:22:34.958080    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-40bda-predictor-6664b4979f-g8jxd" podUID="771a87fe-641c-40bd-8588-a6241156f7b0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
May 25 12:22:34.964292 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:22:34.964276    2549 scope.go:117] "RemoveContainer" containerID="31877611702d518dbb0a94a91ed23914c7737e30b00b344990e3cdf622763444"
May 25 12:22:34.971758 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:22:34.971741    2549 scope.go:117] "RemoveContainer" containerID="a496dd35397c877274a093011441e8747e1f833d098fbf071ba91b40cdc0b50a"
May 25 12:22:34.972024 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:22:34.971998    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a496dd35397c877274a093011441e8747e1f833d098fbf071ba91b40cdc0b50a\": container with ID starting with a496dd35397c877274a093011441e8747e1f833d098fbf071ba91b40cdc0b50a not found: ID does not exist" containerID="a496dd35397c877274a093011441e8747e1f833d098fbf071ba91b40cdc0b50a"
May 25 12:22:34.972136 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:22:34.972035    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a496dd35397c877274a093011441e8747e1f833d098fbf071ba91b40cdc0b50a"} err="failed to get container status \"a496dd35397c877274a093011441e8747e1f833d098fbf071ba91b40cdc0b50a\": rpc error: code = NotFound desc = could not find container \"a496dd35397c877274a093011441e8747e1f833d098fbf071ba91b40cdc0b50a\": container with ID starting with a496dd35397c877274a093011441e8747e1f833d098fbf071ba91b40cdc0b50a not found: ID does not exist"
May 25 12:22:34.972136 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:22:34.972054    2549 scope.go:117] "RemoveContainer" containerID="31877611702d518dbb0a94a91ed23914c7737e30b00b344990e3cdf622763444"
May 25 12:22:34.972356 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:22:34.972340    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31877611702d518dbb0a94a91ed23914c7737e30b00b344990e3cdf622763444\": container with ID starting with 31877611702d518dbb0a94a91ed23914c7737e30b00b344990e3cdf622763444 not found: ID does not exist" containerID="31877611702d518dbb0a94a91ed23914c7737e30b00b344990e3cdf622763444"
May 25 12:22:34.972405 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:22:34.972361    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31877611702d518dbb0a94a91ed23914c7737e30b00b344990e3cdf622763444"} err="failed to get container status \"31877611702d518dbb0a94a91ed23914c7737e30b00b344990e3cdf622763444\": rpc error: code = NotFound desc = could not find container \"31877611702d518dbb0a94a91ed23914c7737e30b00b344990e3cdf622763444\": container with ID starting with 31877611702d518dbb0a94a91ed23914c7737e30b00b344990e3cdf622763444 not found: ID does not exist"
May 25 12:22:34.972405 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:22:34.972376    2549 scope.go:117] "RemoveContainer" containerID="2fc13c2af4e674a2b1cca3810cad7241c299b627565937ff6cb219737fb75b48"
May 25 12:22:34.976871 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:22:34.976850    2549 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-1742e-predictor-5bf4dfd9dd-8j4zf"]
May 25 12:22:34.978732 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:22:34.978711    2549 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-1742e-predictor-5bf4dfd9dd-8j4zf"]
May 25 12:22:34.981147 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:22:34.981132    2549 scope.go:117] "RemoveContainer" containerID="df75b6b4a3bd1bf2ccdb664cff7c045b0c1788fa7f32b7b8abf07f1a0ec8fe1e"
May 25 12:22:34.989267 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:22:34.989246    2549 scope.go:117] "RemoveContainer" containerID="2fc13c2af4e674a2b1cca3810cad7241c299b627565937ff6cb219737fb75b48"
May 25 12:22:34.989581 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:22:34.989559    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2fc13c2af4e674a2b1cca3810cad7241c299b627565937ff6cb219737fb75b48\": container with ID starting with 2fc13c2af4e674a2b1cca3810cad7241c299b627565937ff6cb219737fb75b48 not found: ID does not exist" containerID="2fc13c2af4e674a2b1cca3810cad7241c299b627565937ff6cb219737fb75b48"
May 25 12:22:34.989680 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:22:34.989587    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2fc13c2af4e674a2b1cca3810cad7241c299b627565937ff6cb219737fb75b48"} err="failed to get container status \"2fc13c2af4e674a2b1cca3810cad7241c299b627565937ff6cb219737fb75b48\": rpc error: code = NotFound desc = could not find container \"2fc13c2af4e674a2b1cca3810cad7241c299b627565937ff6cb219737fb75b48\": container with ID starting with 2fc13c2af4e674a2b1cca3810cad7241c299b627565937ff6cb219737fb75b48 not found: ID does not exist"
May 25 12:22:34.989680 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:22:34.989606    2549 scope.go:117] "RemoveContainer" containerID="df75b6b4a3bd1bf2ccdb664cff7c045b0c1788fa7f32b7b8abf07f1a0ec8fe1e"
May 25 12:22:34.989882 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:22:34.989858    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df75b6b4a3bd1bf2ccdb664cff7c045b0c1788fa7f32b7b8abf07f1a0ec8fe1e\": container with ID starting with df75b6b4a3bd1bf2ccdb664cff7c045b0c1788fa7f32b7b8abf07f1a0ec8fe1e not found: ID does not exist" containerID="df75b6b4a3bd1bf2ccdb664cff7c045b0c1788fa7f32b7b8abf07f1a0ec8fe1e"
May 25 12:22:34.989949 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:22:34.989890    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df75b6b4a3bd1bf2ccdb664cff7c045b0c1788fa7f32b7b8abf07f1a0ec8fe1e"} err="failed to get container status \"df75b6b4a3bd1bf2ccdb664cff7c045b0c1788fa7f32b7b8abf07f1a0ec8fe1e\": rpc error: code = NotFound desc = could not find container \"df75b6b4a3bd1bf2ccdb664cff7c045b0c1788fa7f32b7b8abf07f1a0ec8fe1e\": container with ID starting with df75b6b4a3bd1bf2ccdb664cff7c045b0c1788fa7f32b7b8abf07f1a0ec8fe1e not found: ID does not exist"
May 25 12:22:34.992972 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:22:34.992946    2549 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-1742e-predictor-5764c6c5cf-6wk56"]
May 25 12:22:34.997458 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:22:34.997438    2549 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-1742e-predictor-5764c6c5cf-6wk56"]
May 25 12:22:36.848514 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:22:36.848484    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d1e41943-5b0a-4496-ba74-97cc2c381802" path="/var/lib/kubelet/pods/d1e41943-5b0a-4496-ba74-97cc2c381802/volumes"
May 25 12:22:36.848895 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:22:36.848881    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dbeb1945-c642-49c8-ab81-19aeffc568d3" path="/var/lib/kubelet/pods/dbeb1945-c642-49c8-ab81-19aeffc568d3/volumes"
May 25 12:22:38.956338 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:22:38.956308    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-40bda-predictor-776fc95cd4-dr29n"
May 25 12:22:38.956846 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:22:38.956816    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-40bda-predictor-776fc95cd4-dr29n" podUID="7975c442-27c9-43e0-a7b5-5eba5e770eb3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
May 25 12:22:39.963272 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:22:39.963241    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-40bda-predictor-6664b4979f-g8jxd"
May 25 12:22:39.963813 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:22:39.963783    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-40bda-predictor-6664b4979f-g8jxd" podUID="771a87fe-641c-40bd-8588-a6241156f7b0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
May 25 12:22:48.957855 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:22:48.957813    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-40bda-predictor-776fc95cd4-dr29n" podUID="7975c442-27c9-43e0-a7b5-5eba5e770eb3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
May 25 12:22:49.964678 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:22:49.964637    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-40bda-predictor-6664b4979f-g8jxd" podUID="771a87fe-641c-40bd-8588-a6241156f7b0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
May 25 12:22:58.957650 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:22:58.957605    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-40bda-predictor-776fc95cd4-dr29n" podUID="7975c442-27c9-43e0-a7b5-5eba5e770eb3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
May 25 12:22:59.964273 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:22:59.964230    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-40bda-predictor-6664b4979f-g8jxd" podUID="771a87fe-641c-40bd-8588-a6241156f7b0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
May 25 12:23:01.059441 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:23:01.059401    2549 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-833d6-predictor-65cfc8fdbb-xw7b6"]
May 25 12:23:01.060006 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:23:01.059989    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dbeb1945-c642-49c8-ab81-19aeffc568d3" containerName="kserve-container"
May 25 12:23:01.060054 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:23:01.060008    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="dbeb1945-c642-49c8-ab81-19aeffc568d3" containerName="kserve-container"
May 25 12:23:01.060054 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:23:01.060034    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1e41943-5b0a-4496-ba74-97cc2c381802" containerName="kserve-container"
May 25 12:23:01.060054 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:23:01.060043    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1e41943-5b0a-4496-ba74-97cc2c381802" containerName="kserve-container"
May 25 12:23:01.060153 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:23:01.060058    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dbeb1945-c642-49c8-ab81-19aeffc568d3" containerName="kube-rbac-proxy"
May 25 12:23:01.060153 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:23:01.060067    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="dbeb1945-c642-49c8-ab81-19aeffc568d3" containerName="kube-rbac-proxy"
May 25 12:23:01.060153 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:23:01.060081    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1e41943-5b0a-4496-ba74-97cc2c381802" containerName="kube-rbac-proxy"
May 25 12:23:01.060153 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:23:01.060091    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1e41943-5b0a-4496-ba74-97cc2c381802" containerName="kube-rbac-proxy"
May 25 12:23:01.060307 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:23:01.060169    2549 memory_manager.go:356] "RemoveStaleState removing state" podUID="d1e41943-5b0a-4496-ba74-97cc2c381802" containerName="kserve-container"
May 25 12:23:01.060307 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:23:01.060183    2549 memory_manager.go:356] "RemoveStaleState removing state" podUID="dbeb1945-c642-49c8-ab81-19aeffc568d3" containerName="kserve-container"
May 25 12:23:01.060307 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:23:01.060213    2549 memory_manager.go:356] "RemoveStaleState removing state" podUID="dbeb1945-c642-49c8-ab81-19aeffc568d3" containerName="kube-rbac-proxy"
May 25 12:23:01.060307 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:23:01.060224    2549 memory_manager.go:356] "RemoveStaleState removing state" podUID="d1e41943-5b0a-4496-ba74-97cc2c381802" containerName="kube-rbac-proxy"
May 25 12:23:01.064914 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:23:01.064895    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-833d6-predictor-65cfc8fdbb-xw7b6"
May 25 12:23:01.067801 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:23:01.067777    2549 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-833d6-kube-rbac-proxy-sar-config\""
May 25 12:23:01.067898 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:23:01.067789    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-833d6-predictor-serving-cert\""
May 25 12:23:01.074264 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:23:01.074242    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-833d6-predictor-65cfc8fdbb-xw7b6"]
May 25 12:23:01.144937 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:23:01.144893    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffpml\" (UniqueName: \"kubernetes.io/projected/6264ef64-b6b2-43bd-bcfb-f281e7205194-kube-api-access-ffpml\") pod \"success-200-isvc-833d6-predictor-65cfc8fdbb-xw7b6\" (UID: \"6264ef64-b6b2-43bd-bcfb-f281e7205194\") " pod="kserve-ci-e2e-test/success-200-isvc-833d6-predictor-65cfc8fdbb-xw7b6"
May 25 12:23:01.145121 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:23:01.144971    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"success-200-isvc-833d6-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6264ef64-b6b2-43bd-bcfb-f281e7205194-success-200-isvc-833d6-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-833d6-predictor-65cfc8fdbb-xw7b6\" (UID: \"6264ef64-b6b2-43bd-bcfb-f281e7205194\") " pod="kserve-ci-e2e-test/success-200-isvc-833d6-predictor-65cfc8fdbb-xw7b6"
May 25 12:23:01.145220 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:23:01.145135    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6264ef64-b6b2-43bd-bcfb-f281e7205194-proxy-tls\") pod \"success-200-isvc-833d6-predictor-65cfc8fdbb-xw7b6\" (UID: \"6264ef64-b6b2-43bd-bcfb-f281e7205194\") " pod="kserve-ci-e2e-test/success-200-isvc-833d6-predictor-65cfc8fdbb-xw7b6"
May 25 12:23:01.163178 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:23:01.163140    2549 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-833d6-predictor-7dfb76b5f7-8rzb8"]
May 25 12:23:01.167147 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:23:01.167126    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-833d6-predictor-7dfb76b5f7-8rzb8"
May 25 12:23:01.169726 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:23:01.169706    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-833d6-predictor-serving-cert\""
May 25 12:23:01.169726 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:23:01.169715    2549 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-833d6-kube-rbac-proxy-sar-config\""
May 25 12:23:01.178762 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:23:01.178734    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-833d6-predictor-7dfb76b5f7-8rzb8"]
May 25 12:23:01.246620 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:23:01.246569    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"success-200-isvc-833d6-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6264ef64-b6b2-43bd-bcfb-f281e7205194-success-200-isvc-833d6-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-833d6-predictor-65cfc8fdbb-xw7b6\" (UID: \"6264ef64-b6b2-43bd-bcfb-f281e7205194\") " pod="kserve-ci-e2e-test/success-200-isvc-833d6-predictor-65cfc8fdbb-xw7b6"
May 25 12:23:01.246876 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:23:01.246643    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/40582ba8-ab3b-4046-985c-bcfb29eb0b82-proxy-tls\") pod \"error-404-isvc-833d6-predictor-7dfb76b5f7-8rzb8\" (UID: \"40582ba8-ab3b-4046-985c-bcfb29eb0b82\") " pod="kserve-ci-e2e-test/error-404-isvc-833d6-predictor-7dfb76b5f7-8rzb8"
May 25 12:23:01.246876 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:23:01.246704    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"error-404-isvc-833d6-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/40582ba8-ab3b-4046-985c-bcfb29eb0b82-error-404-isvc-833d6-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-833d6-predictor-7dfb76b5f7-8rzb8\" (UID: \"40582ba8-ab3b-4046-985c-bcfb29eb0b82\") " pod="kserve-ci-e2e-test/error-404-isvc-833d6-predictor-7dfb76b5f7-8rzb8"
May 25 12:23:01.246876 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:23:01.246804    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6264ef64-b6b2-43bd-bcfb-f281e7205194-proxy-tls\") pod \"success-200-isvc-833d6-predictor-65cfc8fdbb-xw7b6\" (UID: \"6264ef64-b6b2-43bd-bcfb-f281e7205194\") " pod="kserve-ci-e2e-test/success-200-isvc-833d6-predictor-65cfc8fdbb-xw7b6"
May 25 12:23:01.246876 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:23:01.246856    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gl6wb\" (UniqueName: \"kubernetes.io/projected/40582ba8-ab3b-4046-985c-bcfb29eb0b82-kube-api-access-gl6wb\") pod \"error-404-isvc-833d6-predictor-7dfb76b5f7-8rzb8\" (UID: \"40582ba8-ab3b-4046-985c-bcfb29eb0b82\") " pod="kserve-ci-e2e-test/error-404-isvc-833d6-predictor-7dfb76b5f7-8rzb8"
May 25 12:23:01.247100 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:23:01.246918    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-ffpml\" (UniqueName: \"kubernetes.io/projected/6264ef64-b6b2-43bd-bcfb-f281e7205194-kube-api-access-ffpml\") pod \"success-200-isvc-833d6-predictor-65cfc8fdbb-xw7b6\" (UID: \"6264ef64-b6b2-43bd-bcfb-f281e7205194\") " pod="kserve-ci-e2e-test/success-200-isvc-833d6-predictor-65cfc8fdbb-xw7b6"
May 25 12:23:01.247100 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:23:01.246921    2549 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-833d6-predictor-serving-cert: secret "success-200-isvc-833d6-predictor-serving-cert" not found
May 25 12:23:01.247100 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:23:01.247073    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6264ef64-b6b2-43bd-bcfb-f281e7205194-proxy-tls podName:6264ef64-b6b2-43bd-bcfb-f281e7205194 nodeName:}" failed. No retries permitted until 2026-05-25 12:23:01.74705329 +0000 UTC m=+585.462868569 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/6264ef64-b6b2-43bd-bcfb-f281e7205194-proxy-tls") pod "success-200-isvc-833d6-predictor-65cfc8fdbb-xw7b6" (UID: "6264ef64-b6b2-43bd-bcfb-f281e7205194") : secret "success-200-isvc-833d6-predictor-serving-cert" not found
May 25 12:23:01.247429 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:23:01.247357    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"success-200-isvc-833d6-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6264ef64-b6b2-43bd-bcfb-f281e7205194-success-200-isvc-833d6-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-833d6-predictor-65cfc8fdbb-xw7b6\" (UID: \"6264ef64-b6b2-43bd-bcfb-f281e7205194\") " pod="kserve-ci-e2e-test/success-200-isvc-833d6-predictor-65cfc8fdbb-xw7b6"
May 25 12:23:01.256499 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:23:01.256468    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffpml\" (UniqueName: \"kubernetes.io/projected/6264ef64-b6b2-43bd-bcfb-f281e7205194-kube-api-access-ffpml\") pod \"success-200-isvc-833d6-predictor-65cfc8fdbb-xw7b6\" (UID: \"6264ef64-b6b2-43bd-bcfb-f281e7205194\") " pod="kserve-ci-e2e-test/success-200-isvc-833d6-predictor-65cfc8fdbb-xw7b6"
May 25 12:23:01.347489 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:23:01.347461    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-gl6wb\" (UniqueName: \"kubernetes.io/projected/40582ba8-ab3b-4046-985c-bcfb29eb0b82-kube-api-access-gl6wb\") pod \"error-404-isvc-833d6-predictor-7dfb76b5f7-8rzb8\" (UID: \"40582ba8-ab3b-4046-985c-bcfb29eb0b82\") " pod="kserve-ci-e2e-test/error-404-isvc-833d6-predictor-7dfb76b5f7-8rzb8"
May 25 12:23:01.347650 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:23:01.347548    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/40582ba8-ab3b-4046-985c-bcfb29eb0b82-proxy-tls\") pod \"error-404-isvc-833d6-predictor-7dfb76b5f7-8rzb8\" (UID: \"40582ba8-ab3b-4046-985c-bcfb29eb0b82\") " pod="kserve-ci-e2e-test/error-404-isvc-833d6-predictor-7dfb76b5f7-8rzb8"
May 25 12:23:01.347650 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:23:01.347590    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"error-404-isvc-833d6-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/40582ba8-ab3b-4046-985c-bcfb29eb0b82-error-404-isvc-833d6-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-833d6-predictor-7dfb76b5f7-8rzb8\" (UID: \"40582ba8-ab3b-4046-985c-bcfb29eb0b82\") " pod="kserve-ci-e2e-test/error-404-isvc-833d6-predictor-7dfb76b5f7-8rzb8"
May 25 12:23:01.347734 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:23:01.347705    2549 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-833d6-predictor-serving-cert: secret "error-404-isvc-833d6-predictor-serving-cert" not found
May 25 12:23:01.347821 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:23:01.347810    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40582ba8-ab3b-4046-985c-bcfb29eb0b82-proxy-tls podName:40582ba8-ab3b-4046-985c-bcfb29eb0b82 nodeName:}" failed. No retries permitted until 2026-05-25 12:23:01.84776466 +0000 UTC m=+585.563579928 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/40582ba8-ab3b-4046-985c-bcfb29eb0b82-proxy-tls") pod "error-404-isvc-833d6-predictor-7dfb76b5f7-8rzb8" (UID: "40582ba8-ab3b-4046-985c-bcfb29eb0b82") : secret "error-404-isvc-833d6-predictor-serving-cert" not found
May 25 12:23:01.348180 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:23:01.348162    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"error-404-isvc-833d6-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/40582ba8-ab3b-4046-985c-bcfb29eb0b82-error-404-isvc-833d6-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-833d6-predictor-7dfb76b5f7-8rzb8\" (UID: \"40582ba8-ab3b-4046-985c-bcfb29eb0b82\") " pod="kserve-ci-e2e-test/error-404-isvc-833d6-predictor-7dfb76b5f7-8rzb8"
May 25 12:23:01.356422 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:23:01.356394    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gl6wb\" (UniqueName: \"kubernetes.io/projected/40582ba8-ab3b-4046-985c-bcfb29eb0b82-kube-api-access-gl6wb\") pod \"error-404-isvc-833d6-predictor-7dfb76b5f7-8rzb8\" (UID: \"40582ba8-ab3b-4046-985c-bcfb29eb0b82\") " pod="kserve-ci-e2e-test/error-404-isvc-833d6-predictor-7dfb76b5f7-8rzb8"
May 25 12:23:01.751497 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:23:01.751402    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6264ef64-b6b2-43bd-bcfb-f281e7205194-proxy-tls\") pod \"success-200-isvc-833d6-predictor-65cfc8fdbb-xw7b6\" (UID: \"6264ef64-b6b2-43bd-bcfb-f281e7205194\") " pod="kserve-ci-e2e-test/success-200-isvc-833d6-predictor-65cfc8fdbb-xw7b6"
May 25 12:23:01.753835 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:23:01.753804    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6264ef64-b6b2-43bd-bcfb-f281e7205194-proxy-tls\") pod \"success-200-isvc-833d6-predictor-65cfc8fdbb-xw7b6\" (UID: \"6264ef64-b6b2-43bd-bcfb-f281e7205194\") " pod="kserve-ci-e2e-test/success-200-isvc-833d6-predictor-65cfc8fdbb-xw7b6"
May 25 12:23:01.852086 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:23:01.852048    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/40582ba8-ab3b-4046-985c-bcfb29eb0b82-proxy-tls\") pod \"error-404-isvc-833d6-predictor-7dfb76b5f7-8rzb8\" (UID: \"40582ba8-ab3b-4046-985c-bcfb29eb0b82\") " pod="kserve-ci-e2e-test/error-404-isvc-833d6-predictor-7dfb76b5f7-8rzb8"
May 25 12:23:01.854519 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:23:01.854496    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/40582ba8-ab3b-4046-985c-bcfb29eb0b82-proxy-tls\") pod \"error-404-isvc-833d6-predictor-7dfb76b5f7-8rzb8\" (UID: \"40582ba8-ab3b-4046-985c-bcfb29eb0b82\") " pod="kserve-ci-e2e-test/error-404-isvc-833d6-predictor-7dfb76b5f7-8rzb8"
May 25 12:23:01.978668 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:23:01.978624    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-833d6-predictor-65cfc8fdbb-xw7b6"
May 25 12:23:02.079864 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:23:02.079828    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-833d6-predictor-7dfb76b5f7-8rzb8"
May 25 12:23:02.120119 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:23:02.120009    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-833d6-predictor-65cfc8fdbb-xw7b6"]
May 25 12:23:02.122996 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:23:02.122959    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6264ef64_b6b2_43bd_bcfb_f281e7205194.slice/crio-3d4ffa25d3755171c045fad2d65557223c0451a635bf5440fbed8ce47805feec WatchSource:0}: Error finding container 3d4ffa25d3755171c045fad2d65557223c0451a635bf5440fbed8ce47805feec: Status 404 returned error can't find the container with id 3d4ffa25d3755171c045fad2d65557223c0451a635bf5440fbed8ce47805feec
May 25 12:23:02.226730 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:23:02.226703    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-833d6-predictor-7dfb76b5f7-8rzb8"]
May 25 12:23:02.227941 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:23:02.227914    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40582ba8_ab3b_4046_985c_bcfb29eb0b82.slice/crio-5492c3b492410366b30ccbd88b777d3295f602c71595882d6363395a0216da21 WatchSource:0}: Error finding container 5492c3b492410366b30ccbd88b777d3295f602c71595882d6363395a0216da21: Status 404 returned error can't find the container with id 5492c3b492410366b30ccbd88b777d3295f602c71595882d6363395a0216da21
May 25 12:23:03.063502 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:23:03.063463    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-833d6-predictor-65cfc8fdbb-xw7b6" event={"ID":"6264ef64-b6b2-43bd-bcfb-f281e7205194","Type":"ContainerStarted","Data":"70ae413c651d2a45a21f1ae79d8fa1013f9ba89f9aa51d29a41d72dde95b522c"}
May 25 12:23:03.063502 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:23:03.063506    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-833d6-predictor-65cfc8fdbb-xw7b6" event={"ID":"6264ef64-b6b2-43bd-bcfb-f281e7205194","Type":"ContainerStarted","Data":"faa6f5119adba2a116a1c635be6ab083bb56996ace4397c26b6a8b3c1c8ae2ba"}
May 25 12:23:03.063780 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:23:03.063529    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-833d6-predictor-65cfc8fdbb-xw7b6"
May 25 12:23:03.063780 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:23:03.063540    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-833d6-predictor-65cfc8fdbb-xw7b6" event={"ID":"6264ef64-b6b2-43bd-bcfb-f281e7205194","Type":"ContainerStarted","Data":"3d4ffa25d3755171c045fad2d65557223c0451a635bf5440fbed8ce47805feec"}
May 25 12:23:03.063780 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:23:03.063573    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-833d6-predictor-65cfc8fdbb-xw7b6"
May 25 12:23:03.064874 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:23:03.064848    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-833d6-predictor-65cfc8fdbb-xw7b6" podUID="6264ef64-b6b2-43bd-bcfb-f281e7205194" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
May 25 12:23:03.065151 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:23:03.065132    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-833d6-predictor-7dfb76b5f7-8rzb8" event={"ID":"40582ba8-ab3b-4046-985c-bcfb29eb0b82","Type":"ContainerStarted","Data":"29f4bcbe01f5e4f8d92e6577bb108f04749a8fa53c73783930db5dc1e1da6340"}
May 25 12:23:03.065218 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:23:03.065158    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-833d6-predictor-7dfb76b5f7-8rzb8" event={"ID":"40582ba8-ab3b-4046-985c-bcfb29eb0b82","Type":"ContainerStarted","Data":"a425f8bbcb74f080c3fcf425bafc40db40ec3a200c76c9119be0657bcdd53c40"}
May 25 12:23:03.065218 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:23:03.065169    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-833d6-predictor-7dfb76b5f7-8rzb8" event={"ID":"40582ba8-ab3b-4046-985c-bcfb29eb0b82","Type":"ContainerStarted","Data":"5492c3b492410366b30ccbd88b777d3295f602c71595882d6363395a0216da21"}
May 25 12:23:03.065319 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:23:03.065302    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-833d6-predictor-7dfb76b5f7-8rzb8"
May 25 12:23:03.086659 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:23:03.086603    2549 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-833d6-predictor-65cfc8fdbb-xw7b6" podStartSLOduration=2.086581799 podStartE2EDuration="2.086581799s" podCreationTimestamp="2026-05-25 12:23:01 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 12:23:03.084462379 +0000 UTC m=+586.800277667" watchObservedRunningTime="2026-05-25 12:23:03.086581799 +0000 UTC m=+586.802397086"
May 25 12:23:03.109741 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:23:03.109694    2549 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-833d6-predictor-7dfb76b5f7-8rzb8" podStartSLOduration=2.109679377 podStartE2EDuration="2.109679377s" podCreationTimestamp="2026-05-25 12:23:01 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 12:23:03.107939889 +0000 UTC m=+586.823755177" watchObservedRunningTime="2026-05-25 12:23:03.109679377 +0000 UTC m=+586.825494664"
May 25 12:23:04.069322 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:23:04.069292    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-833d6-predictor-7dfb76b5f7-8rzb8"
May 25 12:23:04.069504 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:23:04.069357    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-833d6-predictor-65cfc8fdbb-xw7b6" podUID="6264ef64-b6b2-43bd-bcfb-f281e7205194" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
May 25 12:23:04.070548 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:23:04.070520    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-833d6-predictor-7dfb76b5f7-8rzb8" podUID="40582ba8-ab3b-4046-985c-bcfb29eb0b82" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
May 25 12:23:05.073299 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:23:05.073256    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-833d6-predictor-7dfb76b5f7-8rzb8" podUID="40582ba8-ab3b-4046-985c-bcfb29eb0b82" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
May 25 12:23:08.957814 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:23:08.957774    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-40bda-predictor-776fc95cd4-dr29n" podUID="7975c442-27c9-43e0-a7b5-5eba5e770eb3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
May 25 12:23:09.075070 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:23:09.075039    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-833d6-predictor-65cfc8fdbb-xw7b6"
May 25 12:23:09.075569 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:23:09.075540    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-833d6-predictor-65cfc8fdbb-xw7b6" podUID="6264ef64-b6b2-43bd-bcfb-f281e7205194" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
May 25 12:23:09.964124 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:23:09.964083    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-40bda-predictor-6664b4979f-g8jxd" podUID="771a87fe-641c-40bd-8588-a6241156f7b0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
May 25 12:23:10.078757 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:23:10.078725    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-833d6-predictor-7dfb76b5f7-8rzb8"
May 25 12:23:10.079299 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:23:10.079271    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-833d6-predictor-7dfb76b5f7-8rzb8" podUID="40582ba8-ab3b-4046-985c-bcfb29eb0b82" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
May 25 12:23:18.958266 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:23:18.958231    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-40bda-predictor-776fc95cd4-dr29n"
May 25 12:23:19.075930 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:23:19.075891    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-833d6-predictor-65cfc8fdbb-xw7b6" podUID="6264ef64-b6b2-43bd-bcfb-f281e7205194" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
May 25 12:23:19.964777 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:23:19.964744    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-40bda-predictor-6664b4979f-g8jxd"
May 25 12:23:20.079308 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:23:20.079266    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-833d6-predictor-7dfb76b5f7-8rzb8" podUID="40582ba8-ab3b-4046-985c-bcfb29eb0b82" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
May 25 12:23:29.075583 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:23:29.075538    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-833d6-predictor-65cfc8fdbb-xw7b6" podUID="6264ef64-b6b2-43bd-bcfb-f281e7205194" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
May 25 12:23:30.080257 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:23:30.080192    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-833d6-predictor-7dfb76b5f7-8rzb8" podUID="40582ba8-ab3b-4046-985c-bcfb29eb0b82" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
May 25 12:23:39.076185 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:23:39.076102    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-833d6-predictor-65cfc8fdbb-xw7b6" podUID="6264ef64-b6b2-43bd-bcfb-f281e7205194" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
May 25 12:23:40.080020 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:23:40.079975    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-833d6-predictor-7dfb76b5f7-8rzb8" podUID="40582ba8-ab3b-4046-985c-bcfb29eb0b82" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
May 25 12:23:49.076338 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:23:49.076308    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-833d6-predictor-65cfc8fdbb-xw7b6"
May 25 12:23:50.080344 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:23:50.080317    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-833d6-predictor-7dfb76b5f7-8rzb8"
May 25 12:31:45.922090 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:31:45.922055    2549 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-40bda-predictor-776fc95cd4-dr29n"]
May 25 12:31:45.922711 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:31:45.922345    2549 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-40bda-predictor-776fc95cd4-dr29n" podUID="7975c442-27c9-43e0-a7b5-5eba5e770eb3" containerName="kserve-container" containerID="cri-o://26988f813bd21746bb89e4aa264d007defcf9fb5128c030a9e6b2a293f450082" gracePeriod=30
May 25 12:31:45.922711 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:31:45.922418    2549 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-40bda-predictor-776fc95cd4-dr29n" podUID="7975c442-27c9-43e0-a7b5-5eba5e770eb3" containerName="kube-rbac-proxy" containerID="cri-o://e78747fa671379bda0badc747bf5aba3acb069b6f548eab73f9f26576427d7f1" gracePeriod=30
May 25 12:31:45.994932 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:31:45.994902    2549 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-40bda-predictor-6664b4979f-g8jxd"]
May 25 12:31:45.995262 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:31:45.995231    2549 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-40bda-predictor-6664b4979f-g8jxd" podUID="771a87fe-641c-40bd-8588-a6241156f7b0" containerName="kube-rbac-proxy" containerID="cri-o://f0d95a203f3797358bfad4055a2dd004f37bc20c03377b430648647864437335" gracePeriod=30
May 25 12:31:45.995330 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:31:45.995311    2549 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-40bda-predictor-6664b4979f-g8jxd" podUID="771a87fe-641c-40bd-8588-a6241156f7b0" containerName="kserve-container" containerID="cri-o://f3a663c372c80352fb12e353d2d07a1103246f0ea31ed16abf773d60834a179e" gracePeriod=30
May 25 12:31:46.019037 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:31:46.019008    2549 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-bcff4-predictor-66c9fb5576-scwgx"]
May 25 12:31:46.023317 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:31:46.023294    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-bcff4-predictor-66c9fb5576-scwgx"
May 25 12:31:46.026946 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:31:46.026921    2549 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-bcff4-kube-rbac-proxy-sar-config\""
May 25 12:31:46.027189 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:31:46.027171    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-bcff4-predictor-serving-cert\""
May 25 12:31:46.045668 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:31:46.045634    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-bcff4-predictor-66c9fb5576-scwgx"]
May 25 12:31:46.091445 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:31:46.091413    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzswg\" (UniqueName: \"kubernetes.io/projected/81f25695-1e0c-4e1c-beda-e1d250bf364b-kube-api-access-bzswg\") pod \"success-200-isvc-bcff4-predictor-66c9fb5576-scwgx\" (UID: \"81f25695-1e0c-4e1c-beda-e1d250bf364b\") " pod="kserve-ci-e2e-test/success-200-isvc-bcff4-predictor-66c9fb5576-scwgx"
May 25 12:31:46.091613 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:31:46.091496    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/81f25695-1e0c-4e1c-beda-e1d250bf364b-proxy-tls\") pod \"success-200-isvc-bcff4-predictor-66c9fb5576-scwgx\" (UID: \"81f25695-1e0c-4e1c-beda-e1d250bf364b\") " pod="kserve-ci-e2e-test/success-200-isvc-bcff4-predictor-66c9fb5576-scwgx"
May 25 12:31:46.091613 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:31:46.091522    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"success-200-isvc-bcff4-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/81f25695-1e0c-4e1c-beda-e1d250bf364b-success-200-isvc-bcff4-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-bcff4-predictor-66c9fb5576-scwgx\" (UID: \"81f25695-1e0c-4e1c-beda-e1d250bf364b\") " pod="kserve-ci-e2e-test/success-200-isvc-bcff4-predictor-66c9fb5576-scwgx"
May 25 12:31:46.093073 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:31:46.093047    2549 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-bcff4-predictor-55cbf45559-tw9hg"]
May 25 12:31:46.097402 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:31:46.097385    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-bcff4-predictor-55cbf45559-tw9hg"
May 25 12:31:46.100697 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:31:46.100670    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-bcff4-predictor-serving-cert\""
May 25 12:31:46.101259 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:31:46.101191    2549 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-bcff4-kube-rbac-proxy-sar-config\""
May 25 12:31:46.106039 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:31:46.106018    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-bcff4-predictor-55cbf45559-tw9hg"]
May 25 12:31:46.192449 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:31:46.192353    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-bzswg\" (UniqueName: \"kubernetes.io/projected/81f25695-1e0c-4e1c-beda-e1d250bf364b-kube-api-access-bzswg\") pod \"success-200-isvc-bcff4-predictor-66c9fb5576-scwgx\" (UID: \"81f25695-1e0c-4e1c-beda-e1d250bf364b\") " pod="kserve-ci-e2e-test/success-200-isvc-bcff4-predictor-66c9fb5576-scwgx"
May 25 12:31:46.192617 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:31:46.192465    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/81f25695-1e0c-4e1c-beda-e1d250bf364b-proxy-tls\") pod \"success-200-isvc-bcff4-predictor-66c9fb5576-scwgx\" (UID: \"81f25695-1e0c-4e1c-beda-e1d250bf364b\") " pod="kserve-ci-e2e-test/success-200-isvc-bcff4-predictor-66c9fb5576-scwgx"
May 25 12:31:46.192617 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:31:46.192493    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"success-200-isvc-bcff4-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/81f25695-1e0c-4e1c-beda-e1d250bf364b-success-200-isvc-bcff4-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-bcff4-predictor-66c9fb5576-scwgx\" (UID: \"81f25695-1e0c-4e1c-beda-e1d250bf364b\") " pod="kserve-ci-e2e-test/success-200-isvc-bcff4-predictor-66c9fb5576-scwgx"
May 25 12:31:46.192617 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:31:46.192541    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2d880044-04f1-4518-a423-305351e6122d-proxy-tls\") pod \"error-404-isvc-bcff4-predictor-55cbf45559-tw9hg\" (UID: \"2d880044-04f1-4518-a423-305351e6122d\") " pod="kserve-ci-e2e-test/error-404-isvc-bcff4-predictor-55cbf45559-tw9hg"
May 25 12:31:46.192617 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:31:46.192565    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"error-404-isvc-bcff4-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2d880044-04f1-4518-a423-305351e6122d-error-404-isvc-bcff4-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-bcff4-predictor-55cbf45559-tw9hg\" (UID: \"2d880044-04f1-4518-a423-305351e6122d\") " pod="kserve-ci-e2e-test/error-404-isvc-bcff4-predictor-55cbf45559-tw9hg"
May 25 12:31:46.192837 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:31:46.192662    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vh7ss\" (UniqueName: \"kubernetes.io/projected/2d880044-04f1-4518-a423-305351e6122d-kube-api-access-vh7ss\") pod \"error-404-isvc-bcff4-predictor-55cbf45559-tw9hg\" (UID: \"2d880044-04f1-4518-a423-305351e6122d\") " pod="kserve-ci-e2e-test/error-404-isvc-bcff4-predictor-55cbf45559-tw9hg"
May 25 12:31:46.193521 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:31:46.193491    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"success-200-isvc-bcff4-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/81f25695-1e0c-4e1c-beda-e1d250bf364b-success-200-isvc-bcff4-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-bcff4-predictor-66c9fb5576-scwgx\" (UID: \"81f25695-1e0c-4e1c-beda-e1d250bf364b\") " pod="kserve-ci-e2e-test/success-200-isvc-bcff4-predictor-66c9fb5576-scwgx"
May 25 12:31:46.195156 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:31:46.195137    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/81f25695-1e0c-4e1c-beda-e1d250bf364b-proxy-tls\") pod \"success-200-isvc-bcff4-predictor-66c9fb5576-scwgx\" (UID: \"81f25695-1e0c-4e1c-beda-e1d250bf364b\") " pod="kserve-ci-e2e-test/success-200-isvc-bcff4-predictor-66c9fb5576-scwgx"
May 25 12:31:46.208362 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:31:46.208340    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzswg\" (UniqueName: \"kubernetes.io/projected/81f25695-1e0c-4e1c-beda-e1d250bf364b-kube-api-access-bzswg\") pod \"success-200-isvc-bcff4-predictor-66c9fb5576-scwgx\" (UID: \"81f25695-1e0c-4e1c-beda-e1d250bf364b\") " pod="kserve-ci-e2e-test/success-200-isvc-bcff4-predictor-66c9fb5576-scwgx"
May 25 12:31:46.293978 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:31:46.293938    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-vh7ss\" (UniqueName: \"kubernetes.io/projected/2d880044-04f1-4518-a423-305351e6122d-kube-api-access-vh7ss\") pod \"error-404-isvc-bcff4-predictor-55cbf45559-tw9hg\" (UID: \"2d880044-04f1-4518-a423-305351e6122d\") " pod="kserve-ci-e2e-test/error-404-isvc-bcff4-predictor-55cbf45559-tw9hg"
May 25 12:31:46.294135 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:31:46.294027    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2d880044-04f1-4518-a423-305351e6122d-proxy-tls\") pod \"error-404-isvc-bcff4-predictor-55cbf45559-tw9hg\" (UID: \"2d880044-04f1-4518-a423-305351e6122d\") " pod="kserve-ci-e2e-test/error-404-isvc-bcff4-predictor-55cbf45559-tw9hg"
May 25 12:31:46.294135 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:31:46.294050    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"error-404-isvc-bcff4-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2d880044-04f1-4518-a423-305351e6122d-error-404-isvc-bcff4-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-bcff4-predictor-55cbf45559-tw9hg\" (UID: \"2d880044-04f1-4518-a423-305351e6122d\") " pod="kserve-ci-e2e-test/error-404-isvc-bcff4-predictor-55cbf45559-tw9hg"
May 25 12:31:46.294255 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:31:46.294163    2549 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-bcff4-predictor-serving-cert: secret "error-404-isvc-bcff4-predictor-serving-cert" not found
May 25 12:31:46.294255 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:31:46.294244    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2d880044-04f1-4518-a423-305351e6122d-proxy-tls podName:2d880044-04f1-4518-a423-305351e6122d nodeName:}" failed. No retries permitted until 2026-05-25 12:31:46.79422341 +0000 UTC m=+1110.510038692 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/2d880044-04f1-4518-a423-305351e6122d-proxy-tls") pod "error-404-isvc-bcff4-predictor-55cbf45559-tw9hg" (UID: "2d880044-04f1-4518-a423-305351e6122d") : secret "error-404-isvc-bcff4-predictor-serving-cert" not found
May 25 12:31:46.294662 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:31:46.294644    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"error-404-isvc-bcff4-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2d880044-04f1-4518-a423-305351e6122d-error-404-isvc-bcff4-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-bcff4-predictor-55cbf45559-tw9hg\" (UID: \"2d880044-04f1-4518-a423-305351e6122d\") " pod="kserve-ci-e2e-test/error-404-isvc-bcff4-predictor-55cbf45559-tw9hg"
May 25 12:31:46.304220 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:31:46.304184    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vh7ss\" (UniqueName: \"kubernetes.io/projected/2d880044-04f1-4518-a423-305351e6122d-kube-api-access-vh7ss\") pod \"error-404-isvc-bcff4-predictor-55cbf45559-tw9hg\" (UID: \"2d880044-04f1-4518-a423-305351e6122d\") " pod="kserve-ci-e2e-test/error-404-isvc-bcff4-predictor-55cbf45559-tw9hg"
May 25 12:31:46.337094 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:31:46.337068    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-bcff4-predictor-66c9fb5576-scwgx"
May 25 12:31:46.490997 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:31:46.490957    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-bcff4-predictor-66c9fb5576-scwgx"]
May 25 12:31:46.493688 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:31:46.493654    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81f25695_1e0c_4e1c_beda_e1d250bf364b.slice/crio-63d5166f345ec78cae7240c5adf87f9497fc784892b8e9a1ed8c01457f056ee6 WatchSource:0}: Error finding container 63d5166f345ec78cae7240c5adf87f9497fc784892b8e9a1ed8c01457f056ee6: Status 404 returned error can't find the container with id 63d5166f345ec78cae7240c5adf87f9497fc784892b8e9a1ed8c01457f056ee6
May 25 12:31:46.495519 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:31:46.495501    2549 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 25 12:31:46.799040 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:31:46.798959    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2d880044-04f1-4518-a423-305351e6122d-proxy-tls\") pod \"error-404-isvc-bcff4-predictor-55cbf45559-tw9hg\" (UID: \"2d880044-04f1-4518-a423-305351e6122d\") " pod="kserve-ci-e2e-test/error-404-isvc-bcff4-predictor-55cbf45559-tw9hg"
May 25 12:31:46.801301 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:31:46.801282    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2d880044-04f1-4518-a423-305351e6122d-proxy-tls\") pod \"error-404-isvc-bcff4-predictor-55cbf45559-tw9hg\" (UID: \"2d880044-04f1-4518-a423-305351e6122d\") " pod="kserve-ci-e2e-test/error-404-isvc-bcff4-predictor-55cbf45559-tw9hg"
May 25 12:31:46.944147 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:31:46.944114    2549 generic.go:358] "Generic (PLEG): container finished" podID="771a87fe-641c-40bd-8588-a6241156f7b0" containerID="f0d95a203f3797358bfad4055a2dd004f37bc20c03377b430648647864437335" exitCode=2
May 25 12:31:46.944572 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:31:46.944183    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-40bda-predictor-6664b4979f-g8jxd" event={"ID":"771a87fe-641c-40bd-8588-a6241156f7b0","Type":"ContainerDied","Data":"f0d95a203f3797358bfad4055a2dd004f37bc20c03377b430648647864437335"}
May 25 12:31:46.945695 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:31:46.945663    2549 generic.go:358] "Generic (PLEG): container finished" podID="7975c442-27c9-43e0-a7b5-5eba5e770eb3" containerID="e78747fa671379bda0badc747bf5aba3acb069b6f548eab73f9f26576427d7f1" exitCode=2
May 25 12:31:46.945810 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:31:46.945729    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-40bda-predictor-776fc95cd4-dr29n" event={"ID":"7975c442-27c9-43e0-a7b5-5eba5e770eb3","Type":"ContainerDied","Data":"e78747fa671379bda0badc747bf5aba3acb069b6f548eab73f9f26576427d7f1"}
May 25 12:31:46.947115 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:31:46.947091    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-bcff4-predictor-66c9fb5576-scwgx" event={"ID":"81f25695-1e0c-4e1c-beda-e1d250bf364b","Type":"ContainerStarted","Data":"c99a1a1714ffb510c36a746aa316e3394fea4789b4c344b6bae10c17e62bd2ab"}
May 25 12:31:46.947115 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:31:46.947121    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-bcff4-predictor-66c9fb5576-scwgx" event={"ID":"81f25695-1e0c-4e1c-beda-e1d250bf364b","Type":"ContainerStarted","Data":"296bb02511e80f95b944db635b8bb3734dfaa9b14243aac6888e4ec48fe137bd"}
May 25 12:31:46.947306 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:31:46.947136    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-bcff4-predictor-66c9fb5576-scwgx" event={"ID":"81f25695-1e0c-4e1c-beda-e1d250bf364b","Type":"ContainerStarted","Data":"63d5166f345ec78cae7240c5adf87f9497fc784892b8e9a1ed8c01457f056ee6"}
May 25 12:31:46.947306 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:31:46.947276    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-bcff4-predictor-66c9fb5576-scwgx"
May 25 12:31:46.971945 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:31:46.971888    2549 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-bcff4-predictor-66c9fb5576-scwgx" podStartSLOduration=1.971874754 podStartE2EDuration="1.971874754s" podCreationTimestamp="2026-05-25 12:31:45 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 12:31:46.968379786 +0000 UTC m=+1110.684195090" watchObservedRunningTime="2026-05-25 12:31:46.971874754 +0000 UTC m=+1110.687690041"
May 25 12:31:47.009807 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:31:47.009781    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-bcff4-predictor-55cbf45559-tw9hg"
May 25 12:31:47.151441 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:31:47.151379    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-bcff4-predictor-55cbf45559-tw9hg"]
May 25 12:31:47.154179 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:31:47.154149    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d880044_04f1_4518_a423_305351e6122d.slice/crio-ebd2fecd9bea773dbbb7a359adf48b1f883599c540c39df6a246c3b3f84cfe5a WatchSource:0}: Error finding container ebd2fecd9bea773dbbb7a359adf48b1f883599c540c39df6a246c3b3f84cfe5a: Status 404 returned error can't find the container with id ebd2fecd9bea773dbbb7a359adf48b1f883599c540c39df6a246c3b3f84cfe5a
May 25 12:31:47.953326 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:31:47.953289    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-bcff4-predictor-55cbf45559-tw9hg" event={"ID":"2d880044-04f1-4518-a423-305351e6122d","Type":"ContainerStarted","Data":"db8d55dfc8f345cdbeef3dc7d22ac5e888074fbe1e4ad51c52c0d7ce21eaa564"}
May 25 12:31:47.953326 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:31:47.953330    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-bcff4-predictor-55cbf45559-tw9hg" event={"ID":"2d880044-04f1-4518-a423-305351e6122d","Type":"ContainerStarted","Data":"3e75fc58162d830e63e668212e51628174efc2fd7070b781edc7c1e6186cec68"}
May 25 12:31:47.953769 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:31:47.953342    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-bcff4-predictor-55cbf45559-tw9hg" event={"ID":"2d880044-04f1-4518-a423-305351e6122d","Type":"ContainerStarted","Data":"ebd2fecd9bea773dbbb7a359adf48b1f883599c540c39df6a246c3b3f84cfe5a"}
May 25 12:31:47.953769 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:31:47.953599    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-bcff4-predictor-66c9fb5576-scwgx"
May 25 12:31:47.954985 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:31:47.954956    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-bcff4-predictor-66c9fb5576-scwgx" podUID="81f25695-1e0c-4e1c-beda-e1d250bf364b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
May 25 12:31:47.980534 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:31:47.980488    2549 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-bcff4-predictor-55cbf45559-tw9hg" podStartSLOduration=1.980473575 podStartE2EDuration="1.980473575s" podCreationTimestamp="2026-05-25 12:31:46 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 12:31:47.978759589 +0000 UTC m=+1111.694574876" watchObservedRunningTime="2026-05-25 12:31:47.980473575 +0000 UTC m=+1111.696288862"
May 25 12:31:48.951740 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:31:48.951695    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-40bda-predictor-776fc95cd4-dr29n" podUID="7975c442-27c9-43e0-a7b5-5eba5e770eb3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.36:8643/healthz\": dial tcp 10.134.0.36:8643: connect: connection refused"
May 25 12:31:48.957238 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:31:48.957191    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-40bda-predictor-776fc95cd4-dr29n" podUID="7975c442-27c9-43e0-a7b5-5eba5e770eb3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
May 25 12:31:48.957612 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:31:48.957330    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-bcff4-predictor-66c9fb5576-scwgx" podUID="81f25695-1e0c-4e1c-beda-e1d250bf364b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
May 25 12:31:48.957612 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:31:48.957373    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-bcff4-predictor-55cbf45559-tw9hg"
May 25 12:31:48.957612 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:31:48.957400    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-bcff4-predictor-55cbf45559-tw9hg"
May 25 12:31:48.958613 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:31:48.958594    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-bcff4-predictor-55cbf45559-tw9hg" podUID="2d880044-04f1-4518-a423-305351e6122d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
May 25 12:31:49.297975 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:31:49.297952    2549 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-40bda-predictor-6664b4979f-g8jxd"
May 25 12:31:49.301269 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:31:49.301250    2549 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-40bda-predictor-776fc95cd4-dr29n"
May 25 12:31:49.422510 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:31:49.422478    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c8vk8\" (UniqueName: \"kubernetes.io/projected/7975c442-27c9-43e0-a7b5-5eba5e770eb3-kube-api-access-c8vk8\") pod \"7975c442-27c9-43e0-a7b5-5eba5e770eb3\" (UID: \"7975c442-27c9-43e0-a7b5-5eba5e770eb3\") "
May 25 12:31:49.422510 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:31:49.422514    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2r5mr\" (UniqueName: \"kubernetes.io/projected/771a87fe-641c-40bd-8588-a6241156f7b0-kube-api-access-2r5mr\") pod \"771a87fe-641c-40bd-8588-a6241156f7b0\" (UID: \"771a87fe-641c-40bd-8588-a6241156f7b0\") "
May 25 12:31:49.422736 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:31:49.422581    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7975c442-27c9-43e0-a7b5-5eba5e770eb3-proxy-tls\") pod \"7975c442-27c9-43e0-a7b5-5eba5e770eb3\" (UID: \"7975c442-27c9-43e0-a7b5-5eba5e770eb3\") "
May 25 12:31:49.422736 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:31:49.422610    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"success-200-isvc-40bda-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7975c442-27c9-43e0-a7b5-5eba5e770eb3-success-200-isvc-40bda-kube-rbac-proxy-sar-config\") pod \"7975c442-27c9-43e0-a7b5-5eba5e770eb3\" (UID: \"7975c442-27c9-43e0-a7b5-5eba5e770eb3\") "
May 25 12:31:49.422736 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:31:49.422638    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"error-404-isvc-40bda-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/771a87fe-641c-40bd-8588-a6241156f7b0-error-404-isvc-40bda-kube-rbac-proxy-sar-config\") pod \"771a87fe-641c-40bd-8588-a6241156f7b0\" (UID: \"771a87fe-641c-40bd-8588-a6241156f7b0\") "
May 25 12:31:49.422736 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:31:49.422667    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/771a87fe-641c-40bd-8588-a6241156f7b0-proxy-tls\") pod \"771a87fe-641c-40bd-8588-a6241156f7b0\" (UID: \"771a87fe-641c-40bd-8588-a6241156f7b0\") "
May 25 12:31:49.423030 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:31:49.422994    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7975c442-27c9-43e0-a7b5-5eba5e770eb3-success-200-isvc-40bda-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "success-200-isvc-40bda-kube-rbac-proxy-sar-config") pod "7975c442-27c9-43e0-a7b5-5eba5e770eb3" (UID: "7975c442-27c9-43e0-a7b5-5eba5e770eb3"). InnerVolumeSpecName "success-200-isvc-40bda-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 12:31:49.423030 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:31:49.423020    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/771a87fe-641c-40bd-8588-a6241156f7b0-error-404-isvc-40bda-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "error-404-isvc-40bda-kube-rbac-proxy-sar-config") pod "771a87fe-641c-40bd-8588-a6241156f7b0" (UID: "771a87fe-641c-40bd-8588-a6241156f7b0"). InnerVolumeSpecName "error-404-isvc-40bda-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 12:31:49.424804 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:31:49.424773    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/771a87fe-641c-40bd-8588-a6241156f7b0-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "771a87fe-641c-40bd-8588-a6241156f7b0" (UID: "771a87fe-641c-40bd-8588-a6241156f7b0"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 12:31:49.424804 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:31:49.424779    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7975c442-27c9-43e0-a7b5-5eba5e770eb3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7975c442-27c9-43e0-a7b5-5eba5e770eb3" (UID: "7975c442-27c9-43e0-a7b5-5eba5e770eb3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 12:31:49.425169 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:31:49.425146    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7975c442-27c9-43e0-a7b5-5eba5e770eb3-kube-api-access-c8vk8" (OuterVolumeSpecName: "kube-api-access-c8vk8") pod "7975c442-27c9-43e0-a7b5-5eba5e770eb3" (UID: "7975c442-27c9-43e0-a7b5-5eba5e770eb3"). InnerVolumeSpecName "kube-api-access-c8vk8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 12:31:49.425285 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:31:49.425176    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/771a87fe-641c-40bd-8588-a6241156f7b0-kube-api-access-2r5mr" (OuterVolumeSpecName: "kube-api-access-2r5mr") pod "771a87fe-641c-40bd-8588-a6241156f7b0" (UID: "771a87fe-641c-40bd-8588-a6241156f7b0"). InnerVolumeSpecName "kube-api-access-2r5mr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 12:31:49.523960 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:31:49.523862    2549 reconciler_common.go:299] "Volume detached for volume \"success-200-isvc-40bda-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7975c442-27c9-43e0-a7b5-5eba5e770eb3-success-200-isvc-40bda-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
May 25 12:31:49.523960 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:31:49.523910    2549 reconciler_common.go:299] "Volume detached for volume \"error-404-isvc-40bda-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/771a87fe-641c-40bd-8588-a6241156f7b0-error-404-isvc-40bda-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
May 25 12:31:49.523960 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:31:49.523922    2549 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/771a87fe-641c-40bd-8588-a6241156f7b0-proxy-tls\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
May 25 12:31:49.523960 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:31:49.523933    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c8vk8\" (UniqueName: \"kubernetes.io/projected/7975c442-27c9-43e0-a7b5-5eba5e770eb3-kube-api-access-c8vk8\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
May 25 12:31:49.523960 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:31:49.523943    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2r5mr\" (UniqueName: \"kubernetes.io/projected/771a87fe-641c-40bd-8588-a6241156f7b0-kube-api-access-2r5mr\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
May 25 12:31:49.523960 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:31:49.523952    2549 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7975c442-27c9-43e0-a7b5-5eba5e770eb3-proxy-tls\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
May 25 12:31:49.961802 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:31:49.961765    2549 generic.go:358] "Generic (PLEG): container finished" podID="771a87fe-641c-40bd-8588-a6241156f7b0" containerID="f3a663c372c80352fb12e353d2d07a1103246f0ea31ed16abf773d60834a179e" exitCode=0
May 25 12:31:49.962295 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:31:49.961838    2549 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-40bda-predictor-6664b4979f-g8jxd"
May 25 12:31:49.962295 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:31:49.961836    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-40bda-predictor-6664b4979f-g8jxd" event={"ID":"771a87fe-641c-40bd-8588-a6241156f7b0","Type":"ContainerDied","Data":"f3a663c372c80352fb12e353d2d07a1103246f0ea31ed16abf773d60834a179e"}
May 25 12:31:49.962295 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:31:49.961959    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-40bda-predictor-6664b4979f-g8jxd" event={"ID":"771a87fe-641c-40bd-8588-a6241156f7b0","Type":"ContainerDied","Data":"17141695b02291d3a4daf9e2aa9d8570feec818030c5ac60d0ba8f650611d317"}
May 25 12:31:49.962295 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:31:49.961987    2549 scope.go:117] "RemoveContainer" containerID="f0d95a203f3797358bfad4055a2dd004f37bc20c03377b430648647864437335"
May 25 12:31:49.966571 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:31:49.966542    2549 generic.go:358] "Generic (PLEG): container finished" podID="7975c442-27c9-43e0-a7b5-5eba5e770eb3" containerID="26988f813bd21746bb89e4aa264d007defcf9fb5128c030a9e6b2a293f450082" exitCode=0
May 25 12:31:49.966670 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:31:49.966621    2549 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-40bda-predictor-776fc95cd4-dr29n"
May 25 12:31:49.966735 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:31:49.966617    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-40bda-predictor-776fc95cd4-dr29n" event={"ID":"7975c442-27c9-43e0-a7b5-5eba5e770eb3","Type":"ContainerDied","Data":"26988f813bd21746bb89e4aa264d007defcf9fb5128c030a9e6b2a293f450082"}
May 25 12:31:49.966735 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:31:49.966730    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-40bda-predictor-776fc95cd4-dr29n" event={"ID":"7975c442-27c9-43e0-a7b5-5eba5e770eb3","Type":"ContainerDied","Data":"36452567b8f9888733a43275332c57b822bfb19628f3f148a2b9249d0108594f"}
May 25 12:31:49.967136 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:31:49.967113    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-bcff4-predictor-55cbf45559-tw9hg" podUID="2d880044-04f1-4518-a423-305351e6122d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
May 25 12:31:49.974816 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:31:49.974795    2549 scope.go:117] "RemoveContainer" containerID="f3a663c372c80352fb12e353d2d07a1103246f0ea31ed16abf773d60834a179e"
May 25 12:31:49.983059 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:31:49.983042    2549 scope.go:117] "RemoveContainer" containerID="f0d95a203f3797358bfad4055a2dd004f37bc20c03377b430648647864437335"
May 25 12:31:49.983329 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:31:49.983307    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0d95a203f3797358bfad4055a2dd004f37bc20c03377b430648647864437335\": container with ID starting with f0d95a203f3797358bfad4055a2dd004f37bc20c03377b430648647864437335 not found: ID does not exist" containerID="f0d95a203f3797358bfad4055a2dd004f37bc20c03377b430648647864437335"
May 25 12:31:49.983381 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:31:49.983341    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0d95a203f3797358bfad4055a2dd004f37bc20c03377b430648647864437335"} err="failed to get container status \"f0d95a203f3797358bfad4055a2dd004f37bc20c03377b430648647864437335\": rpc error: code = NotFound desc = could not find container \"f0d95a203f3797358bfad4055a2dd004f37bc20c03377b430648647864437335\": container with ID starting with f0d95a203f3797358bfad4055a2dd004f37bc20c03377b430648647864437335 not found: ID does not exist"
May 25 12:31:49.983381 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:31:49.983363    2549 scope.go:117] "RemoveContainer" containerID="f3a663c372c80352fb12e353d2d07a1103246f0ea31ed16abf773d60834a179e"
May 25 12:31:49.983606 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:31:49.983590    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3a663c372c80352fb12e353d2d07a1103246f0ea31ed16abf773d60834a179e\": container with ID starting with f3a663c372c80352fb12e353d2d07a1103246f0ea31ed16abf773d60834a179e not found: ID does not exist" containerID="f3a663c372c80352fb12e353d2d07a1103246f0ea31ed16abf773d60834a179e"
May 25 12:31:49.983665 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:31:49.983613    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3a663c372c80352fb12e353d2d07a1103246f0ea31ed16abf773d60834a179e"} err="failed to get container status \"f3a663c372c80352fb12e353d2d07a1103246f0ea31ed16abf773d60834a179e\": rpc error: code = NotFound desc = could not find container \"f3a663c372c80352fb12e353d2d07a1103246f0ea31ed16abf773d60834a179e\": container with ID starting with f3a663c372c80352fb12e353d2d07a1103246f0ea31ed16abf773d60834a179e not found: ID does not exist"
May 25 12:31:49.983665 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:31:49.983632    2549 scope.go:117] "RemoveContainer" containerID="e78747fa671379bda0badc747bf5aba3acb069b6f548eab73f9f26576427d7f1"
May 25 12:31:49.991403 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:31:49.991383    2549 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-40bda-predictor-6664b4979f-g8jxd"]
May 25 12:31:49.992590 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:31:49.992567    2549 scope.go:117] "RemoveContainer" containerID="26988f813bd21746bb89e4aa264d007defcf9fb5128c030a9e6b2a293f450082"
May 25 12:31:49.993987 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:31:49.993968    2549 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-40bda-predictor-6664b4979f-g8jxd"]
May 25 12:31:49.999570 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:31:49.999555    2549 scope.go:117] "RemoveContainer" containerID="e78747fa671379bda0badc747bf5aba3acb069b6f548eab73f9f26576427d7f1"
May 25 12:31:49.999790 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:31:49.999774    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e78747fa671379bda0badc747bf5aba3acb069b6f548eab73f9f26576427d7f1\": container with ID starting with e78747fa671379bda0badc747bf5aba3acb069b6f548eab73f9f26576427d7f1 not found: ID does not exist" containerID="e78747fa671379bda0badc747bf5aba3acb069b6f548eab73f9f26576427d7f1"
May 25 12:31:49.999841 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:31:49.999797    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e78747fa671379bda0badc747bf5aba3acb069b6f548eab73f9f26576427d7f1"} err="failed to get container status \"e78747fa671379bda0badc747bf5aba3acb069b6f548eab73f9f26576427d7f1\": rpc error: code = NotFound desc = could not find container \"e78747fa671379bda0badc747bf5aba3acb069b6f548eab73f9f26576427d7f1\": container with ID starting with e78747fa671379bda0badc747bf5aba3acb069b6f548eab73f9f26576427d7f1 not found: ID does not exist"
May 25 12:31:49.999841 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:31:49.999813    2549 scope.go:117] "RemoveContainer" containerID="26988f813bd21746bb89e4aa264d007defcf9fb5128c030a9e6b2a293f450082"
May 25 12:31:50.000000 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:31:49.999986    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26988f813bd21746bb89e4aa264d007defcf9fb5128c030a9e6b2a293f450082\": container with ID starting with 26988f813bd21746bb89e4aa264d007defcf9fb5128c030a9e6b2a293f450082 not found: ID does not exist" containerID="26988f813bd21746bb89e4aa264d007defcf9fb5128c030a9e6b2a293f450082"
May 25 12:31:50.000035 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:31:50.000003    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26988f813bd21746bb89e4aa264d007defcf9fb5128c030a9e6b2a293f450082"} err="failed to get container status \"26988f813bd21746bb89e4aa264d007defcf9fb5128c030a9e6b2a293f450082\": rpc error: code = NotFound desc = could not find container \"26988f813bd21746bb89e4aa264d007defcf9fb5128c030a9e6b2a293f450082\": container with ID starting with 26988f813bd21746bb89e4aa264d007defcf9fb5128c030a9e6b2a293f450082 not found: ID does not exist"
May 25 12:31:50.006063 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:31:50.006042    2549 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-40bda-predictor-776fc95cd4-dr29n"]
May 25 12:31:50.010308 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:31:50.010289    2549 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-40bda-predictor-776fc95cd4-dr29n"]
May 25 12:31:50.848049 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:31:50.848016    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="771a87fe-641c-40bd-8588-a6241156f7b0" path="/var/lib/kubelet/pods/771a87fe-641c-40bd-8588-a6241156f7b0/volumes"
May 25 12:31:50.848458 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:31:50.848443    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7975c442-27c9-43e0-a7b5-5eba5e770eb3" path="/var/lib/kubelet/pods/7975c442-27c9-43e0-a7b5-5eba5e770eb3/volumes"
May 25 12:31:53.961814 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:31:53.961786    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-bcff4-predictor-66c9fb5576-scwgx"
May 25 12:31:53.962416 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:31:53.962389    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-bcff4-predictor-66c9fb5576-scwgx" podUID="81f25695-1e0c-4e1c-beda-e1d250bf364b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
May 25 12:31:54.971852 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:31:54.971822    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-bcff4-predictor-55cbf45559-tw9hg"
May 25 12:31:54.972368 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:31:54.972337    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-bcff4-predictor-55cbf45559-tw9hg" podUID="2d880044-04f1-4518-a423-305351e6122d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
May 25 12:32:03.962837 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:03.962786    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-bcff4-predictor-66c9fb5576-scwgx" podUID="81f25695-1e0c-4e1c-beda-e1d250bf364b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
May 25 12:32:04.972778 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:04.972730    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-bcff4-predictor-55cbf45559-tw9hg" podUID="2d880044-04f1-4518-a423-305351e6122d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
May 25 12:32:13.963085 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:13.963042    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-bcff4-predictor-66c9fb5576-scwgx" podUID="81f25695-1e0c-4e1c-beda-e1d250bf364b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
May 25 12:32:14.972613 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:14.972577    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-bcff4-predictor-55cbf45559-tw9hg" podUID="2d880044-04f1-4518-a423-305351e6122d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
May 25 12:32:15.966302 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:15.966266    2549 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-833d6-predictor-65cfc8fdbb-xw7b6"]
May 25 12:32:15.966641 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:15.966606    2549 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-833d6-predictor-65cfc8fdbb-xw7b6" podUID="6264ef64-b6b2-43bd-bcfb-f281e7205194" containerName="kserve-container" containerID="cri-o://faa6f5119adba2a116a1c635be6ab083bb56996ace4397c26b6a8b3c1c8ae2ba" gracePeriod=30
May 25 12:32:15.966774 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:15.966666    2549 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-833d6-predictor-65cfc8fdbb-xw7b6" podUID="6264ef64-b6b2-43bd-bcfb-f281e7205194" containerName="kube-rbac-proxy" containerID="cri-o://70ae413c651d2a45a21f1ae79d8fa1013f9ba89f9aa51d29a41d72dde95b522c" gracePeriod=30
May 25 12:32:16.025883 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:16.025847    2549 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-acbca-predictor-59cffd4b6b-zwcgm"]
May 25 12:32:16.026328 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:16.026308    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="771a87fe-641c-40bd-8588-a6241156f7b0" containerName="kube-rbac-proxy"
May 25 12:32:16.026388 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:16.026332    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="771a87fe-641c-40bd-8588-a6241156f7b0" containerName="kube-rbac-proxy"
May 25 12:32:16.026388 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:16.026349    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7975c442-27c9-43e0-a7b5-5eba5e770eb3" containerName="kserve-container"
May 25 12:32:16.026388 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:16.026359    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="7975c442-27c9-43e0-a7b5-5eba5e770eb3" containerName="kserve-container"
May 25 12:32:16.026388 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:16.026373    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7975c442-27c9-43e0-a7b5-5eba5e770eb3" containerName="kube-rbac-proxy"
May 25 12:32:16.026388 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:16.026379    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="7975c442-27c9-43e0-a7b5-5eba5e770eb3" containerName="kube-rbac-proxy"
May 25 12:32:16.026579 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:16.026396    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="771a87fe-641c-40bd-8588-a6241156f7b0" containerName="kserve-container"
May 25 12:32:16.026579 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:16.026402    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="771a87fe-641c-40bd-8588-a6241156f7b0" containerName="kserve-container"
May 25 12:32:16.026579 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:16.026462    2549 memory_manager.go:356] "RemoveStaleState removing state" podUID="7975c442-27c9-43e0-a7b5-5eba5e770eb3" containerName="kube-rbac-proxy"
May 25 12:32:16.026579 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:16.026473    2549 memory_manager.go:356] "RemoveStaleState removing state" podUID="7975c442-27c9-43e0-a7b5-5eba5e770eb3" containerName="kserve-container"
May 25 12:32:16.026579 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:16.026482    2549 memory_manager.go:356] "RemoveStaleState removing state" podUID="771a87fe-641c-40bd-8588-a6241156f7b0" containerName="kserve-container"
May 25 12:32:16.026579 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:16.026489    2549 memory_manager.go:356] "RemoveStaleState removing state" podUID="771a87fe-641c-40bd-8588-a6241156f7b0" containerName="kube-rbac-proxy"
May 25 12:32:16.029994 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:16.029972    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-acbca-predictor-59cffd4b6b-zwcgm"
May 25 12:32:16.035458 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:16.035354    2549 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-acbca-kube-rbac-proxy-sar-config\""
May 25 12:32:16.035901 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:16.035883    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-acbca-predictor-serving-cert\""
May 25 12:32:16.063682 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:16.063469    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"success-200-isvc-acbca-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f2f7c261-0a24-4aa1-b841-a92ba3120164-success-200-isvc-acbca-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-acbca-predictor-59cffd4b6b-zwcgm\" (UID: \"f2f7c261-0a24-4aa1-b841-a92ba3120164\") " pod="kserve-ci-e2e-test/success-200-isvc-acbca-predictor-59cffd4b6b-zwcgm"
May 25 12:32:16.063682 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:16.063541    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chlw6\" (UniqueName: \"kubernetes.io/projected/f2f7c261-0a24-4aa1-b841-a92ba3120164-kube-api-access-chlw6\") pod \"success-200-isvc-acbca-predictor-59cffd4b6b-zwcgm\" (UID: \"f2f7c261-0a24-4aa1-b841-a92ba3120164\") " pod="kserve-ci-e2e-test/success-200-isvc-acbca-predictor-59cffd4b6b-zwcgm"
May 25 12:32:16.063682 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:16.063591    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f2f7c261-0a24-4aa1-b841-a92ba3120164-proxy-tls\") pod \"success-200-isvc-acbca-predictor-59cffd4b6b-zwcgm\" (UID: \"f2f7c261-0a24-4aa1-b841-a92ba3120164\") " pod="kserve-ci-e2e-test/success-200-isvc-acbca-predictor-59cffd4b6b-zwcgm"
May 25 12:32:16.065146 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:16.065118    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-acbca-predictor-59cffd4b6b-zwcgm"]
May 25 12:32:16.120182 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:16.120147    2549 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-833d6-predictor-7dfb76b5f7-8rzb8"]
May 25 12:32:16.120562 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:16.120530    2549 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-833d6-predictor-7dfb76b5f7-8rzb8" podUID="40582ba8-ab3b-4046-985c-bcfb29eb0b82" containerName="kserve-container" containerID="cri-o://a425f8bbcb74f080c3fcf425bafc40db40ec3a200c76c9119be0657bcdd53c40" gracePeriod=30
May 25 12:32:16.120701 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:16.120557    2549 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-833d6-predictor-7dfb76b5f7-8rzb8" podUID="40582ba8-ab3b-4046-985c-bcfb29eb0b82" containerName="kube-rbac-proxy" containerID="cri-o://29f4bcbe01f5e4f8d92e6577bb108f04749a8fa53c73783930db5dc1e1da6340" gracePeriod=30
May 25 12:32:16.164403 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:16.164362    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"success-200-isvc-acbca-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f2f7c261-0a24-4aa1-b841-a92ba3120164-success-200-isvc-acbca-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-acbca-predictor-59cffd4b6b-zwcgm\" (UID: \"f2f7c261-0a24-4aa1-b841-a92ba3120164\") " pod="kserve-ci-e2e-test/success-200-isvc-acbca-predictor-59cffd4b6b-zwcgm"
May 25 12:32:16.164594 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:16.164438    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-chlw6\" (UniqueName: \"kubernetes.io/projected/f2f7c261-0a24-4aa1-b841-a92ba3120164-kube-api-access-chlw6\") pod \"success-200-isvc-acbca-predictor-59cffd4b6b-zwcgm\" (UID: \"f2f7c261-0a24-4aa1-b841-a92ba3120164\") " pod="kserve-ci-e2e-test/success-200-isvc-acbca-predictor-59cffd4b6b-zwcgm"
May 25 12:32:16.164594 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:16.164487    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f2f7c261-0a24-4aa1-b841-a92ba3120164-proxy-tls\") pod \"success-200-isvc-acbca-predictor-59cffd4b6b-zwcgm\" (UID: \"f2f7c261-0a24-4aa1-b841-a92ba3120164\") " pod="kserve-ci-e2e-test/success-200-isvc-acbca-predictor-59cffd4b6b-zwcgm"
May 25 12:32:16.164720 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:32:16.164624    2549 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-acbca-predictor-serving-cert: secret "success-200-isvc-acbca-predictor-serving-cert" not found
May 25 12:32:16.164720 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:32:16.164685    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2f7c261-0a24-4aa1-b841-a92ba3120164-proxy-tls podName:f2f7c261-0a24-4aa1-b841-a92ba3120164 nodeName:}" failed. No retries permitted until 2026-05-25 12:32:16.664664099 +0000 UTC m=+1140.380479377 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/f2f7c261-0a24-4aa1-b841-a92ba3120164-proxy-tls") pod "success-200-isvc-acbca-predictor-59cffd4b6b-zwcgm" (UID: "f2f7c261-0a24-4aa1-b841-a92ba3120164") : secret "success-200-isvc-acbca-predictor-serving-cert" not found
May 25 12:32:16.165207 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:16.165165    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"success-200-isvc-acbca-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f2f7c261-0a24-4aa1-b841-a92ba3120164-success-200-isvc-acbca-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-acbca-predictor-59cffd4b6b-zwcgm\" (UID: \"f2f7c261-0a24-4aa1-b841-a92ba3120164\") " pod="kserve-ci-e2e-test/success-200-isvc-acbca-predictor-59cffd4b6b-zwcgm"
May 25 12:32:16.183081 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:16.183053    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-chlw6\" (UniqueName: \"kubernetes.io/projected/f2f7c261-0a24-4aa1-b841-a92ba3120164-kube-api-access-chlw6\") pod \"success-200-isvc-acbca-predictor-59cffd4b6b-zwcgm\" (UID: \"f2f7c261-0a24-4aa1-b841-a92ba3120164\") " pod="kserve-ci-e2e-test/success-200-isvc-acbca-predictor-59cffd4b6b-zwcgm"
May 25 12:32:16.232856 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:16.232788    2549 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-acbca-predictor-687469f549-r5cwv"]
May 25 12:32:16.236583 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:16.236562    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-acbca-predictor-687469f549-r5cwv"
May 25 12:32:16.242016 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:16.241995    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-acbca-predictor-serving-cert\""
May 25 12:32:16.242599 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:16.242571    2549 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-acbca-kube-rbac-proxy-sar-config\""
May 25 12:32:16.265112 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:16.265087    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nsdmc\" (UniqueName: \"kubernetes.io/projected/51175389-4295-493b-82bc-812c2e34c778-kube-api-access-nsdmc\") pod \"error-404-isvc-acbca-predictor-687469f549-r5cwv\" (UID: \"51175389-4295-493b-82bc-812c2e34c778\") " pod="kserve-ci-e2e-test/error-404-isvc-acbca-predictor-687469f549-r5cwv"
May 25 12:32:16.265338 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:16.265313    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"error-404-isvc-acbca-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/51175389-4295-493b-82bc-812c2e34c778-error-404-isvc-acbca-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-acbca-predictor-687469f549-r5cwv\" (UID: \"51175389-4295-493b-82bc-812c2e34c778\") " pod="kserve-ci-e2e-test/error-404-isvc-acbca-predictor-687469f549-r5cwv"
May 25 12:32:16.265409 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:16.265377    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/51175389-4295-493b-82bc-812c2e34c778-proxy-tls\") pod \"error-404-isvc-acbca-predictor-687469f549-r5cwv\" (UID: \"51175389-4295-493b-82bc-812c2e34c778\") " pod="kserve-ci-e2e-test/error-404-isvc-acbca-predictor-687469f549-r5cwv"
May 25 12:32:16.267364 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:16.267345    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-acbca-predictor-687469f549-r5cwv"]
May 25 12:32:16.366148 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:16.366110    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-nsdmc\" (UniqueName: \"kubernetes.io/projected/51175389-4295-493b-82bc-812c2e34c778-kube-api-access-nsdmc\") pod \"error-404-isvc-acbca-predictor-687469f549-r5cwv\" (UID: \"51175389-4295-493b-82bc-812c2e34c778\") " pod="kserve-ci-e2e-test/error-404-isvc-acbca-predictor-687469f549-r5cwv"
May 25 12:32:16.366334 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:16.366218    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"error-404-isvc-acbca-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/51175389-4295-493b-82bc-812c2e34c778-error-404-isvc-acbca-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-acbca-predictor-687469f549-r5cwv\" (UID: \"51175389-4295-493b-82bc-812c2e34c778\") " pod="kserve-ci-e2e-test/error-404-isvc-acbca-predictor-687469f549-r5cwv"
May 25 12:32:16.366334 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:16.366253    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/51175389-4295-493b-82bc-812c2e34c778-proxy-tls\") pod \"error-404-isvc-acbca-predictor-687469f549-r5cwv\" (UID: \"51175389-4295-493b-82bc-812c2e34c778\") " pod="kserve-ci-e2e-test/error-404-isvc-acbca-predictor-687469f549-r5cwv"
May 25 12:32:16.366437 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:32:16.366364    2549 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-acbca-predictor-serving-cert: secret "error-404-isvc-acbca-predictor-serving-cert" not found
May 25 12:32:16.366437 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:32:16.366421    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51175389-4295-493b-82bc-812c2e34c778-proxy-tls podName:51175389-4295-493b-82bc-812c2e34c778 nodeName:}" failed. No retries permitted until 2026-05-25 12:32:16.866402803 +0000 UTC m=+1140.582218083 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/51175389-4295-493b-82bc-812c2e34c778-proxy-tls") pod "error-404-isvc-acbca-predictor-687469f549-r5cwv" (UID: "51175389-4295-493b-82bc-812c2e34c778") : secret "error-404-isvc-acbca-predictor-serving-cert" not found
May 25 12:32:16.366894 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:16.366875    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"error-404-isvc-acbca-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/51175389-4295-493b-82bc-812c2e34c778-error-404-isvc-acbca-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-acbca-predictor-687469f549-r5cwv\" (UID: \"51175389-4295-493b-82bc-812c2e34c778\") " pod="kserve-ci-e2e-test/error-404-isvc-acbca-predictor-687469f549-r5cwv"
May 25 12:32:16.382415 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:16.382386    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nsdmc\" (UniqueName: \"kubernetes.io/projected/51175389-4295-493b-82bc-812c2e34c778-kube-api-access-nsdmc\") pod \"error-404-isvc-acbca-predictor-687469f549-r5cwv\" (UID: \"51175389-4295-493b-82bc-812c2e34c778\") " pod="kserve-ci-e2e-test/error-404-isvc-acbca-predictor-687469f549-r5cwv"
May 25 12:32:16.669188 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:16.669140    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f2f7c261-0a24-4aa1-b841-a92ba3120164-proxy-tls\") pod \"success-200-isvc-acbca-predictor-59cffd4b6b-zwcgm\" (UID: \"f2f7c261-0a24-4aa1-b841-a92ba3120164\") " pod="kserve-ci-e2e-test/success-200-isvc-acbca-predictor-59cffd4b6b-zwcgm"
May 25 12:32:16.671676 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:16.671639    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f2f7c261-0a24-4aa1-b841-a92ba3120164-proxy-tls\") pod \"success-200-isvc-acbca-predictor-59cffd4b6b-zwcgm\" (UID: \"f2f7c261-0a24-4aa1-b841-a92ba3120164\") " pod="kserve-ci-e2e-test/success-200-isvc-acbca-predictor-59cffd4b6b-zwcgm"
May 25 12:32:16.871536 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:16.871495    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/51175389-4295-493b-82bc-812c2e34c778-proxy-tls\") pod \"error-404-isvc-acbca-predictor-687469f549-r5cwv\" (UID: \"51175389-4295-493b-82bc-812c2e34c778\") " pod="kserve-ci-e2e-test/error-404-isvc-acbca-predictor-687469f549-r5cwv"
May 25 12:32:16.873855 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:16.873831    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/51175389-4295-493b-82bc-812c2e34c778-proxy-tls\") pod \"error-404-isvc-acbca-predictor-687469f549-r5cwv\" (UID: \"51175389-4295-493b-82bc-812c2e34c778\") " pod="kserve-ci-e2e-test/error-404-isvc-acbca-predictor-687469f549-r5cwv"
May 25 12:32:16.943393 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:16.943314    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-acbca-predictor-59cffd4b6b-zwcgm"
May 25 12:32:17.068158 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:17.068127    2549 generic.go:358] "Generic (PLEG): container finished" podID="40582ba8-ab3b-4046-985c-bcfb29eb0b82" containerID="29f4bcbe01f5e4f8d92e6577bb108f04749a8fa53c73783930db5dc1e1da6340" exitCode=2
May 25 12:32:17.068528 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:17.068190    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-833d6-predictor-7dfb76b5f7-8rzb8" event={"ID":"40582ba8-ab3b-4046-985c-bcfb29eb0b82","Type":"ContainerDied","Data":"29f4bcbe01f5e4f8d92e6577bb108f04749a8fa53c73783930db5dc1e1da6340"}
May 25 12:32:17.069763 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:17.069739    2549 generic.go:358] "Generic (PLEG): container finished" podID="6264ef64-b6b2-43bd-bcfb-f281e7205194" containerID="70ae413c651d2a45a21f1ae79d8fa1013f9ba89f9aa51d29a41d72dde95b522c" exitCode=2
May 25 12:32:17.069870 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:17.069811    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-833d6-predictor-65cfc8fdbb-xw7b6" event={"ID":"6264ef64-b6b2-43bd-bcfb-f281e7205194","Type":"ContainerDied","Data":"70ae413c651d2a45a21f1ae79d8fa1013f9ba89f9aa51d29a41d72dde95b522c"}
May 25 12:32:17.105429 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:17.105140    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-acbca-predictor-59cffd4b6b-zwcgm"]
May 25 12:32:17.107934 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:32:17.107908    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2f7c261_0a24_4aa1_b841_a92ba3120164.slice/crio-abb0e3c26eceb7b4838571f823baaba9c86606c96e112f41f557d9003500e27c WatchSource:0}: Error finding container abb0e3c26eceb7b4838571f823baaba9c86606c96e112f41f557d9003500e27c: Status 404 returned error can't find the container with id abb0e3c26eceb7b4838571f823baaba9c86606c96e112f41f557d9003500e27c
May 25 12:32:17.147965 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:17.147943    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-acbca-predictor-687469f549-r5cwv"
May 25 12:32:17.295034 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:17.295008    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-acbca-predictor-687469f549-r5cwv"]
May 25 12:32:17.298446 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:32:17.298411    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51175389_4295_493b_82bc_812c2e34c778.slice/crio-c2f964ace85188a3f90591687fc247044f187e5d16d1c379f14e6fad37f88bca WatchSource:0}: Error finding container c2f964ace85188a3f90591687fc247044f187e5d16d1c379f14e6fad37f88bca: Status 404 returned error can't find the container with id c2f964ace85188a3f90591687fc247044f187e5d16d1c379f14e6fad37f88bca
May 25 12:32:18.076597 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:18.076560    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-acbca-predictor-687469f549-r5cwv" event={"ID":"51175389-4295-493b-82bc-812c2e34c778","Type":"ContainerStarted","Data":"08e7dd83266cb5136a1b42ee1de5c01799d760726127fed0fd7c845cd9a8d082"}
May 25 12:32:18.077052 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:18.076604    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-acbca-predictor-687469f549-r5cwv" event={"ID":"51175389-4295-493b-82bc-812c2e34c778","Type":"ContainerStarted","Data":"1b1e92734dd67ad756c33fbf5939df977d7ac842685b56521b694534adec3c39"}
May 25 12:32:18.077052 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:18.076619    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-acbca-predictor-687469f549-r5cwv" event={"ID":"51175389-4295-493b-82bc-812c2e34c778","Type":"ContainerStarted","Data":"c2f964ace85188a3f90591687fc247044f187e5d16d1c379f14e6fad37f88bca"}
May 25 12:32:18.077052 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:18.076732    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-acbca-predictor-687469f549-r5cwv"
May 25 12:32:18.077052 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:18.076752    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-acbca-predictor-687469f549-r5cwv"
May 25 12:32:18.078119 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:18.078091    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-acbca-predictor-687469f549-r5cwv" podUID="51175389-4295-493b-82bc-812c2e34c778" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
May 25 12:32:18.078260 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:18.078228    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-acbca-predictor-59cffd4b6b-zwcgm" event={"ID":"f2f7c261-0a24-4aa1-b841-a92ba3120164","Type":"ContainerStarted","Data":"e02216eab1f2133c978c0fc004ba611ed778f495507f9d3aeb5b435b6bb7acb0"}
May 25 12:32:18.078260 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:18.078255    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-acbca-predictor-59cffd4b6b-zwcgm" event={"ID":"f2f7c261-0a24-4aa1-b841-a92ba3120164","Type":"ContainerStarted","Data":"12872b3e036cc6bd133452554c75f0b39f480650f7eb3f21a3ef7f4fbea6ca53"}
May 25 12:32:18.078376 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:18.078264    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-acbca-predictor-59cffd4b6b-zwcgm" event={"ID":"f2f7c261-0a24-4aa1-b841-a92ba3120164","Type":"ContainerStarted","Data":"abb0e3c26eceb7b4838571f823baaba9c86606c96e112f41f557d9003500e27c"}
May 25 12:32:18.078429 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:18.078420    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-acbca-predictor-59cffd4b6b-zwcgm"
May 25 12:32:18.078479 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:18.078439    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-acbca-predictor-59cffd4b6b-zwcgm"
May 25 12:32:18.079409 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:18.079387    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-acbca-predictor-59cffd4b6b-zwcgm" podUID="f2f7c261-0a24-4aa1-b841-a92ba3120164" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
May 25 12:32:18.101678 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:18.101637    2549 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-acbca-predictor-687469f549-r5cwv" podStartSLOduration=2.101624149 podStartE2EDuration="2.101624149s" podCreationTimestamp="2026-05-25 12:32:16 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 12:32:18.100550271 +0000 UTC m=+1141.816365558" watchObservedRunningTime="2026-05-25 12:32:18.101624149 +0000 UTC m=+1141.817439436"
May 25 12:32:18.124888 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:18.124843    2549 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-acbca-predictor-59cffd4b6b-zwcgm" podStartSLOduration=2.124830356 podStartE2EDuration="2.124830356s" podCreationTimestamp="2026-05-25 12:32:16 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 12:32:18.122642729 +0000 UTC m=+1141.838458015" watchObservedRunningTime="2026-05-25 12:32:18.124830356 +0000 UTC m=+1141.840645643"
May 25 12:32:19.070121 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:19.070083    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-833d6-predictor-65cfc8fdbb-xw7b6" podUID="6264ef64-b6b2-43bd-bcfb-f281e7205194" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.38:8643/healthz\": dial tcp 10.134.0.38:8643: connect: connection refused"
May 25 12:32:19.076441 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:19.076416    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-833d6-predictor-65cfc8fdbb-xw7b6" podUID="6264ef64-b6b2-43bd-bcfb-f281e7205194" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
May 25 12:32:19.082558 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:19.082526    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-acbca-predictor-687469f549-r5cwv" podUID="51175389-4295-493b-82bc-812c2e34c778" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
May 25 12:32:19.082859 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:19.082529    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-acbca-predictor-59cffd4b6b-zwcgm" podUID="f2f7c261-0a24-4aa1-b841-a92ba3120164" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
May 25 12:32:20.031810 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:20.031787    2549 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-833d6-predictor-65cfc8fdbb-xw7b6"
May 25 12:32:20.068632 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:20.068611    2549 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-833d6-predictor-7dfb76b5f7-8rzb8"
May 25 12:32:20.087985 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:20.087951    2549 generic.go:358] "Generic (PLEG): container finished" podID="6264ef64-b6b2-43bd-bcfb-f281e7205194" containerID="faa6f5119adba2a116a1c635be6ab083bb56996ace4397c26b6a8b3c1c8ae2ba" exitCode=0
May 25 12:32:20.088341 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:20.088138    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-833d6-predictor-65cfc8fdbb-xw7b6" event={"ID":"6264ef64-b6b2-43bd-bcfb-f281e7205194","Type":"ContainerDied","Data":"faa6f5119adba2a116a1c635be6ab083bb56996ace4397c26b6a8b3c1c8ae2ba"}
May 25 12:32:20.088341 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:20.088168    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-833d6-predictor-65cfc8fdbb-xw7b6" event={"ID":"6264ef64-b6b2-43bd-bcfb-f281e7205194","Type":"ContainerDied","Data":"3d4ffa25d3755171c045fad2d65557223c0451a635bf5440fbed8ce47805feec"}
May 25 12:32:20.088341 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:20.088189    2549 scope.go:117] "RemoveContainer" containerID="70ae413c651d2a45a21f1ae79d8fa1013f9ba89f9aa51d29a41d72dde95b522c"
May 25 12:32:20.088460 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:20.088374    2549 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-833d6-predictor-65cfc8fdbb-xw7b6"
May 25 12:32:20.090407 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:20.090374    2549 generic.go:358] "Generic (PLEG): container finished" podID="40582ba8-ab3b-4046-985c-bcfb29eb0b82" containerID="a425f8bbcb74f080c3fcf425bafc40db40ec3a200c76c9119be0657bcdd53c40" exitCode=0
May 25 12:32:20.090529 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:20.090423    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-833d6-predictor-7dfb76b5f7-8rzb8" event={"ID":"40582ba8-ab3b-4046-985c-bcfb29eb0b82","Type":"ContainerDied","Data":"a425f8bbcb74f080c3fcf425bafc40db40ec3a200c76c9119be0657bcdd53c40"}
May 25 12:32:20.090529 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:20.090449    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-833d6-predictor-7dfb76b5f7-8rzb8" event={"ID":"40582ba8-ab3b-4046-985c-bcfb29eb0b82","Type":"ContainerDied","Data":"5492c3b492410366b30ccbd88b777d3295f602c71595882d6363395a0216da21"}
May 25 12:32:20.090654 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:20.090538    2549 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-833d6-predictor-7dfb76b5f7-8rzb8"
May 25 12:32:20.097589 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:20.097379    2549 scope.go:117] "RemoveContainer" containerID="faa6f5119adba2a116a1c635be6ab083bb56996ace4397c26b6a8b3c1c8ae2ba"
May 25 12:32:20.101884 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:20.101855    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6264ef64-b6b2-43bd-bcfb-f281e7205194-proxy-tls\") pod \"6264ef64-b6b2-43bd-bcfb-f281e7205194\" (UID: \"6264ef64-b6b2-43bd-bcfb-f281e7205194\") "
May 25 12:32:20.102008 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:20.101909    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gl6wb\" (UniqueName: \"kubernetes.io/projected/40582ba8-ab3b-4046-985c-bcfb29eb0b82-kube-api-access-gl6wb\") pod \"40582ba8-ab3b-4046-985c-bcfb29eb0b82\" (UID: \"40582ba8-ab3b-4046-985c-bcfb29eb0b82\") "
May 25 12:32:20.102080 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:20.102004    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"error-404-isvc-833d6-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/40582ba8-ab3b-4046-985c-bcfb29eb0b82-error-404-isvc-833d6-kube-rbac-proxy-sar-config\") pod \"40582ba8-ab3b-4046-985c-bcfb29eb0b82\" (UID: \"40582ba8-ab3b-4046-985c-bcfb29eb0b82\") "
May 25 12:32:20.102080 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:20.102051    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ffpml\" (UniqueName: \"kubernetes.io/projected/6264ef64-b6b2-43bd-bcfb-f281e7205194-kube-api-access-ffpml\") pod \"6264ef64-b6b2-43bd-bcfb-f281e7205194\" (UID: \"6264ef64-b6b2-43bd-bcfb-f281e7205194\") "
May 25 12:32:20.102186 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:20.102085    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/40582ba8-ab3b-4046-985c-bcfb29eb0b82-proxy-tls\") pod \"40582ba8-ab3b-4046-985c-bcfb29eb0b82\" (UID: \"40582ba8-ab3b-4046-985c-bcfb29eb0b82\") "
May 25 12:32:20.102186 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:20.102125    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"success-200-isvc-833d6-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6264ef64-b6b2-43bd-bcfb-f281e7205194-success-200-isvc-833d6-kube-rbac-proxy-sar-config\") pod \"6264ef64-b6b2-43bd-bcfb-f281e7205194\" (UID: \"6264ef64-b6b2-43bd-bcfb-f281e7205194\") "
May 25 12:32:20.103537 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:20.103485    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40582ba8-ab3b-4046-985c-bcfb29eb0b82-error-404-isvc-833d6-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "error-404-isvc-833d6-kube-rbac-proxy-sar-config") pod "40582ba8-ab3b-4046-985c-bcfb29eb0b82" (UID: "40582ba8-ab3b-4046-985c-bcfb29eb0b82"). InnerVolumeSpecName "error-404-isvc-833d6-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 12:32:20.103898 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:20.102738    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6264ef64-b6b2-43bd-bcfb-f281e7205194-success-200-isvc-833d6-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "success-200-isvc-833d6-kube-rbac-proxy-sar-config") pod "6264ef64-b6b2-43bd-bcfb-f281e7205194" (UID: "6264ef64-b6b2-43bd-bcfb-f281e7205194"). InnerVolumeSpecName "success-200-isvc-833d6-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 12:32:20.106225 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:20.105445    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40582ba8-ab3b-4046-985c-bcfb29eb0b82-kube-api-access-gl6wb" (OuterVolumeSpecName: "kube-api-access-gl6wb") pod "40582ba8-ab3b-4046-985c-bcfb29eb0b82" (UID: "40582ba8-ab3b-4046-985c-bcfb29eb0b82"). InnerVolumeSpecName "kube-api-access-gl6wb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 12:32:20.106225 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:20.105688    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6264ef64-b6b2-43bd-bcfb-f281e7205194-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6264ef64-b6b2-43bd-bcfb-f281e7205194" (UID: "6264ef64-b6b2-43bd-bcfb-f281e7205194"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 12:32:20.106225 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:20.105946    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6264ef64-b6b2-43bd-bcfb-f281e7205194-kube-api-access-ffpml" (OuterVolumeSpecName: "kube-api-access-ffpml") pod "6264ef64-b6b2-43bd-bcfb-f281e7205194" (UID: "6264ef64-b6b2-43bd-bcfb-f281e7205194"). InnerVolumeSpecName "kube-api-access-ffpml". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 12:32:20.113282 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:20.112388    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40582ba8-ab3b-4046-985c-bcfb29eb0b82-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "40582ba8-ab3b-4046-985c-bcfb29eb0b82" (UID: "40582ba8-ab3b-4046-985c-bcfb29eb0b82"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 12:32:20.113892 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:20.113610    2549 scope.go:117] "RemoveContainer" containerID="70ae413c651d2a45a21f1ae79d8fa1013f9ba89f9aa51d29a41d72dde95b522c"
May 25 12:32:20.114170 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:32:20.114147    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70ae413c651d2a45a21f1ae79d8fa1013f9ba89f9aa51d29a41d72dde95b522c\": container with ID starting with 70ae413c651d2a45a21f1ae79d8fa1013f9ba89f9aa51d29a41d72dde95b522c not found: ID does not exist" containerID="70ae413c651d2a45a21f1ae79d8fa1013f9ba89f9aa51d29a41d72dde95b522c"
May 25 12:32:20.114436 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:20.114411    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70ae413c651d2a45a21f1ae79d8fa1013f9ba89f9aa51d29a41d72dde95b522c"} err="failed to get container status \"70ae413c651d2a45a21f1ae79d8fa1013f9ba89f9aa51d29a41d72dde95b522c\": rpc error: code = NotFound desc = could not find container \"70ae413c651d2a45a21f1ae79d8fa1013f9ba89f9aa51d29a41d72dde95b522c\": container with ID starting with 70ae413c651d2a45a21f1ae79d8fa1013f9ba89f9aa51d29a41d72dde95b522c not found: ID does not exist"
May 25 12:32:20.114485 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:20.114440    2549 scope.go:117] "RemoveContainer" containerID="faa6f5119adba2a116a1c635be6ab083bb56996ace4397c26b6a8b3c1c8ae2ba"
May 25 12:32:20.114730 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:32:20.114707    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"faa6f5119adba2a116a1c635be6ab083bb56996ace4397c26b6a8b3c1c8ae2ba\": container with ID starting with faa6f5119adba2a116a1c635be6ab083bb56996ace4397c26b6a8b3c1c8ae2ba not found: ID does not exist" containerID="faa6f5119adba2a116a1c635be6ab083bb56996ace4397c26b6a8b3c1c8ae2ba"
May 25 12:32:20.114788 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:20.114744    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"faa6f5119adba2a116a1c635be6ab083bb56996ace4397c26b6a8b3c1c8ae2ba"} err="failed to get container status \"faa6f5119adba2a116a1c635be6ab083bb56996ace4397c26b6a8b3c1c8ae2ba\": rpc error: code = NotFound desc = could not find container \"faa6f5119adba2a116a1c635be6ab083bb56996ace4397c26b6a8b3c1c8ae2ba\": container with ID starting with faa6f5119adba2a116a1c635be6ab083bb56996ace4397c26b6a8b3c1c8ae2ba not found: ID does not exist"
May 25 12:32:20.114788 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:20.114761    2549 scope.go:117] "RemoveContainer" containerID="29f4bcbe01f5e4f8d92e6577bb108f04749a8fa53c73783930db5dc1e1da6340"
May 25 12:32:20.122095 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:20.122070    2549 scope.go:117] "RemoveContainer" containerID="a425f8bbcb74f080c3fcf425bafc40db40ec3a200c76c9119be0657bcdd53c40"
May 25 12:32:20.129316 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:20.129297    2549 scope.go:117] "RemoveContainer" containerID="29f4bcbe01f5e4f8d92e6577bb108f04749a8fa53c73783930db5dc1e1da6340"
May 25 12:32:20.129579 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:32:20.129558    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29f4bcbe01f5e4f8d92e6577bb108f04749a8fa53c73783930db5dc1e1da6340\": container with ID starting with 29f4bcbe01f5e4f8d92e6577bb108f04749a8fa53c73783930db5dc1e1da6340 not found: ID does not exist" containerID="29f4bcbe01f5e4f8d92e6577bb108f04749a8fa53c73783930db5dc1e1da6340"
May 25 12:32:20.129656 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:20.129582    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29f4bcbe01f5e4f8d92e6577bb108f04749a8fa53c73783930db5dc1e1da6340"} err="failed to get container status \"29f4bcbe01f5e4f8d92e6577bb108f04749a8fa53c73783930db5dc1e1da6340\": rpc error: code = NotFound desc = could not find container \"29f4bcbe01f5e4f8d92e6577bb108f04749a8fa53c73783930db5dc1e1da6340\": container with ID starting with 29f4bcbe01f5e4f8d92e6577bb108f04749a8fa53c73783930db5dc1e1da6340 not found: ID does not exist"
May 25 12:32:20.129656 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:20.129598    2549 scope.go:117] "RemoveContainer" containerID="a425f8bbcb74f080c3fcf425bafc40db40ec3a200c76c9119be0657bcdd53c40"
May 25 12:32:20.129849 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:32:20.129834    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a425f8bbcb74f080c3fcf425bafc40db40ec3a200c76c9119be0657bcdd53c40\": container with ID starting with a425f8bbcb74f080c3fcf425bafc40db40ec3a200c76c9119be0657bcdd53c40 not found: ID does not exist" containerID="a425f8bbcb74f080c3fcf425bafc40db40ec3a200c76c9119be0657bcdd53c40"
May 25 12:32:20.129905 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:20.129852    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a425f8bbcb74f080c3fcf425bafc40db40ec3a200c76c9119be0657bcdd53c40"} err="failed to get container status \"a425f8bbcb74f080c3fcf425bafc40db40ec3a200c76c9119be0657bcdd53c40\": rpc error: code = NotFound desc = could not find container \"a425f8bbcb74f080c3fcf425bafc40db40ec3a200c76c9119be0657bcdd53c40\": container with ID starting with a425f8bbcb74f080c3fcf425bafc40db40ec3a200c76c9119be0657bcdd53c40 not found: ID does not exist"
May 25 12:32:20.203818 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:20.203767    2549 reconciler_common.go:299] "Volume detached for volume \"error-404-isvc-833d6-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/40582ba8-ab3b-4046-985c-bcfb29eb0b82-error-404-isvc-833d6-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
May 25 12:32:20.203818 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:20.203814    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ffpml\" (UniqueName: \"kubernetes.io/projected/6264ef64-b6b2-43bd-bcfb-f281e7205194-kube-api-access-ffpml\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
May 25 12:32:20.203818 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:20.203826    2549 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/40582ba8-ab3b-4046-985c-bcfb29eb0b82-proxy-tls\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
May 25 12:32:20.204033 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:20.203839    2549 reconciler_common.go:299] "Volume detached for volume \"success-200-isvc-833d6-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6264ef64-b6b2-43bd-bcfb-f281e7205194-success-200-isvc-833d6-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
May 25 12:32:20.204033 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:20.203849    2549 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6264ef64-b6b2-43bd-bcfb-f281e7205194-proxy-tls\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
May 25 12:32:20.204033 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:20.203858    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gl6wb\" (UniqueName: \"kubernetes.io/projected/40582ba8-ab3b-4046-985c-bcfb29eb0b82-kube-api-access-gl6wb\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
May 25 12:32:20.417323 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:20.417292    2549 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-833d6-predictor-65cfc8fdbb-xw7b6"]
May 25 12:32:20.427515 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:20.427490    2549 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-833d6-predictor-65cfc8fdbb-xw7b6"]
May 25 12:32:20.453606 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:20.453571    2549 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-833d6-predictor-7dfb76b5f7-8rzb8"]
May 25 12:32:20.455346 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:20.455327    2549 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-833d6-predictor-7dfb76b5f7-8rzb8"]
May 25 12:32:20.847871 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:20.847840    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="40582ba8-ab3b-4046-985c-bcfb29eb0b82" path="/var/lib/kubelet/pods/40582ba8-ab3b-4046-985c-bcfb29eb0b82/volumes"
May 25 12:32:20.848290 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:20.848276    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6264ef64-b6b2-43bd-bcfb-f281e7205194" path="/var/lib/kubelet/pods/6264ef64-b6b2-43bd-bcfb-f281e7205194/volumes"
May 25 12:32:23.963112 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:23.963069    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-bcff4-predictor-66c9fb5576-scwgx" podUID="81f25695-1e0c-4e1c-beda-e1d250bf364b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
May 25 12:32:24.087591 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:24.087560    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-acbca-predictor-687469f549-r5cwv"
May 25 12:32:24.088253 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:24.088226    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-acbca-predictor-59cffd4b6b-zwcgm"
May 25 12:32:24.088798 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:24.088429    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-acbca-predictor-687469f549-r5cwv" podUID="51175389-4295-493b-82bc-812c2e34c778" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
May 25 12:32:24.088999 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:24.088966    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-acbca-predictor-59cffd4b6b-zwcgm" podUID="f2f7c261-0a24-4aa1-b841-a92ba3120164" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
May 25 12:32:24.972910 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:24.972863    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-bcff4-predictor-55cbf45559-tw9hg" podUID="2d880044-04f1-4518-a423-305351e6122d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
May 25 12:32:33.963095 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:33.963062    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-bcff4-predictor-66c9fb5576-scwgx"
May 25 12:32:34.088529 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:34.088491    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-acbca-predictor-687469f549-r5cwv" podUID="51175389-4295-493b-82bc-812c2e34c778" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
May 25 12:32:34.088702 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:34.088632    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-acbca-predictor-59cffd4b6b-zwcgm" podUID="f2f7c261-0a24-4aa1-b841-a92ba3120164" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
May 25 12:32:34.972998 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:34.972959    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-bcff4-predictor-55cbf45559-tw9hg"
May 25 12:32:44.089098 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:44.089053    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-acbca-predictor-59cffd4b6b-zwcgm" podUID="f2f7c261-0a24-4aa1-b841-a92ba3120164" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
May 25 12:32:44.089588 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:44.089053    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-acbca-predictor-687469f549-r5cwv" podUID="51175389-4295-493b-82bc-812c2e34c778" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
May 25 12:32:54.089181 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:54.089133    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-acbca-predictor-59cffd4b6b-zwcgm" podUID="f2f7c261-0a24-4aa1-b841-a92ba3120164" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
May 25 12:32:54.089673 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:54.089135    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-acbca-predictor-687469f549-r5cwv" podUID="51175389-4295-493b-82bc-812c2e34c778" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
May 25 12:32:56.376072 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:56.376032    2549 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-155ee-predictor-6b9f6fd99b-qvjql"]
May 25 12:32:56.376560 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:56.376525    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6264ef64-b6b2-43bd-bcfb-f281e7205194" containerName="kube-rbac-proxy"
May 25 12:32:56.376560 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:56.376541    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="6264ef64-b6b2-43bd-bcfb-f281e7205194" containerName="kube-rbac-proxy"
May 25 12:32:56.376680 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:56.376564    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40582ba8-ab3b-4046-985c-bcfb29eb0b82" containerName="kserve-container"
May 25 12:32:56.376680 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:56.376571    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="40582ba8-ab3b-4046-985c-bcfb29eb0b82" containerName="kserve-container"
May 25 12:32:56.376680 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:56.376584    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6264ef64-b6b2-43bd-bcfb-f281e7205194" containerName="kserve-container"
May 25 12:32:56.376680 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:56.376590    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="6264ef64-b6b2-43bd-bcfb-f281e7205194" containerName="kserve-container"
May 25 12:32:56.376680 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:56.376596    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40582ba8-ab3b-4046-985c-bcfb29eb0b82" containerName="kube-rbac-proxy"
May 25 12:32:56.376680 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:56.376601    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="40582ba8-ab3b-4046-985c-bcfb29eb0b82" containerName="kube-rbac-proxy"
May 25 12:32:56.376680 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:56.376658    2549 memory_manager.go:356] "RemoveStaleState removing state" podUID="6264ef64-b6b2-43bd-bcfb-f281e7205194" containerName="kserve-container"
May 25 12:32:56.376680 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:56.376667    2549 memory_manager.go:356] "RemoveStaleState removing state" podUID="6264ef64-b6b2-43bd-bcfb-f281e7205194" containerName="kube-rbac-proxy"
May 25 12:32:56.376680 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:56.376674    2549 memory_manager.go:356] "RemoveStaleState removing state" podUID="40582ba8-ab3b-4046-985c-bcfb29eb0b82" containerName="kube-rbac-proxy"
May 25 12:32:56.376680 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:56.376684    2549 memory_manager.go:356] "RemoveStaleState removing state" podUID="40582ba8-ab3b-4046-985c-bcfb29eb0b82" containerName="kserve-container"
May 25 12:32:56.380152 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:56.380129    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-155ee-predictor-6b9f6fd99b-qvjql"
May 25 12:32:56.382743 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:56.382721    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-155ee-predictor-serving-cert\""
May 25 12:32:56.382870 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:56.382841    2549 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-155ee-kube-rbac-proxy-sar-config\""
May 25 12:32:56.390748 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:56.390722    2549 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-bcff4-predictor-66c9fb5576-scwgx"]
May 25 12:32:56.391121 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:56.391091    2549 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-bcff4-predictor-66c9fb5576-scwgx" podUID="81f25695-1e0c-4e1c-beda-e1d250bf364b" containerName="kserve-container" containerID="cri-o://296bb02511e80f95b944db635b8bb3734dfaa9b14243aac6888e4ec48fe137bd" gracePeriod=30
May 25 12:32:56.391236 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:56.391119    2549 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-bcff4-predictor-66c9fb5576-scwgx" podUID="81f25695-1e0c-4e1c-beda-e1d250bf364b" containerName="kube-rbac-proxy" containerID="cri-o://c99a1a1714ffb510c36a746aa316e3394fea4789b4c344b6bae10c17e62bd2ab" gracePeriod=30
May 25 12:32:56.394152 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:56.394123    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-155ee-predictor-6b9f6fd99b-qvjql"]
May 25 12:32:56.430036 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:56.430009    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/efcb96a7-a292-4595-880f-e7670a2d9928-proxy-tls\") pod \"success-200-isvc-155ee-predictor-6b9f6fd99b-qvjql\" (UID: \"efcb96a7-a292-4595-880f-e7670a2d9928\") " pod="kserve-ci-e2e-test/success-200-isvc-155ee-predictor-6b9f6fd99b-qvjql"
May 25 12:32:56.430161 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:56.430057    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqxnk\" (UniqueName: \"kubernetes.io/projected/efcb96a7-a292-4595-880f-e7670a2d9928-kube-api-access-sqxnk\") pod \"success-200-isvc-155ee-predictor-6b9f6fd99b-qvjql\" (UID: \"efcb96a7-a292-4595-880f-e7670a2d9928\") " pod="kserve-ci-e2e-test/success-200-isvc-155ee-predictor-6b9f6fd99b-qvjql"
May 25 12:32:56.430161 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:56.430144    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"success-200-isvc-155ee-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/efcb96a7-a292-4595-880f-e7670a2d9928-success-200-isvc-155ee-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-155ee-predictor-6b9f6fd99b-qvjql\" (UID: \"efcb96a7-a292-4595-880f-e7670a2d9928\") " pod="kserve-ci-e2e-test/success-200-isvc-155ee-predictor-6b9f6fd99b-qvjql"
May 25 12:32:56.483895 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:56.483856    2549 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-bcff4-predictor-55cbf45559-tw9hg"]
May 25 12:32:56.484293 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:56.484264    2549 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-bcff4-predictor-55cbf45559-tw9hg" podUID="2d880044-04f1-4518-a423-305351e6122d" containerName="kserve-container" containerID="cri-o://3e75fc58162d830e63e668212e51628174efc2fd7070b781edc7c1e6186cec68" gracePeriod=30
May 25 12:32:56.484909 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:56.484485    2549 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-bcff4-predictor-55cbf45559-tw9hg" podUID="2d880044-04f1-4518-a423-305351e6122d" containerName="kube-rbac-proxy" containerID="cri-o://db8d55dfc8f345cdbeef3dc7d22ac5e888074fbe1e4ad51c52c0d7ce21eaa564" gracePeriod=30
May 25 12:32:56.531348 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:56.531312    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/efcb96a7-a292-4595-880f-e7670a2d9928-proxy-tls\") pod \"success-200-isvc-155ee-predictor-6b9f6fd99b-qvjql\" (UID: \"efcb96a7-a292-4595-880f-e7670a2d9928\") " pod="kserve-ci-e2e-test/success-200-isvc-155ee-predictor-6b9f6fd99b-qvjql"
May 25 12:32:56.531504 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:56.531366    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-sqxnk\" (UniqueName: \"kubernetes.io/projected/efcb96a7-a292-4595-880f-e7670a2d9928-kube-api-access-sqxnk\") pod \"success-200-isvc-155ee-predictor-6b9f6fd99b-qvjql\" (UID: \"efcb96a7-a292-4595-880f-e7670a2d9928\") " pod="kserve-ci-e2e-test/success-200-isvc-155ee-predictor-6b9f6fd99b-qvjql"
May 25 12:32:56.531504 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:56.531416    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"success-200-isvc-155ee-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/efcb96a7-a292-4595-880f-e7670a2d9928-success-200-isvc-155ee-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-155ee-predictor-6b9f6fd99b-qvjql\" (UID: \"efcb96a7-a292-4595-880f-e7670a2d9928\") " pod="kserve-ci-e2e-test/success-200-isvc-155ee-predictor-6b9f6fd99b-qvjql"
May 25 12:32:56.532046 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:56.532026    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"success-200-isvc-155ee-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/efcb96a7-a292-4595-880f-e7670a2d9928-success-200-isvc-155ee-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-155ee-predictor-6b9f6fd99b-qvjql\" (UID: \"efcb96a7-a292-4595-880f-e7670a2d9928\") " pod="kserve-ci-e2e-test/success-200-isvc-155ee-predictor-6b9f6fd99b-qvjql"
May 25 12:32:56.534016 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:56.533991    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/efcb96a7-a292-4595-880f-e7670a2d9928-proxy-tls\") pod \"success-200-isvc-155ee-predictor-6b9f6fd99b-qvjql\" (UID: \"efcb96a7-a292-4595-880f-e7670a2d9928\") " pod="kserve-ci-e2e-test/success-200-isvc-155ee-predictor-6b9f6fd99b-qvjql"
May 25 12:32:56.536713 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:56.536689    2549 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-155ee-predictor-57c99b6f99-mtr6n"]
May 25 12:32:56.541396 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:56.541374    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-155ee-predictor-57c99b6f99-mtr6n"
May 25 12:32:56.541743 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:56.541716    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqxnk\" (UniqueName: \"kubernetes.io/projected/efcb96a7-a292-4595-880f-e7670a2d9928-kube-api-access-sqxnk\") pod \"success-200-isvc-155ee-predictor-6b9f6fd99b-qvjql\" (UID: \"efcb96a7-a292-4595-880f-e7670a2d9928\") " pod="kserve-ci-e2e-test/success-200-isvc-155ee-predictor-6b9f6fd99b-qvjql"
May 25 12:32:56.544556 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:56.544528    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-155ee-predictor-serving-cert\""
May 25 12:32:56.544673 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:56.544593    2549 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-155ee-kube-rbac-proxy-sar-config\""
May 25 12:32:56.551288 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:56.551265    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-155ee-predictor-57c99b6f99-mtr6n"]
May 25 12:32:56.632601 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:56.632522    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3f43e011-dd0d-4717-8d72-3e2882bb8650-proxy-tls\") pod \"error-404-isvc-155ee-predictor-57c99b6f99-mtr6n\" (UID: \"3f43e011-dd0d-4717-8d72-3e2882bb8650\") " pod="kserve-ci-e2e-test/error-404-isvc-155ee-predictor-57c99b6f99-mtr6n"
May 25 12:32:56.632601 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:56.632567    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxjlh\" (UniqueName: \"kubernetes.io/projected/3f43e011-dd0d-4717-8d72-3e2882bb8650-kube-api-access-dxjlh\") pod \"error-404-isvc-155ee-predictor-57c99b6f99-mtr6n\" (UID: \"3f43e011-dd0d-4717-8d72-3e2882bb8650\") " pod="kserve-ci-e2e-test/error-404-isvc-155ee-predictor-57c99b6f99-mtr6n"
May 25 12:32:56.632768 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:56.632685    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"error-404-isvc-155ee-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3f43e011-dd0d-4717-8d72-3e2882bb8650-error-404-isvc-155ee-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-155ee-predictor-57c99b6f99-mtr6n\" (UID: \"3f43e011-dd0d-4717-8d72-3e2882bb8650\") " pod="kserve-ci-e2e-test/error-404-isvc-155ee-predictor-57c99b6f99-mtr6n"
May 25 12:32:56.694864 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:56.694831    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-155ee-predictor-6b9f6fd99b-qvjql"
May 25 12:32:56.734172 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:56.734069    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"error-404-isvc-155ee-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3f43e011-dd0d-4717-8d72-3e2882bb8650-error-404-isvc-155ee-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-155ee-predictor-57c99b6f99-mtr6n\" (UID: \"3f43e011-dd0d-4717-8d72-3e2882bb8650\") " pod="kserve-ci-e2e-test/error-404-isvc-155ee-predictor-57c99b6f99-mtr6n"
May 25 12:32:56.734377 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:56.734182    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3f43e011-dd0d-4717-8d72-3e2882bb8650-proxy-tls\") pod \"error-404-isvc-155ee-predictor-57c99b6f99-mtr6n\" (UID: \"3f43e011-dd0d-4717-8d72-3e2882bb8650\") " pod="kserve-ci-e2e-test/error-404-isvc-155ee-predictor-57c99b6f99-mtr6n"
May 25 12:32:56.734377 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:56.734232    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-dxjlh\" (UniqueName: \"kubernetes.io/projected/3f43e011-dd0d-4717-8d72-3e2882bb8650-kube-api-access-dxjlh\") pod \"error-404-isvc-155ee-predictor-57c99b6f99-mtr6n\" (UID: \"3f43e011-dd0d-4717-8d72-3e2882bb8650\") " pod="kserve-ci-e2e-test/error-404-isvc-155ee-predictor-57c99b6f99-mtr6n"
May 25 12:32:56.734977 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:56.734950    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"error-404-isvc-155ee-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3f43e011-dd0d-4717-8d72-3e2882bb8650-error-404-isvc-155ee-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-155ee-predictor-57c99b6f99-mtr6n\" (UID: \"3f43e011-dd0d-4717-8d72-3e2882bb8650\") " pod="kserve-ci-e2e-test/error-404-isvc-155ee-predictor-57c99b6f99-mtr6n"
May 25 12:32:56.736619 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:56.736597    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3f43e011-dd0d-4717-8d72-3e2882bb8650-proxy-tls\") pod \"error-404-isvc-155ee-predictor-57c99b6f99-mtr6n\" (UID: \"3f43e011-dd0d-4717-8d72-3e2882bb8650\") " pod="kserve-ci-e2e-test/error-404-isvc-155ee-predictor-57c99b6f99-mtr6n"
May 25 12:32:56.750356 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:56.750310    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxjlh\" (UniqueName: \"kubernetes.io/projected/3f43e011-dd0d-4717-8d72-3e2882bb8650-kube-api-access-dxjlh\") pod \"error-404-isvc-155ee-predictor-57c99b6f99-mtr6n\" (UID: \"3f43e011-dd0d-4717-8d72-3e2882bb8650\") " pod="kserve-ci-e2e-test/error-404-isvc-155ee-predictor-57c99b6f99-mtr6n"
May 25 12:32:56.840722 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:56.840686    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-155ee-predictor-6b9f6fd99b-qvjql"]
May 25 12:32:56.842238 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:32:56.842184    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefcb96a7_a292_4595_880f_e7670a2d9928.slice/crio-a0224c025846d15e1d902708e01ebfd4bde6e06915bbc7ca35d02d369c221582 WatchSource:0}: Error finding container a0224c025846d15e1d902708e01ebfd4bde6e06915bbc7ca35d02d369c221582: Status 404 returned error can't find the container with id a0224c025846d15e1d902708e01ebfd4bde6e06915bbc7ca35d02d369c221582
May 25 12:32:56.865594 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:56.865563    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-155ee-predictor-57c99b6f99-mtr6n"
May 25 12:32:57.016647 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:57.016615    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-155ee-predictor-57c99b6f99-mtr6n"]
May 25 12:32:57.019264 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:32:57.019225    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f43e011_dd0d_4717_8d72_3e2882bb8650.slice/crio-30076f522f43210a25ca0e03f272eb59d2dfd223c31e20a5f3dcac7cc3ba3ee9 WatchSource:0}: Error finding container 30076f522f43210a25ca0e03f272eb59d2dfd223c31e20a5f3dcac7cc3ba3ee9: Status 404 returned error can't find the container with id 30076f522f43210a25ca0e03f272eb59d2dfd223c31e20a5f3dcac7cc3ba3ee9
May 25 12:32:57.228062 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:57.228018    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-155ee-predictor-6b9f6fd99b-qvjql" event={"ID":"efcb96a7-a292-4595-880f-e7670a2d9928","Type":"ContainerStarted","Data":"35cba0cdaa0151f32daa59468dfafa895877d26e02ea9dae0e2bebe8e6b2c36a"}
May 25 12:32:57.228273 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:57.228073    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-155ee-predictor-6b9f6fd99b-qvjql" event={"ID":"efcb96a7-a292-4595-880f-e7670a2d9928","Type":"ContainerStarted","Data":"705521478faa138c09b52e9f4f8bc7ae915575fc304e739b0110b0137fedee21"}
May 25 12:32:57.228273 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:57.228090    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-155ee-predictor-6b9f6fd99b-qvjql" event={"ID":"efcb96a7-a292-4595-880f-e7670a2d9928","Type":"ContainerStarted","Data":"a0224c025846d15e1d902708e01ebfd4bde6e06915bbc7ca35d02d369c221582"}
May 25 12:32:57.228273 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:57.228136    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-155ee-predictor-6b9f6fd99b-qvjql"
May 25 12:32:57.229633 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:57.229605    2549 generic.go:358] "Generic (PLEG): container finished" podID="81f25695-1e0c-4e1c-beda-e1d250bf364b" containerID="c99a1a1714ffb510c36a746aa316e3394fea4789b4c344b6bae10c17e62bd2ab" exitCode=2
May 25 12:32:57.229764 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:57.229672    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-bcff4-predictor-66c9fb5576-scwgx" event={"ID":"81f25695-1e0c-4e1c-beda-e1d250bf364b","Type":"ContainerDied","Data":"c99a1a1714ffb510c36a746aa316e3394fea4789b4c344b6bae10c17e62bd2ab"}
May 25 12:32:57.231179 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:57.231158    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-155ee-predictor-57c99b6f99-mtr6n" event={"ID":"3f43e011-dd0d-4717-8d72-3e2882bb8650","Type":"ContainerStarted","Data":"b234d4a0306944df8548d229741e8cbfa6101f667f0b33781f772e33652514fd"}
May 25 12:32:57.231283 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:57.231188    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-155ee-predictor-57c99b6f99-mtr6n" event={"ID":"3f43e011-dd0d-4717-8d72-3e2882bb8650","Type":"ContainerStarted","Data":"a6bc0a84961219878a56ca53f42d73fb8f31d3a695ce162918376d483eb1a9d1"}
May 25 12:32:57.231283 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:57.231220    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-155ee-predictor-57c99b6f99-mtr6n" event={"ID":"3f43e011-dd0d-4717-8d72-3e2882bb8650","Type":"ContainerStarted","Data":"30076f522f43210a25ca0e03f272eb59d2dfd223c31e20a5f3dcac7cc3ba3ee9"}
May 25 12:32:57.231368 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:57.231313    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-155ee-predictor-57c99b6f99-mtr6n"
May 25 12:32:57.232673 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:57.232652    2549 generic.go:358] "Generic (PLEG): container finished" podID="2d880044-04f1-4518-a423-305351e6122d" containerID="db8d55dfc8f345cdbeef3dc7d22ac5e888074fbe1e4ad51c52c0d7ce21eaa564" exitCode=2
May 25 12:32:57.232757 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:57.232706    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-bcff4-predictor-55cbf45559-tw9hg" event={"ID":"2d880044-04f1-4518-a423-305351e6122d","Type":"ContainerDied","Data":"db8d55dfc8f345cdbeef3dc7d22ac5e888074fbe1e4ad51c52c0d7ce21eaa564"}
May 25 12:32:57.250772 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:57.250720    2549 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-155ee-predictor-6b9f6fd99b-qvjql" podStartSLOduration=1.250705662 podStartE2EDuration="1.250705662s" podCreationTimestamp="2026-05-25 12:32:56 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 12:32:57.250434559 +0000 UTC m=+1180.966249850" watchObservedRunningTime="2026-05-25 12:32:57.250705662 +0000 UTC m=+1180.966520948"
May 25 12:32:57.272702 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:57.272653    2549 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-155ee-predictor-57c99b6f99-mtr6n" podStartSLOduration=1.272635649 podStartE2EDuration="1.272635649s" podCreationTimestamp="2026-05-25 12:32:56 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 12:32:57.27154111 +0000 UTC m=+1180.987356401" watchObservedRunningTime="2026-05-25 12:32:57.272635649 +0000 UTC m=+1180.988450936"
May 25 12:32:58.246787 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:58.246748    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-155ee-predictor-57c99b6f99-mtr6n"
May 25 12:32:58.246787 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:58.246788    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-155ee-predictor-6b9f6fd99b-qvjql"
May 25 12:32:58.247779 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:58.247750    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-155ee-predictor-6b9f6fd99b-qvjql" podUID="efcb96a7-a292-4595-880f-e7670a2d9928" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
May 25 12:32:58.247919 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:58.247753    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-155ee-predictor-57c99b6f99-mtr6n" podUID="3f43e011-dd0d-4717-8d72-3e2882bb8650" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
May 25 12:32:58.957593 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:58.957550    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-bcff4-predictor-66c9fb5576-scwgx" podUID="81f25695-1e0c-4e1c-beda-e1d250bf364b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.40:8643/healthz\": dial tcp 10.134.0.40:8643: connect: connection refused"
May 25 12:32:59.252154 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:59.252061    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-155ee-predictor-6b9f6fd99b-qvjql" podUID="efcb96a7-a292-4595-880f-e7670a2d9928" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
May 25 12:32:59.252754 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:59.252724    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-155ee-predictor-57c99b6f99-mtr6n" podUID="3f43e011-dd0d-4717-8d72-3e2882bb8650" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
May 25 12:32:59.968076 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:32:59.968031    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-bcff4-predictor-55cbf45559-tw9hg" podUID="2d880044-04f1-4518-a423-305351e6122d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.41:8643/healthz\": dial tcp 10.134.0.41:8643: connect: connection refused"
May 25 12:33:00.149640 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:33:00.149614    2549 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-bcff4-predictor-66c9fb5576-scwgx"
May 25 12:33:00.256856 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:33:00.256772    2549 generic.go:358] "Generic (PLEG): container finished" podID="81f25695-1e0c-4e1c-beda-e1d250bf364b" containerID="296bb02511e80f95b944db635b8bb3734dfaa9b14243aac6888e4ec48fe137bd" exitCode=0
May 25 12:33:00.256856 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:33:00.256832    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-bcff4-predictor-66c9fb5576-scwgx" event={"ID":"81f25695-1e0c-4e1c-beda-e1d250bf364b","Type":"ContainerDied","Data":"296bb02511e80f95b944db635b8bb3734dfaa9b14243aac6888e4ec48fe137bd"}
May 25 12:33:00.256856 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:33:00.256841    2549 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-bcff4-predictor-66c9fb5576-scwgx"
May 25 12:33:00.256856 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:33:00.256859    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-bcff4-predictor-66c9fb5576-scwgx" event={"ID":"81f25695-1e0c-4e1c-beda-e1d250bf364b","Type":"ContainerDied","Data":"63d5166f345ec78cae7240c5adf87f9497fc784892b8e9a1ed8c01457f056ee6"}
May 25 12:33:00.257414 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:33:00.256875    2549 scope.go:117] "RemoveContainer" containerID="c99a1a1714ffb510c36a746aa316e3394fea4789b4c344b6bae10c17e62bd2ab"
May 25 12:33:00.264921 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:33:00.264893    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"success-200-isvc-bcff4-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/81f25695-1e0c-4e1c-beda-e1d250bf364b-success-200-isvc-bcff4-kube-rbac-proxy-sar-config\") pod \"81f25695-1e0c-4e1c-beda-e1d250bf364b\" (UID: \"81f25695-1e0c-4e1c-beda-e1d250bf364b\") "
May 25 12:33:00.265046 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:33:00.264939    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bzswg\" (UniqueName: \"kubernetes.io/projected/81f25695-1e0c-4e1c-beda-e1d250bf364b-kube-api-access-bzswg\") pod \"81f25695-1e0c-4e1c-beda-e1d250bf364b\" (UID: \"81f25695-1e0c-4e1c-beda-e1d250bf364b\") "
May 25 12:33:00.265046 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:33:00.264963    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/81f25695-1e0c-4e1c-beda-e1d250bf364b-proxy-tls\") pod \"81f25695-1e0c-4e1c-beda-e1d250bf364b\" (UID: \"81f25695-1e0c-4e1c-beda-e1d250bf364b\") "
May 25 12:33:00.265276 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:33:00.265254    2549 scope.go:117] "RemoveContainer" containerID="296bb02511e80f95b944db635b8bb3734dfaa9b14243aac6888e4ec48fe137bd"
May 25 12:33:00.265346 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:33:00.265308    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81f25695-1e0c-4e1c-beda-e1d250bf364b-success-200-isvc-bcff4-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "success-200-isvc-bcff4-kube-rbac-proxy-sar-config") pod "81f25695-1e0c-4e1c-beda-e1d250bf364b" (UID: "81f25695-1e0c-4e1c-beda-e1d250bf364b"). InnerVolumeSpecName "success-200-isvc-bcff4-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 12:33:00.265452 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:33:00.265428    2549 reconciler_common.go:299] "Volume detached for volume \"success-200-isvc-bcff4-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/81f25695-1e0c-4e1c-beda-e1d250bf364b-success-200-isvc-bcff4-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
May 25 12:33:00.267169 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:33:00.267144    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81f25695-1e0c-4e1c-beda-e1d250bf364b-kube-api-access-bzswg" (OuterVolumeSpecName: "kube-api-access-bzswg") pod "81f25695-1e0c-4e1c-beda-e1d250bf364b" (UID: "81f25695-1e0c-4e1c-beda-e1d250bf364b"). InnerVolumeSpecName "kube-api-access-bzswg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 12:33:00.267169 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:33:00.267153    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/81f25695-1e0c-4e1c-beda-e1d250bf364b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "81f25695-1e0c-4e1c-beda-e1d250bf364b" (UID: "81f25695-1e0c-4e1c-beda-e1d250bf364b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 12:33:00.284883 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:33:00.284860    2549 scope.go:117] "RemoveContainer" containerID="c99a1a1714ffb510c36a746aa316e3394fea4789b4c344b6bae10c17e62bd2ab"
May 25 12:33:00.285172 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:33:00.285149    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c99a1a1714ffb510c36a746aa316e3394fea4789b4c344b6bae10c17e62bd2ab\": container with ID starting with c99a1a1714ffb510c36a746aa316e3394fea4789b4c344b6bae10c17e62bd2ab not found: ID does not exist" containerID="c99a1a1714ffb510c36a746aa316e3394fea4789b4c344b6bae10c17e62bd2ab"
May 25 12:33:00.285295 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:33:00.285183    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c99a1a1714ffb510c36a746aa316e3394fea4789b4c344b6bae10c17e62bd2ab"} err="failed to get container status \"c99a1a1714ffb510c36a746aa316e3394fea4789b4c344b6bae10c17e62bd2ab\": rpc error: code = NotFound desc = could not find container \"c99a1a1714ffb510c36a746aa316e3394fea4789b4c344b6bae10c17e62bd2ab\": container with ID starting with c99a1a1714ffb510c36a746aa316e3394fea4789b4c344b6bae10c17e62bd2ab not found: ID does not exist"
May 25 12:33:00.285295 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:33:00.285233    2549 scope.go:117] "RemoveContainer" containerID="296bb02511e80f95b944db635b8bb3734dfaa9b14243aac6888e4ec48fe137bd"
May 25 12:33:00.285577 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:33:00.285549    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"296bb02511e80f95b944db635b8bb3734dfaa9b14243aac6888e4ec48fe137bd\": container with ID starting with 296bb02511e80f95b944db635b8bb3734dfaa9b14243aac6888e4ec48fe137bd not found: ID does not exist" containerID="296bb02511e80f95b944db635b8bb3734dfaa9b14243aac6888e4ec48fe137bd"
May 25 12:33:00.285667 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:33:00.285585    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"296bb02511e80f95b944db635b8bb3734dfaa9b14243aac6888e4ec48fe137bd"} err="failed to get container status \"296bb02511e80f95b944db635b8bb3734dfaa9b14243aac6888e4ec48fe137bd\": rpc error: code = NotFound desc = could not find container \"296bb02511e80f95b944db635b8bb3734dfaa9b14243aac6888e4ec48fe137bd\": container with ID starting with 296bb02511e80f95b944db635b8bb3734dfaa9b14243aac6888e4ec48fe137bd not found: ID does not exist"
May 25 12:33:00.366542 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:33:00.366506    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bzswg\" (UniqueName: \"kubernetes.io/projected/81f25695-1e0c-4e1c-beda-e1d250bf364b-kube-api-access-bzswg\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
May 25 12:33:00.366542 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:33:00.366541    2549 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/81f25695-1e0c-4e1c-beda-e1d250bf364b-proxy-tls\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
May 25 12:33:00.583186 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:33:00.583157    2549 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-bcff4-predictor-66c9fb5576-scwgx"]
May 25 12:33:00.588713 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:33:00.588671    2549 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-bcff4-predictor-66c9fb5576-scwgx"]
May 25 12:33:00.848868 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:33:00.848837    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="81f25695-1e0c-4e1c-beda-e1d250bf364b" path="/var/lib/kubelet/pods/81f25695-1e0c-4e1c-beda-e1d250bf364b/volumes"
May 25 12:33:00.912478 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:33:00.912456    2549 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-bcff4-predictor-55cbf45559-tw9hg"
May 25 12:33:00.971782 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:33:00.971737    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"error-404-isvc-bcff4-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2d880044-04f1-4518-a423-305351e6122d-error-404-isvc-bcff4-kube-rbac-proxy-sar-config\") pod \"2d880044-04f1-4518-a423-305351e6122d\" (UID: \"2d880044-04f1-4518-a423-305351e6122d\") "
May 25 12:33:00.971973 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:33:00.971816    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2d880044-04f1-4518-a423-305351e6122d-proxy-tls\") pod \"2d880044-04f1-4518-a423-305351e6122d\" (UID: \"2d880044-04f1-4518-a423-305351e6122d\") "
May 25 12:33:00.971973 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:33:00.971856    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vh7ss\" (UniqueName: \"kubernetes.io/projected/2d880044-04f1-4518-a423-305351e6122d-kube-api-access-vh7ss\") pod \"2d880044-04f1-4518-a423-305351e6122d\" (UID: \"2d880044-04f1-4518-a423-305351e6122d\") "
May 25 12:33:00.972098 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:33:00.972080    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d880044-04f1-4518-a423-305351e6122d-error-404-isvc-bcff4-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "error-404-isvc-bcff4-kube-rbac-proxy-sar-config") pod "2d880044-04f1-4518-a423-305351e6122d" (UID: "2d880044-04f1-4518-a423-305351e6122d"). InnerVolumeSpecName "error-404-isvc-bcff4-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 12:33:00.973960 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:33:00.973933    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d880044-04f1-4518-a423-305351e6122d-kube-api-access-vh7ss" (OuterVolumeSpecName: "kube-api-access-vh7ss") pod "2d880044-04f1-4518-a423-305351e6122d" (UID: "2d880044-04f1-4518-a423-305351e6122d"). InnerVolumeSpecName "kube-api-access-vh7ss". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 12:33:00.974062 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:33:00.973972    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d880044-04f1-4518-a423-305351e6122d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2d880044-04f1-4518-a423-305351e6122d" (UID: "2d880044-04f1-4518-a423-305351e6122d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 12:33:01.072890 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:33:01.072850    2549 reconciler_common.go:299] "Volume detached for volume \"error-404-isvc-bcff4-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2d880044-04f1-4518-a423-305351e6122d-error-404-isvc-bcff4-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
May 25 12:33:01.072890 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:33:01.072881    2549 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2d880044-04f1-4518-a423-305351e6122d-proxy-tls\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
May 25 12:33:01.072890 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:33:01.072897    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vh7ss\" (UniqueName: \"kubernetes.io/projected/2d880044-04f1-4518-a423-305351e6122d-kube-api-access-vh7ss\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
May 25 12:33:01.263413 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:33:01.263323    2549 generic.go:358] "Generic (PLEG): container finished" podID="2d880044-04f1-4518-a423-305351e6122d" containerID="3e75fc58162d830e63e668212e51628174efc2fd7070b781edc7c1e6186cec68" exitCode=0
May 25 12:33:01.263413 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:33:01.263354    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-bcff4-predictor-55cbf45559-tw9hg" event={"ID":"2d880044-04f1-4518-a423-305351e6122d","Type":"ContainerDied","Data":"3e75fc58162d830e63e668212e51628174efc2fd7070b781edc7c1e6186cec68"}
May 25 12:33:01.263413 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:33:01.263397    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-bcff4-predictor-55cbf45559-tw9hg" event={"ID":"2d880044-04f1-4518-a423-305351e6122d","Type":"ContainerDied","Data":"ebd2fecd9bea773dbbb7a359adf48b1f883599c540c39df6a246c3b3f84cfe5a"}
May 25 12:33:01.263413 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:33:01.263404    2549 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-bcff4-predictor-55cbf45559-tw9hg"
May 25 12:33:01.263413 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:33:01.263414    2549 scope.go:117] "RemoveContainer" containerID="db8d55dfc8f345cdbeef3dc7d22ac5e888074fbe1e4ad51c52c0d7ce21eaa564"
May 25 12:33:01.272365 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:33:01.272342    2549 scope.go:117] "RemoveContainer" containerID="3e75fc58162d830e63e668212e51628174efc2fd7070b781edc7c1e6186cec68"
May 25 12:33:01.279978 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:33:01.279959    2549 scope.go:117] "RemoveContainer" containerID="db8d55dfc8f345cdbeef3dc7d22ac5e888074fbe1e4ad51c52c0d7ce21eaa564"
May 25 12:33:01.280247 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:33:01.280225    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db8d55dfc8f345cdbeef3dc7d22ac5e888074fbe1e4ad51c52c0d7ce21eaa564\": container with ID starting with db8d55dfc8f345cdbeef3dc7d22ac5e888074fbe1e4ad51c52c0d7ce21eaa564 not found: ID does not exist" containerID="db8d55dfc8f345cdbeef3dc7d22ac5e888074fbe1e4ad51c52c0d7ce21eaa564"
May 25 12:33:01.280318 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:33:01.280256    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db8d55dfc8f345cdbeef3dc7d22ac5e888074fbe1e4ad51c52c0d7ce21eaa564"} err="failed to get container status \"db8d55dfc8f345cdbeef3dc7d22ac5e888074fbe1e4ad51c52c0d7ce21eaa564\": rpc error: code = NotFound desc = could not find container \"db8d55dfc8f345cdbeef3dc7d22ac5e888074fbe1e4ad51c52c0d7ce21eaa564\": container with ID starting with db8d55dfc8f345cdbeef3dc7d22ac5e888074fbe1e4ad51c52c0d7ce21eaa564 not found: ID does not exist"
May 25 12:33:01.280318 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:33:01.280273    2549 scope.go:117] "RemoveContainer" containerID="3e75fc58162d830e63e668212e51628174efc2fd7070b781edc7c1e6186cec68"
May 25 12:33:01.280506 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:33:01.280492    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e75fc58162d830e63e668212e51628174efc2fd7070b781edc7c1e6186cec68\": container with ID starting with 3e75fc58162d830e63e668212e51628174efc2fd7070b781edc7c1e6186cec68 not found: ID does not exist" containerID="3e75fc58162d830e63e668212e51628174efc2fd7070b781edc7c1e6186cec68"
May 25 12:33:01.280550 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:33:01.280510    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e75fc58162d830e63e668212e51628174efc2fd7070b781edc7c1e6186cec68"} err="failed to get container status \"3e75fc58162d830e63e668212e51628174efc2fd7070b781edc7c1e6186cec68\": rpc error: code = NotFound desc = could not find container \"3e75fc58162d830e63e668212e51628174efc2fd7070b781edc7c1e6186cec68\": container with ID starting with 3e75fc58162d830e63e668212e51628174efc2fd7070b781edc7c1e6186cec68 not found: ID does not exist"
May 25 12:33:01.289089 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:33:01.289059    2549 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-bcff4-predictor-55cbf45559-tw9hg"]
May 25 12:33:01.294619 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:33:01.294597    2549 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-bcff4-predictor-55cbf45559-tw9hg"]
May 25 12:33:02.848561 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:33:02.848528    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2d880044-04f1-4518-a423-305351e6122d" path="/var/lib/kubelet/pods/2d880044-04f1-4518-a423-305351e6122d/volumes"
May 25 12:33:04.089329 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:33:04.089267    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-acbca-predictor-59cffd4b6b-zwcgm" podUID="f2f7c261-0a24-4aa1-b841-a92ba3120164" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
May 25 12:33:04.089933 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:33:04.089911    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-acbca-predictor-687469f549-r5cwv"
May 25 12:33:04.256815 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:33:04.256775    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-155ee-predictor-6b9f6fd99b-qvjql"
May 25 12:33:04.257358 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:33:04.257318    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-155ee-predictor-6b9f6fd99b-qvjql" podUID="efcb96a7-a292-4595-880f-e7670a2d9928" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
May 25 12:33:04.259135 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:33:04.259113    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-155ee-predictor-57c99b6f99-mtr6n"
May 25 12:33:04.259669 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:33:04.259642    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-155ee-predictor-57c99b6f99-mtr6n" podUID="3f43e011-dd0d-4717-8d72-3e2882bb8650" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
May 25 12:33:14.089390 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:33:14.089357    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-acbca-predictor-59cffd4b6b-zwcgm"
May 25 12:33:14.258887 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:33:14.258839    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-155ee-predictor-6b9f6fd99b-qvjql" podUID="efcb96a7-a292-4595-880f-e7670a2d9928" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
May 25 12:33:14.259544 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:33:14.259500    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-155ee-predictor-57c99b6f99-mtr6n" podUID="3f43e011-dd0d-4717-8d72-3e2882bb8650" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
May 25 12:33:24.257382 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:33:24.257343    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-155ee-predictor-6b9f6fd99b-qvjql" podUID="efcb96a7-a292-4595-880f-e7670a2d9928" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
May 25 12:33:24.259702 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:33:24.259665    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-155ee-predictor-57c99b6f99-mtr6n" podUID="3f43e011-dd0d-4717-8d72-3e2882bb8650" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
May 25 12:33:34.257859 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:33:34.257822    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-155ee-predictor-6b9f6fd99b-qvjql" podUID="efcb96a7-a292-4595-880f-e7670a2d9928" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
May 25 12:33:34.260117 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:33:34.260091    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-155ee-predictor-57c99b6f99-mtr6n" podUID="3f43e011-dd0d-4717-8d72-3e2882bb8650" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
May 25 12:33:36.202649 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:33:36.202614    2549 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-acbca-predictor-687469f549-r5cwv"]
May 25 12:33:36.203054 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:33:36.202907    2549 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-acbca-predictor-687469f549-r5cwv" podUID="51175389-4295-493b-82bc-812c2e34c778" containerName="kserve-container" containerID="cri-o://1b1e92734dd67ad756c33fbf5939df977d7ac842685b56521b694534adec3c39" gracePeriod=30
May 25 12:33:36.203054 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:33:36.202941    2549 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-acbca-predictor-687469f549-r5cwv" podUID="51175389-4295-493b-82bc-812c2e34c778" containerName="kube-rbac-proxy" containerID="cri-o://08e7dd83266cb5136a1b42ee1de5c01799d760726127fed0fd7c845cd9a8d082" gracePeriod=30
May 25 12:33:36.282470 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:33:36.282432    2549 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-acbca-predictor-59cffd4b6b-zwcgm"]
May 25 12:33:36.282786 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:33:36.282761    2549 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-acbca-predictor-59cffd4b6b-zwcgm" podUID="f2f7c261-0a24-4aa1-b841-a92ba3120164" containerName="kserve-container" containerID="cri-o://12872b3e036cc6bd133452554c75f0b39f480650f7eb3f21a3ef7f4fbea6ca53" gracePeriod=30
May 25 12:33:36.282944 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:33:36.282878    2549 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-acbca-predictor-59cffd4b6b-zwcgm" podUID="f2f7c261-0a24-4aa1-b841-a92ba3120164" containerName="kube-rbac-proxy" containerID="cri-o://e02216eab1f2133c978c0fc004ba611ed778f495507f9d3aeb5b435b6bb7acb0" gracePeriod=30
May 25 12:33:36.298917 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:33:36.298891    2549 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-4d75e-predictor-98c496fb-m8ddw"]
May 25 12:33:36.299335 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:33:36.299322    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81f25695-1e0c-4e1c-beda-e1d250bf364b" containerName="kserve-container"
May 25 12:33:36.299391 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:33:36.299337    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="81f25695-1e0c-4e1c-beda-e1d250bf364b" containerName="kserve-container"
May 25 12:33:36.299391 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:33:36.299347    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d880044-04f1-4518-a423-305351e6122d" containerName="kserve-container"
May 25 12:33:36.299391 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:33:36.299353    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d880044-04f1-4518-a423-305351e6122d" containerName="kserve-container"
May 25 12:33:36.299391 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:33:36.299368    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d880044-04f1-4518-a423-305351e6122d" containerName="kube-rbac-proxy"
May 25 12:33:36.299391 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:33:36.299374    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d880044-04f1-4518-a423-305351e6122d" containerName="kube-rbac-proxy"
May 25 12:33:36.299391 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:33:36.299385    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81f25695-1e0c-4e1c-beda-e1d250bf364b" containerName="kube-rbac-proxy"
May 25 12:33:36.299391 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:33:36.299390    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="81f25695-1e0c-4e1c-beda-e1d250bf364b" containerName="kube-rbac-proxy"
May 25 12:33:36.299623 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:33:36.299445    2549 memory_manager.go:356] "RemoveStaleState removing state" podUID="81f25695-1e0c-4e1c-beda-e1d250bf364b" containerName="kserve-container"
May 25 12:33:36.299623 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:33:36.299459    2549 memory_manager.go:356] "RemoveStaleState removing state" podUID="2d880044-04f1-4518-a423-305351e6122d" containerName="kube-rbac-proxy"
May 25 12:33:36.299623 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:33:36.299465    2549 memory_manager.go:356] "RemoveStaleState removing state" podUID="81f25695-1e0c-4e1c-beda-e1d250bf364b" containerName="kube-rbac-proxy"
May 25 12:33:36.299623 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:33:36.299473    2549 memory_manager.go:356] "RemoveStaleState removing state" podUID="2d880044-04f1-4518-a423-305351e6122d" containerName="kserve-container"
May 25 12:33:36.304595 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:33:36.304572    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-4d75e-predictor-98c496fb-m8ddw"
May 25 12:33:36.307908 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:33:36.307872    2549 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-4d75e-kube-rbac-proxy-sar-config\""
May 25 12:33:36.308003 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:33:36.307962    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-4d75e-predictor-serving-cert\""
May 25 12:33:36.325884 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:33:36.325844    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-4d75e-predictor-98c496fb-m8ddw"]
May 25 12:33:36.396920 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:33:36.396891    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b5232f59-357c-4a8a-a062-13d1161f8a81-proxy-tls\") pod \"success-200-isvc-4d75e-predictor-98c496fb-m8ddw\" (UID: \"b5232f59-357c-4a8a-a062-13d1161f8a81\") " pod="kserve-ci-e2e-test/success-200-isvc-4d75e-predictor-98c496fb-m8ddw"
May 25 12:33:36.397157 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:33:36.397126    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"success-200-isvc-4d75e-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b5232f59-357c-4a8a-a062-13d1161f8a81-success-200-isvc-4d75e-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-4d75e-predictor-98c496fb-m8ddw\" (UID: \"b5232f59-357c-4a8a-a062-13d1161f8a81\") " pod="kserve-ci-e2e-test/success-200-isvc-4d75e-predictor-98c496fb-m8ddw"
May 25 12:33:36.397307 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:33:36.397182    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrdhv\" (UniqueName: \"kubernetes.io/projected/b5232f59-357c-4a8a-a062-13d1161f8a81-kube-api-access-lrdhv\") pod \"success-200-isvc-4d75e-predictor-98c496fb-m8ddw\" (UID: \"b5232f59-357c-4a8a-a062-13d1161f8a81\") " pod="kserve-ci-e2e-test/success-200-isvc-4d75e-predictor-98c496fb-m8ddw"
May 25 12:33:36.416875 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:33:36.416826    2549 generic.go:358] "Generic (PLEG): container finished" podID="51175389-4295-493b-82bc-812c2e34c778" containerID="08e7dd83266cb5136a1b42ee1de5c01799d760726127fed0fd7c845cd9a8d082" exitCode=2
May 25 12:33:36.417037 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:33:36.416895    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-acbca-predictor-687469f549-r5cwv" event={"ID":"51175389-4295-493b-82bc-812c2e34c778","Type":"ContainerDied","Data":"08e7dd83266cb5136a1b42ee1de5c01799d760726127fed0fd7c845cd9a8d082"}
May 25 12:33:36.418776 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:33:36.418735    2549 generic.go:358] "Generic (PLEG): container finished" podID="f2f7c261-0a24-4aa1-b841-a92ba3120164" containerID="e02216eab1f2133c978c0fc004ba611ed778f495507f9d3aeb5b435b6bb7acb0" exitCode=2
May 25 12:33:36.418934 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:33:36.418798    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-acbca-predictor-59cffd4b6b-zwcgm" event={"ID":"f2f7c261-0a24-4aa1-b841-a92ba3120164","Type":"ContainerDied","Data":"e02216eab1f2133c978c0fc004ba611ed778f495507f9d3aeb5b435b6bb7acb0"}
May 25 12:33:36.443611 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:33:36.443573    2549 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-4d75e-predictor-5c8bb955c5-mqnb5"]
May 25 12:33:36.447505 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:33:36.447485    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-4d75e-predictor-5c8bb955c5-mqnb5"
May 25 12:33:36.450415 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:33:36.450387    2549 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-4d75e-kube-rbac-proxy-sar-config\""
May 25 12:33:36.450543 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:33:36.450389    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-4d75e-predictor-serving-cert\""
May 25 12:33:36.457619 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:33:36.457591    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-4d75e-predictor-5c8bb955c5-mqnb5"]
May 25 12:33:36.497706 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:33:36.497664    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b5232f59-357c-4a8a-a062-13d1161f8a81-proxy-tls\") pod \"success-200-isvc-4d75e-predictor-98c496fb-m8ddw\" (UID: \"b5232f59-357c-4a8a-a062-13d1161f8a81\") " pod="kserve-ci-e2e-test/success-200-isvc-4d75e-predictor-98c496fb-m8ddw"
May 25 12:33:36.497887 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:33:36.497712    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d0a4c463-e906-4147-b266-b3d9b2fd6c29-proxy-tls\") pod \"error-404-isvc-4d75e-predictor-5c8bb955c5-mqnb5\" (UID: \"d0a4c463-e906-4147-b266-b3d9b2fd6c29\") " pod="kserve-ci-e2e-test/error-404-isvc-4d75e-predictor-5c8bb955c5-mqnb5"
May 25 12:33:36.497887 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:33:36.497744    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twxq9\" (UniqueName: \"kubernetes.io/projected/d0a4c463-e906-4147-b266-b3d9b2fd6c29-kube-api-access-twxq9\") pod \"error-404-isvc-4d75e-predictor-5c8bb955c5-mqnb5\" (UID: \"d0a4c463-e906-4147-b266-b3d9b2fd6c29\") " pod="kserve-ci-e2e-test/error-404-isvc-4d75e-predictor-5c8bb955c5-mqnb5"
May 25 12:33:36.497887 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:33:36.497818    2549 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-4d75e-predictor-serving-cert: secret "success-200-isvc-4d75e-predictor-serving-cert" not found
May 25 12:33:36.497887 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:33:36.497836    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"error-404-isvc-4d75e-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d0a4c463-e906-4147-b266-b3d9b2fd6c29-error-404-isvc-4d75e-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-4d75e-predictor-5c8bb955c5-mqnb5\" (UID: \"d0a4c463-e906-4147-b266-b3d9b2fd6c29\") " pod="kserve-ci-e2e-test/error-404-isvc-4d75e-predictor-5c8bb955c5-mqnb5"
May 25 12:33:36.498081 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:33:36.497891    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5232f59-357c-4a8a-a062-13d1161f8a81-proxy-tls podName:b5232f59-357c-4a8a-a062-13d1161f8a81 nodeName:}" failed. No retries permitted until 2026-05-25 12:33:36.997874331 +0000 UTC m=+1220.713689597 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/b5232f59-357c-4a8a-a062-13d1161f8a81-proxy-tls") pod "success-200-isvc-4d75e-predictor-98c496fb-m8ddw" (UID: "b5232f59-357c-4a8a-a062-13d1161f8a81") : secret "success-200-isvc-4d75e-predictor-serving-cert" not found
May 25 12:33:36.498081 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:33:36.498002    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"success-200-isvc-4d75e-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b5232f59-357c-4a8a-a062-13d1161f8a81-success-200-isvc-4d75e-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-4d75e-predictor-98c496fb-m8ddw\" (UID: \"b5232f59-357c-4a8a-a062-13d1161f8a81\") " pod="kserve-ci-e2e-test/success-200-isvc-4d75e-predictor-98c496fb-m8ddw"
May 25 12:33:36.498081 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:33:36.498022    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-lrdhv\" (UniqueName: \"kubernetes.io/projected/b5232f59-357c-4a8a-a062-13d1161f8a81-kube-api-access-lrdhv\") pod \"success-200-isvc-4d75e-predictor-98c496fb-m8ddw\" (UID: \"b5232f59-357c-4a8a-a062-13d1161f8a81\") " pod="kserve-ci-e2e-test/success-200-isvc-4d75e-predictor-98c496fb-m8ddw"
May 25 12:33:36.498812 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:33:36.498782    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"success-200-isvc-4d75e-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b5232f59-357c-4a8a-a062-13d1161f8a81-success-200-isvc-4d75e-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-4d75e-predictor-98c496fb-m8ddw\" (UID: \"b5232f59-357c-4a8a-a062-13d1161f8a81\") " pod="kserve-ci-e2e-test/success-200-isvc-4d75e-predictor-98c496fb-m8ddw"
May 25 12:33:36.508578 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:33:36.508551    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrdhv\" (UniqueName: \"kubernetes.io/projected/b5232f59-357c-4a8a-a062-13d1161f8a81-kube-api-access-lrdhv\") pod \"success-200-isvc-4d75e-predictor-98c496fb-m8ddw\" (UID: \"b5232f59-357c-4a8a-a062-13d1161f8a81\") " pod="kserve-ci-e2e-test/success-200-isvc-4d75e-predictor-98c496fb-m8ddw"
May 25 12:33:36.598711 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:33:36.598663    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d0a4c463-e906-4147-b266-b3d9b2fd6c29-proxy-tls\") pod \"error-404-isvc-4d75e-predictor-5c8bb955c5-mqnb5\" (UID: \"d0a4c463-e906-4147-b266-b3d9b2fd6c29\") " pod="kserve-ci-e2e-test/error-404-isvc-4d75e-predictor-5c8bb955c5-mqnb5"
May 25 12:33:36.598908 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:33:36.598719    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-twxq9\" (UniqueName: \"kubernetes.io/projected/d0a4c463-e906-4147-b266-b3d9b2fd6c29-kube-api-access-twxq9\") pod \"error-404-isvc-4d75e-predictor-5c8bb955c5-mqnb5\" (UID: \"d0a4c463-e906-4147-b266-b3d9b2fd6c29\") " pod="kserve-ci-e2e-test/error-404-isvc-4d75e-predictor-5c8bb955c5-mqnb5"
May 25 12:33:36.598908 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:33:36.598756    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"error-404-isvc-4d75e-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d0a4c463-e906-4147-b266-b3d9b2fd6c29-error-404-isvc-4d75e-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-4d75e-predictor-5c8bb955c5-mqnb5\" (UID: \"d0a4c463-e906-4147-b266-b3d9b2fd6c29\") " pod="kserve-ci-e2e-test/error-404-isvc-4d75e-predictor-5c8bb955c5-mqnb5"
May 25 12:33:36.599585 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:33:36.599554    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"error-404-isvc-4d75e-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d0a4c463-e906-4147-b266-b3d9b2fd6c29-error-404-isvc-4d75e-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-4d75e-predictor-5c8bb955c5-mqnb5\" (UID: \"d0a4c463-e906-4147-b266-b3d9b2fd6c29\") " pod="kserve-ci-e2e-test/error-404-isvc-4d75e-predictor-5c8bb955c5-mqnb5"
May 25 12:33:36.601171 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:33:36.601148    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d0a4c463-e906-4147-b266-b3d9b2fd6c29-proxy-tls\") pod \"error-404-isvc-4d75e-predictor-5c8bb955c5-mqnb5\" (UID: \"d0a4c463-e906-4147-b266-b3d9b2fd6c29\") " pod="kserve-ci-e2e-test/error-404-isvc-4d75e-predictor-5c8bb955c5-mqnb5"
May 25 12:33:36.619208 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:33:36.619170    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-twxq9\" (UniqueName: \"kubernetes.io/projected/d0a4c463-e906-4147-b266-b3d9b2fd6c29-kube-api-access-twxq9\") pod \"error-404-isvc-4d75e-predictor-5c8bb955c5-mqnb5\" (UID: \"d0a4c463-e906-4147-b266-b3d9b2fd6c29\") " pod="kserve-ci-e2e-test/error-404-isvc-4d75e-predictor-5c8bb955c5-mqnb5"
May 25 12:33:36.760265 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:33:36.760139    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-4d75e-predictor-5c8bb955c5-mqnb5"
May 25 12:33:36.907255 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:33:36.907222    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-4d75e-predictor-5c8bb955c5-mqnb5"]
May 25 12:33:36.909824 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:33:36.909790    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0a4c463_e906_4147_b266_b3d9b2fd6c29.slice/crio-d377bd8704559f64a7eeb934ac9b84c97c007b6ef6d07b3c409deb7d094a5f70 WatchSource:0}: Error finding container d377bd8704559f64a7eeb934ac9b84c97c007b6ef6d07b3c409deb7d094a5f70: Status 404 returned error can't find the container with id d377bd8704559f64a7eeb934ac9b84c97c007b6ef6d07b3c409deb7d094a5f70
May 25 12:33:37.004183 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:33:37.004156    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b5232f59-357c-4a8a-a062-13d1161f8a81-proxy-tls\") pod \"success-200-isvc-4d75e-predictor-98c496fb-m8ddw\" (UID: \"b5232f59-357c-4a8a-a062-13d1161f8a81\") " pod="kserve-ci-e2e-test/success-200-isvc-4d75e-predictor-98c496fb-m8ddw"
May 25 12:33:37.006476 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:33:37.006457    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b5232f59-357c-4a8a-a062-13d1161f8a81-proxy-tls\") pod \"success-200-isvc-4d75e-predictor-98c496fb-m8ddw\" (UID: \"b5232f59-357c-4a8a-a062-13d1161f8a81\") " pod="kserve-ci-e2e-test/success-200-isvc-4d75e-predictor-98c496fb-m8ddw"
May 25 12:33:37.217814 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:33:37.217772    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-4d75e-predictor-98c496fb-m8ddw"
May 25 12:33:37.362901 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:33:37.362859    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-4d75e-predictor-98c496fb-m8ddw"]
May 25 12:33:37.366138 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:33:37.366112    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5232f59_357c_4a8a_a062_13d1161f8a81.slice/crio-debff1ab0c64f9a45e7c889864b30238945d4e7faf79b9c2b9b90efdecc47677 WatchSource:0}: Error finding container debff1ab0c64f9a45e7c889864b30238945d4e7faf79b9c2b9b90efdecc47677: Status 404 returned error can't find the container with id debff1ab0c64f9a45e7c889864b30238945d4e7faf79b9c2b9b90efdecc47677
May 25 12:33:37.424127 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:33:37.424096    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-4d75e-predictor-98c496fb-m8ddw" event={"ID":"b5232f59-357c-4a8a-a062-13d1161f8a81","Type":"ContainerStarted","Data":"debff1ab0c64f9a45e7c889864b30238945d4e7faf79b9c2b9b90efdecc47677"}
May 25 12:33:37.425735 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:33:37.425709    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-4d75e-predictor-5c8bb955c5-mqnb5" event={"ID":"d0a4c463-e906-4147-b266-b3d9b2fd6c29","Type":"ContainerStarted","Data":"34dca45b8b17a4dbf5731d8a197ac7f86e9338dcc48dac93149c4a73cdc4f4c3"}
May 25 12:33:37.425861 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:33:37.425741    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-4d75e-predictor-5c8bb955c5-mqnb5" event={"ID":"d0a4c463-e906-4147-b266-b3d9b2fd6c29","Type":"ContainerStarted","Data":"f28b53db05c8eb0d7ccd2e1105707864e565f141a219a60d1d01e58e48298b4d"}
May 25 12:33:37.425861 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:33:37.425750    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-4d75e-predictor-5c8bb955c5-mqnb5" event={"ID":"d0a4c463-e906-4147-b266-b3d9b2fd6c29","Type":"ContainerStarted","Data":"d377bd8704559f64a7eeb934ac9b84c97c007b6ef6d07b3c409deb7d094a5f70"}
May 25 12:33:37.425861 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:33:37.425792    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-4d75e-predictor-5c8bb955c5-mqnb5"
May 25 12:33:38.434342 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:33:38.434310    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-4d75e-predictor-98c496fb-m8ddw" event={"ID":"b5232f59-357c-4a8a-a062-13d1161f8a81","Type":"ContainerStarted","Data":"0e589999a9931208d6901d9f586a935980ebc01727722bebd4a22a3a3f2cb81e"}
May 25 12:33:38.434812 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:33:38.434349    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-4d75e-predictor-98c496fb-m8ddw" event={"ID":"b5232f59-357c-4a8a-a062-13d1161f8a81","Type":"ContainerStarted","Data":"298328194cbed0020fcfc4bd1bc4dfcace27fc89e79fb5abe29c99a22b4f01c8"}
May 25 12:33:38.434812 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:33:38.434689    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-4d75e-predictor-98c496fb-m8ddw"
May 25 12:33:38.434812 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:33:38.434721    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-4d75e-predictor-98c496fb-m8ddw"
May 25 12:33:38.434812 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:33:38.434734    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-4d75e-predictor-5c8bb955c5-mqnb5"
May 25 12:33:38.436140 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:33:38.436114    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-4d75e-predictor-98c496fb-m8ddw" podUID="b5232f59-357c-4a8a-a062-13d1161f8a81" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
May 25 12:33:38.436253 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:33:38.436148    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-4d75e-predictor-5c8bb955c5-mqnb5" podUID="d0a4c463-e906-4147-b266-b3d9b2fd6c29" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
May 25 12:33:38.457992 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:33:38.457952    2549 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-4d75e-predictor-5c8bb955c5-mqnb5" podStartSLOduration=2.45794034 podStartE2EDuration="2.45794034s" podCreationTimestamp="2026-05-25 12:33:36 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 12:33:37.448087202 +0000 UTC m=+1221.163902488" watchObservedRunningTime="2026-05-25 12:33:38.45794034 +0000 UTC m=+1222.173755672"
May 25 12:33:38.458424 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:33:38.458401    2549 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-4d75e-predictor-98c496fb-m8ddw" podStartSLOduration=2.458394385 podStartE2EDuration="2.458394385s" podCreationTimestamp="2026-05-25 12:33:36 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 12:33:38.455680988 +0000 UTC m=+1222.171496286" watchObservedRunningTime="2026-05-25 12:33:38.458394385 +0000 UTC m=+1222.174209671"
May 25 12:33:39.083040 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:33:39.082996    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-acbca-predictor-59cffd4b6b-zwcgm" podUID="f2f7c261-0a24-4aa1-b841-a92ba3120164" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.42:8643/healthz\": dial tcp 10.134.0.42:8643: connect: connection refused"
May 25 12:33:39.083286 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:33:39.082994    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-acbca-predictor-687469f549-r5cwv" podUID="51175389-4295-493b-82bc-812c2e34c778" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.43:8643/healthz\": dial tcp 10.134.0.43:8643: connect: connection refused"
May 25 12:33:39.438170 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:33:39.438120    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-4d75e-predictor-5c8bb955c5-mqnb5" podUID="d0a4c463-e906-4147-b266-b3d9b2fd6c29" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
May 25 12:33:39.438583 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:33:39.438137    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-4d75e-predictor-98c496fb-m8ddw" podUID="b5232f59-357c-4a8a-a062-13d1161f8a81" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
May 25 12:33:40.262514 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:33:40.262371    2549 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-acbca-predictor-687469f549-r5cwv"
May 25 12:33:40.338521 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:33:40.338492    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nsdmc\" (UniqueName: \"kubernetes.io/projected/51175389-4295-493b-82bc-812c2e34c778-kube-api-access-nsdmc\") pod \"51175389-4295-493b-82bc-812c2e34c778\" (UID: \"51175389-4295-493b-82bc-812c2e34c778\") "
May 25 12:33:40.338656 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:33:40.338556    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/51175389-4295-493b-82bc-812c2e34c778-proxy-tls\") pod \"51175389-4295-493b-82bc-812c2e34c778\" (UID: \"51175389-4295-493b-82bc-812c2e34c778\") "
May 25 12:33:40.338656 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:33:40.338620    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"error-404-isvc-acbca-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/51175389-4295-493b-82bc-812c2e34c778-error-404-isvc-acbca-kube-rbac-proxy-sar-config\") pod \"51175389-4295-493b-82bc-812c2e34c778\" (UID: \"51175389-4295-493b-82bc-812c2e34c778\") "
May 25 12:33:40.339019 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:33:40.338986    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51175389-4295-493b-82bc-812c2e34c778-error-404-isvc-acbca-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "error-404-isvc-acbca-kube-rbac-proxy-sar-config") pod "51175389-4295-493b-82bc-812c2e34c778" (UID: "51175389-4295-493b-82bc-812c2e34c778"). InnerVolumeSpecName "error-404-isvc-acbca-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 12:33:40.340715 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:33:40.340688    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51175389-4295-493b-82bc-812c2e34c778-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "51175389-4295-493b-82bc-812c2e34c778" (UID: "51175389-4295-493b-82bc-812c2e34c778"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 12:33:40.340795 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:33:40.340778    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51175389-4295-493b-82bc-812c2e34c778-kube-api-access-nsdmc" (OuterVolumeSpecName: "kube-api-access-nsdmc") pod "51175389-4295-493b-82bc-812c2e34c778" (UID: "51175389-4295-493b-82bc-812c2e34c778"). InnerVolumeSpecName "kube-api-access-nsdmc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 12:33:40.347546 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:33:40.347484    2549 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-acbca-predictor-59cffd4b6b-zwcgm"
May 25 12:33:40.439239 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:33:40.439183    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"success-200-isvc-acbca-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f2f7c261-0a24-4aa1-b841-a92ba3120164-success-200-isvc-acbca-kube-rbac-proxy-sar-config\") pod \"f2f7c261-0a24-4aa1-b841-a92ba3120164\" (UID: \"f2f7c261-0a24-4aa1-b841-a92ba3120164\") "
May 25 12:33:40.439239 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:33:40.439242    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f2f7c261-0a24-4aa1-b841-a92ba3120164-proxy-tls\") pod \"f2f7c261-0a24-4aa1-b841-a92ba3120164\" (UID: \"f2f7c261-0a24-4aa1-b841-a92ba3120164\") "
May 25 12:33:40.439760 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:33:40.439288    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-chlw6\" (UniqueName: \"kubernetes.io/projected/f2f7c261-0a24-4aa1-b841-a92ba3120164-kube-api-access-chlw6\") pod \"f2f7c261-0a24-4aa1-b841-a92ba3120164\" (UID: \"f2f7c261-0a24-4aa1-b841-a92ba3120164\") "
May 25 12:33:40.439760 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:33:40.439539    2549 reconciler_common.go:299] "Volume detached for volume \"error-404-isvc-acbca-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/51175389-4295-493b-82bc-812c2e34c778-error-404-isvc-acbca-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
May 25 12:33:40.439760 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:33:40.439563    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nsdmc\" (UniqueName: \"kubernetes.io/projected/51175389-4295-493b-82bc-812c2e34c778-kube-api-access-nsdmc\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
May 25 12:33:40.439760 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:33:40.439577    2549 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/51175389-4295-493b-82bc-812c2e34c778-proxy-tls\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
May 25 12:33:40.439760 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:33:40.439652    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2f7c261-0a24-4aa1-b841-a92ba3120164-success-200-isvc-acbca-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "success-200-isvc-acbca-kube-rbac-proxy-sar-config") pod "f2f7c261-0a24-4aa1-b841-a92ba3120164" (UID: "f2f7c261-0a24-4aa1-b841-a92ba3120164"). InnerVolumeSpecName "success-200-isvc-acbca-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 12:33:40.441661 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:33:40.441627    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2f7c261-0a24-4aa1-b841-a92ba3120164-kube-api-access-chlw6" (OuterVolumeSpecName: "kube-api-access-chlw6") pod "f2f7c261-0a24-4aa1-b841-a92ba3120164" (UID: "f2f7c261-0a24-4aa1-b841-a92ba3120164"). InnerVolumeSpecName "kube-api-access-chlw6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 12:33:40.441661 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:33:40.441652    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2f7c261-0a24-4aa1-b841-a92ba3120164-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f2f7c261-0a24-4aa1-b841-a92ba3120164" (UID: "f2f7c261-0a24-4aa1-b841-a92ba3120164"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 12:33:40.443575 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:33:40.443546    2549 generic.go:358] "Generic (PLEG): container finished" podID="51175389-4295-493b-82bc-812c2e34c778" containerID="1b1e92734dd67ad756c33fbf5939df977d7ac842685b56521b694534adec3c39" exitCode=0
May 25 12:33:40.443681 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:33:40.443629    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-acbca-predictor-687469f549-r5cwv" event={"ID":"51175389-4295-493b-82bc-812c2e34c778","Type":"ContainerDied","Data":"1b1e92734dd67ad756c33fbf5939df977d7ac842685b56521b694534adec3c39"}
May 25 12:33:40.443681 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:33:40.443675    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-acbca-predictor-687469f549-r5cwv" event={"ID":"51175389-4295-493b-82bc-812c2e34c778","Type":"ContainerDied","Data":"c2f964ace85188a3f90591687fc247044f187e5d16d1c379f14e6fad37f88bca"}
May 25 12:33:40.443959 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:33:40.443646    2549 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-acbca-predictor-687469f549-r5cwv"
May 25 12:33:40.443959 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:33:40.443696    2549 scope.go:117] "RemoveContainer" containerID="08e7dd83266cb5136a1b42ee1de5c01799d760726127fed0fd7c845cd9a8d082"
May 25 12:33:40.445428 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:33:40.445403    2549 generic.go:358] "Generic (PLEG): container finished" podID="f2f7c261-0a24-4aa1-b841-a92ba3120164" containerID="12872b3e036cc6bd133452554c75f0b39f480650f7eb3f21a3ef7f4fbea6ca53" exitCode=0
May 25 12:33:40.445514 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:33:40.445489    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-acbca-predictor-59cffd4b6b-zwcgm" event={"ID":"f2f7c261-0a24-4aa1-b841-a92ba3120164","Type":"ContainerDied","Data":"12872b3e036cc6bd133452554c75f0b39f480650f7eb3f21a3ef7f4fbea6ca53"}
May 25 12:33:40.445563 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:33:40.445543    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-acbca-predictor-59cffd4b6b-zwcgm" event={"ID":"f2f7c261-0a24-4aa1-b841-a92ba3120164","Type":"ContainerDied","Data":"abb0e3c26eceb7b4838571f823baaba9c86606c96e112f41f557d9003500e27c"}
May 25 12:33:40.445563 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:33:40.445496    2549 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-acbca-predictor-59cffd4b6b-zwcgm"
May 25 12:33:40.445944 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:33:40.445917    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-4d75e-predictor-98c496fb-m8ddw" podUID="b5232f59-357c-4a8a-a062-13d1161f8a81" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
May 25 12:33:40.453471 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:33:40.453450    2549 scope.go:117] "RemoveContainer" containerID="1b1e92734dd67ad756c33fbf5939df977d7ac842685b56521b694534adec3c39"
May 25 12:33:40.462909 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:33:40.462887    2549 scope.go:117] "RemoveContainer" containerID="08e7dd83266cb5136a1b42ee1de5c01799d760726127fed0fd7c845cd9a8d082"
May 25 12:33:40.463240 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:33:40.463215    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08e7dd83266cb5136a1b42ee1de5c01799d760726127fed0fd7c845cd9a8d082\": container with ID starting with 08e7dd83266cb5136a1b42ee1de5c01799d760726127fed0fd7c845cd9a8d082 not found: ID does not exist" containerID="08e7dd83266cb5136a1b42ee1de5c01799d760726127fed0fd7c845cd9a8d082"
May 25 12:33:40.463306 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:33:40.463252    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08e7dd83266cb5136a1b42ee1de5c01799d760726127fed0fd7c845cd9a8d082"} err="failed to get container status \"08e7dd83266cb5136a1b42ee1de5c01799d760726127fed0fd7c845cd9a8d082\": rpc error: code = NotFound desc = could not find container \"08e7dd83266cb5136a1b42ee1de5c01799d760726127fed0fd7c845cd9a8d082\": container with ID starting with 08e7dd83266cb5136a1b42ee1de5c01799d760726127fed0fd7c845cd9a8d082 not found: ID does not exist"
May 25 12:33:40.463306 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:33:40.463273    2549 scope.go:117] "RemoveContainer" containerID="1b1e92734dd67ad756c33fbf5939df977d7ac842685b56521b694534adec3c39"
May 25 12:33:40.463526 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:33:40.463507    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b1e92734dd67ad756c33fbf5939df977d7ac842685b56521b694534adec3c39\": container with ID starting with 1b1e92734dd67ad756c33fbf5939df977d7ac842685b56521b694534adec3c39 not found: ID does not exist" containerID="1b1e92734dd67ad756c33fbf5939df977d7ac842685b56521b694534adec3c39"
May 25 12:33:40.463565 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:33:40.463535    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b1e92734dd67ad756c33fbf5939df977d7ac842685b56521b694534adec3c39"} err="failed to get container status \"1b1e92734dd67ad756c33fbf5939df977d7ac842685b56521b694534adec3c39\": rpc error: code = NotFound desc = could not find container \"1b1e92734dd67ad756c33fbf5939df977d7ac842685b56521b694534adec3c39\": container with ID starting with 1b1e92734dd67ad756c33fbf5939df977d7ac842685b56521b694534adec3c39 not found: ID does not exist"
May 25 12:33:40.463565 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:33:40.463556    2549 scope.go:117] "RemoveContainer" containerID="e02216eab1f2133c978c0fc004ba611ed778f495507f9d3aeb5b435b6bb7acb0"
May 25 12:33:40.471274 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:33:40.471252    2549 scope.go:117] "RemoveContainer" containerID="12872b3e036cc6bd133452554c75f0b39f480650f7eb3f21a3ef7f4fbea6ca53"
May 25 12:33:40.475081 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:33:40.475057    2549 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-acbca-predictor-687469f549-r5cwv"]
May 25 12:33:40.480254 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:33:40.480230    2549 scope.go:117] "RemoveContainer" containerID="e02216eab1f2133c978c0fc004ba611ed778f495507f9d3aeb5b435b6bb7acb0"
May 25 12:33:40.480427 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:33:40.480406    2549 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-acbca-predictor-687469f549-r5cwv"]
May 25 12:33:40.480539 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:33:40.480523    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e02216eab1f2133c978c0fc004ba611ed778f495507f9d3aeb5b435b6bb7acb0\": container with ID starting with e02216eab1f2133c978c0fc004ba611ed778f495507f9d3aeb5b435b6bb7acb0 not found: ID does not exist" containerID="e02216eab1f2133c978c0fc004ba611ed778f495507f9d3aeb5b435b6bb7acb0"
May 25 12:33:40.480589 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:33:40.480547    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e02216eab1f2133c978c0fc004ba611ed778f495507f9d3aeb5b435b6bb7acb0"} err="failed to get container status \"e02216eab1f2133c978c0fc004ba611ed778f495507f9d3aeb5b435b6bb7acb0\": rpc error: code = NotFound desc = could not find container \"e02216eab1f2133c978c0fc004ba611ed778f495507f9d3aeb5b435b6bb7acb0\": container with ID starting with e02216eab1f2133c978c0fc004ba611ed778f495507f9d3aeb5b435b6bb7acb0 not found: ID does not exist"
May 25 12:33:40.480589 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:33:40.480566    2549 scope.go:117] "RemoveContainer" containerID="12872b3e036cc6bd133452554c75f0b39f480650f7eb3f21a3ef7f4fbea6ca53"
May 25 12:33:40.480817 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:33:40.480802    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12872b3e036cc6bd133452554c75f0b39f480650f7eb3f21a3ef7f4fbea6ca53\": container with ID starting with 12872b3e036cc6bd133452554c75f0b39f480650f7eb3f21a3ef7f4fbea6ca53 not found: ID does not exist" containerID="12872b3e036cc6bd133452554c75f0b39f480650f7eb3f21a3ef7f4fbea6ca53"
May 25 12:33:40.480861 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:33:40.480820    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12872b3e036cc6bd133452554c75f0b39f480650f7eb3f21a3ef7f4fbea6ca53"} err="failed to get container status \"12872b3e036cc6bd133452554c75f0b39f480650f7eb3f21a3ef7f4fbea6ca53\": rpc error: code = NotFound desc = could not find container \"12872b3e036cc6bd133452554c75f0b39f480650f7eb3f21a3ef7f4fbea6ca53\": container with ID starting with 12872b3e036cc6bd133452554c75f0b39f480650f7eb3f21a3ef7f4fbea6ca53 not found: ID does not exist"
May 25 12:33:40.494713 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:33:40.494677    2549 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-acbca-predictor-59cffd4b6b-zwcgm"]
May 25 12:33:40.504439 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:33:40.504405    2549 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-acbca-predictor-59cffd4b6b-zwcgm"]
May 25 12:33:40.540272 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:33:40.540243    2549 reconciler_common.go:299] "Volume detached for volume \"success-200-isvc-acbca-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f2f7c261-0a24-4aa1-b841-a92ba3120164-success-200-isvc-acbca-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
May 25 12:33:40.540272 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:33:40.540267    2549 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f2f7c261-0a24-4aa1-b841-a92ba3120164-proxy-tls\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
May 25 12:33:40.540272 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:33:40.540280    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-chlw6\" (UniqueName: \"kubernetes.io/projected/f2f7c261-0a24-4aa1-b841-a92ba3120164-kube-api-access-chlw6\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
May 25 12:33:40.847994 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:33:40.847961    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="51175389-4295-493b-82bc-812c2e34c778" path="/var/lib/kubelet/pods/51175389-4295-493b-82bc-812c2e34c778/volumes"
May 25 12:33:40.848425 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:33:40.848410    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f2f7c261-0a24-4aa1-b841-a92ba3120164" path="/var/lib/kubelet/pods/f2f7c261-0a24-4aa1-b841-a92ba3120164/volumes"
May 25 12:33:44.257413 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:33:44.257369    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-155ee-predictor-6b9f6fd99b-qvjql" podUID="efcb96a7-a292-4595-880f-e7670a2d9928" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
May 25 12:33:44.259678 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:33:44.259642    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-155ee-predictor-57c99b6f99-mtr6n" podUID="3f43e011-dd0d-4717-8d72-3e2882bb8650" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
May 25 12:33:44.442266 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:33:44.442236    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-4d75e-predictor-5c8bb955c5-mqnb5"
May 25 12:33:44.442846 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:33:44.442809    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-4d75e-predictor-5c8bb955c5-mqnb5" podUID="d0a4c463-e906-4147-b266-b3d9b2fd6c29" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
May 25 12:33:45.450123 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:33:45.450091    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-4d75e-predictor-98c496fb-m8ddw"
May 25 12:33:45.450703 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:33:45.450669    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-4d75e-predictor-98c496fb-m8ddw" podUID="b5232f59-357c-4a8a-a062-13d1161f8a81" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
May 25 12:33:54.258366 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:33:54.258332    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-155ee-predictor-6b9f6fd99b-qvjql"
May 25 12:33:54.260354 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:33:54.260332    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-155ee-predictor-57c99b6f99-mtr6n"
May 25 12:33:54.442783 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:33:54.442742    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-4d75e-predictor-5c8bb955c5-mqnb5" podUID="d0a4c463-e906-4147-b266-b3d9b2fd6c29" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
May 25 12:33:55.451018 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:33:55.450975    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-4d75e-predictor-98c496fb-m8ddw" podUID="b5232f59-357c-4a8a-a062-13d1161f8a81" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
May 25 12:34:04.443020 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:34:04.442920    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-4d75e-predictor-5c8bb955c5-mqnb5" podUID="d0a4c463-e906-4147-b266-b3d9b2fd6c29" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
May 25 12:34:05.450672 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:34:05.450633    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-4d75e-predictor-98c496fb-m8ddw" podUID="b5232f59-357c-4a8a-a062-13d1161f8a81" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
May 25 12:34:14.442827 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:34:14.442768    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-4d75e-predictor-5c8bb955c5-mqnb5" podUID="d0a4c463-e906-4147-b266-b3d9b2fd6c29" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
May 25 12:34:15.451150 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:34:15.451105    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-4d75e-predictor-98c496fb-m8ddw" podUID="b5232f59-357c-4a8a-a062-13d1161f8a81" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
May 25 12:34:24.443411 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:34:24.443371    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-4d75e-predictor-5c8bb955c5-mqnb5"
May 25 12:34:25.451380 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:34:25.451345    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-4d75e-predictor-98c496fb-m8ddw"
May 25 12:42:21.188758 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:21.188719    2549 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-155ee-predictor-6b9f6fd99b-qvjql"]
May 25 12:42:21.189277 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:21.189097    2549 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-155ee-predictor-6b9f6fd99b-qvjql" podUID="efcb96a7-a292-4595-880f-e7670a2d9928" containerName="kserve-container" containerID="cri-o://705521478faa138c09b52e9f4f8bc7ae915575fc304e739b0110b0137fedee21" gracePeriod=30
May 25 12:42:21.189448 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:21.189398    2549 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-155ee-predictor-6b9f6fd99b-qvjql" podUID="efcb96a7-a292-4595-880f-e7670a2d9928" containerName="kube-rbac-proxy" containerID="cri-o://35cba0cdaa0151f32daa59468dfafa895877d26e02ea9dae0e2bebe8e6b2c36a" gracePeriod=30
May 25 12:42:21.261605 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:21.261571    2549 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-155ee-predictor-57c99b6f99-mtr6n"]
May 25 12:42:21.261905 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:21.261853    2549 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-155ee-predictor-57c99b6f99-mtr6n" podUID="3f43e011-dd0d-4717-8d72-3e2882bb8650" containerName="kserve-container" containerID="cri-o://a6bc0a84961219878a56ca53f42d73fb8f31d3a695ce162918376d483eb1a9d1" gracePeriod=30
May 25 12:42:21.262009 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:21.261890    2549 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-155ee-predictor-57c99b6f99-mtr6n" podUID="3f43e011-dd0d-4717-8d72-3e2882bb8650" containerName="kube-rbac-proxy" containerID="cri-o://b234d4a0306944df8548d229741e8cbfa6101f667f0b33781f772e33652514fd" gracePeriod=30
May 25 12:42:21.265788 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:21.265758    2549 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-fd4b4-predictor-59d7bccbf8-pl6qv"]
May 25 12:42:21.266291 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:21.266273    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51175389-4295-493b-82bc-812c2e34c778" containerName="kserve-container"
May 25 12:42:21.266291 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:21.266293    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="51175389-4295-493b-82bc-812c2e34c778" containerName="kserve-container"
May 25 12:42:21.266441 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:21.266309    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2f7c261-0a24-4aa1-b841-a92ba3120164" containerName="kube-rbac-proxy"
May 25 12:42:21.266441 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:21.266315    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2f7c261-0a24-4aa1-b841-a92ba3120164" containerName="kube-rbac-proxy"
May 25 12:42:21.266441 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:21.266322    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51175389-4295-493b-82bc-812c2e34c778" containerName="kube-rbac-proxy"
May 25 12:42:21.266441 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:21.266328    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="51175389-4295-493b-82bc-812c2e34c778" containerName="kube-rbac-proxy"
May 25 12:42:21.266441 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:21.266340    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2f7c261-0a24-4aa1-b841-a92ba3120164" containerName="kserve-container"
May 25 12:42:21.266441 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:21.266346    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2f7c261-0a24-4aa1-b841-a92ba3120164" containerName="kserve-container"
May 25 12:42:21.266441 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:21.266414    2549 memory_manager.go:356] "RemoveStaleState removing state" podUID="51175389-4295-493b-82bc-812c2e34c778" containerName="kserve-container"
May 25 12:42:21.266441 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:21.266424    2549 memory_manager.go:356] "RemoveStaleState removing state" podUID="f2f7c261-0a24-4aa1-b841-a92ba3120164" containerName="kube-rbac-proxy"
May 25 12:42:21.266441 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:21.266433    2549 memory_manager.go:356] "RemoveStaleState removing state" podUID="f2f7c261-0a24-4aa1-b841-a92ba3120164" containerName="kserve-container"
May 25 12:42:21.266441 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:21.266442    2549 memory_manager.go:356] "RemoveStaleState removing state" podUID="51175389-4295-493b-82bc-812c2e34c778" containerName="kube-rbac-proxy"
May 25 12:42:21.269767 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:21.269744    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-fd4b4-predictor-59d7bccbf8-pl6qv"
May 25 12:42:21.272146 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:21.272124    2549 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-fd4b4-kube-rbac-proxy-sar-config\""
May 25 12:42:21.272249 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:21.272156    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-fd4b4-predictor-serving-cert\""
May 25 12:42:21.285412 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:21.285363    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-fd4b4-predictor-59d7bccbf8-pl6qv"]
May 25 12:42:21.340367 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:21.340326    2549 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-fd4b4-predictor-7694f9fcbf-n88zh"]
May 25 12:42:21.344375 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:21.344345    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-fd4b4-predictor-7694f9fcbf-n88zh"
May 25 12:42:21.347179 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:21.347149    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-fd4b4-predictor-serving-cert\""
May 25 12:42:21.347179 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:21.347170    2549 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-fd4b4-kube-rbac-proxy-sar-config\""
May 25 12:42:21.351058 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:21.351034    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-fd4b4-predictor-7694f9fcbf-n88zh"]
May 25 12:42:21.369734 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:21.369686    2549 generic.go:358] "Generic (PLEG): container finished" podID="efcb96a7-a292-4595-880f-e7670a2d9928" containerID="35cba0cdaa0151f32daa59468dfafa895877d26e02ea9dae0e2bebe8e6b2c36a" exitCode=2
May 25 12:42:21.369734 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:21.369726    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-155ee-predictor-6b9f6fd99b-qvjql" event={"ID":"efcb96a7-a292-4595-880f-e7670a2d9928","Type":"ContainerDied","Data":"35cba0cdaa0151f32daa59468dfafa895877d26e02ea9dae0e2bebe8e6b2c36a"}
May 25 12:42:21.432401 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:21.432343    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnlz8\" (UniqueName: \"kubernetes.io/projected/90636a58-53a5-4c10-864f-cdf6647bd677-kube-api-access-xnlz8\") pod \"success-200-isvc-fd4b4-predictor-59d7bccbf8-pl6qv\" (UID: \"90636a58-53a5-4c10-864f-cdf6647bd677\") " pod="kserve-ci-e2e-test/success-200-isvc-fd4b4-predictor-59d7bccbf8-pl6qv"
May 25 12:42:21.432596 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:21.432512    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/90636a58-53a5-4c10-864f-cdf6647bd677-proxy-tls\") pod \"success-200-isvc-fd4b4-predictor-59d7bccbf8-pl6qv\" (UID: \"90636a58-53a5-4c10-864f-cdf6647bd677\") " pod="kserve-ci-e2e-test/success-200-isvc-fd4b4-predictor-59d7bccbf8-pl6qv"
May 25 12:42:21.432596 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:21.432553    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"success-200-isvc-fd4b4-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/90636a58-53a5-4c10-864f-cdf6647bd677-success-200-isvc-fd4b4-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-fd4b4-predictor-59d7bccbf8-pl6qv\" (UID: \"90636a58-53a5-4c10-864f-cdf6647bd677\") " pod="kserve-ci-e2e-test/success-200-isvc-fd4b4-predictor-59d7bccbf8-pl6qv"
May 25 12:42:21.533593 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:21.533481    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/90636a58-53a5-4c10-864f-cdf6647bd677-proxy-tls\") pod \"success-200-isvc-fd4b4-predictor-59d7bccbf8-pl6qv\" (UID: \"90636a58-53a5-4c10-864f-cdf6647bd677\") " pod="kserve-ci-e2e-test/success-200-isvc-fd4b4-predictor-59d7bccbf8-pl6qv"
May 25 12:42:21.533593 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:21.533539    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"error-404-isvc-fd4b4-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4ab201cd-f34e-4d4c-aa98-179dee7a9ede-error-404-isvc-fd4b4-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-fd4b4-predictor-7694f9fcbf-n88zh\" (UID: \"4ab201cd-f34e-4d4c-aa98-179dee7a9ede\") " pod="kserve-ci-e2e-test/error-404-isvc-fd4b4-predictor-7694f9fcbf-n88zh"
May 25 12:42:21.533593 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:21.533561    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwtxw\" (UniqueName: \"kubernetes.io/projected/4ab201cd-f34e-4d4c-aa98-179dee7a9ede-kube-api-access-jwtxw\") pod \"error-404-isvc-fd4b4-predictor-7694f9fcbf-n88zh\" (UID: \"4ab201cd-f34e-4d4c-aa98-179dee7a9ede\") " pod="kserve-ci-e2e-test/error-404-isvc-fd4b4-predictor-7694f9fcbf-n88zh"
May 25 12:42:21.533878 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:21.533618    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"success-200-isvc-fd4b4-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/90636a58-53a5-4c10-864f-cdf6647bd677-success-200-isvc-fd4b4-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-fd4b4-predictor-59d7bccbf8-pl6qv\" (UID: \"90636a58-53a5-4c10-864f-cdf6647bd677\") " pod="kserve-ci-e2e-test/success-200-isvc-fd4b4-predictor-59d7bccbf8-pl6qv"
May 25 12:42:21.533878 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:42:21.533650    2549 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-fd4b4-predictor-serving-cert: secret "success-200-isvc-fd4b4-predictor-serving-cert" not found
May 25 12:42:21.533878 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:21.533710    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-xnlz8\" (UniqueName: \"kubernetes.io/projected/90636a58-53a5-4c10-864f-cdf6647bd677-kube-api-access-xnlz8\") pod \"success-200-isvc-fd4b4-predictor-59d7bccbf8-pl6qv\" (UID: \"90636a58-53a5-4c10-864f-cdf6647bd677\") " pod="kserve-ci-e2e-test/success-200-isvc-fd4b4-predictor-59d7bccbf8-pl6qv"
May 25 12:42:21.533878 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:42:21.533730    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90636a58-53a5-4c10-864f-cdf6647bd677-proxy-tls podName:90636a58-53a5-4c10-864f-cdf6647bd677 nodeName:}" failed. No retries permitted until 2026-05-25 12:42:22.03370859 +0000 UTC m=+1745.749523871 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/90636a58-53a5-4c10-864f-cdf6647bd677-proxy-tls") pod "success-200-isvc-fd4b4-predictor-59d7bccbf8-pl6qv" (UID: "90636a58-53a5-4c10-864f-cdf6647bd677") : secret "success-200-isvc-fd4b4-predictor-serving-cert" not found
May 25 12:42:21.533878 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:21.533777    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4ab201cd-f34e-4d4c-aa98-179dee7a9ede-proxy-tls\") pod \"error-404-isvc-fd4b4-predictor-7694f9fcbf-n88zh\" (UID: \"4ab201cd-f34e-4d4c-aa98-179dee7a9ede\") " pod="kserve-ci-e2e-test/error-404-isvc-fd4b4-predictor-7694f9fcbf-n88zh"
May 25 12:42:21.534252 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:21.534233    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"success-200-isvc-fd4b4-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/90636a58-53a5-4c10-864f-cdf6647bd677-success-200-isvc-fd4b4-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-fd4b4-predictor-59d7bccbf8-pl6qv\" (UID: \"90636a58-53a5-4c10-864f-cdf6647bd677\") " pod="kserve-ci-e2e-test/success-200-isvc-fd4b4-predictor-59d7bccbf8-pl6qv"
May 25 12:42:21.549257 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:21.549224    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnlz8\" (UniqueName: \"kubernetes.io/projected/90636a58-53a5-4c10-864f-cdf6647bd677-kube-api-access-xnlz8\") pod \"success-200-isvc-fd4b4-predictor-59d7bccbf8-pl6qv\" (UID: \"90636a58-53a5-4c10-864f-cdf6647bd677\") " pod="kserve-ci-e2e-test/success-200-isvc-fd4b4-predictor-59d7bccbf8-pl6qv"
May 25 12:42:21.634712 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:21.634669    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4ab201cd-f34e-4d4c-aa98-179dee7a9ede-proxy-tls\") pod \"error-404-isvc-fd4b4-predictor-7694f9fcbf-n88zh\" (UID: \"4ab201cd-f34e-4d4c-aa98-179dee7a9ede\") " pod="kserve-ci-e2e-test/error-404-isvc-fd4b4-predictor-7694f9fcbf-n88zh"
May 25 12:42:21.634932 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:21.634781    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"error-404-isvc-fd4b4-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4ab201cd-f34e-4d4c-aa98-179dee7a9ede-error-404-isvc-fd4b4-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-fd4b4-predictor-7694f9fcbf-n88zh\" (UID: \"4ab201cd-f34e-4d4c-aa98-179dee7a9ede\") " pod="kserve-ci-e2e-test/error-404-isvc-fd4b4-predictor-7694f9fcbf-n88zh"
May 25 12:42:21.634932 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:21.634805    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-jwtxw\" (UniqueName: \"kubernetes.io/projected/4ab201cd-f34e-4d4c-aa98-179dee7a9ede-kube-api-access-jwtxw\") pod \"error-404-isvc-fd4b4-predictor-7694f9fcbf-n88zh\" (UID: \"4ab201cd-f34e-4d4c-aa98-179dee7a9ede\") " pod="kserve-ci-e2e-test/error-404-isvc-fd4b4-predictor-7694f9fcbf-n88zh"
May 25 12:42:21.635512 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:21.635490    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"error-404-isvc-fd4b4-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4ab201cd-f34e-4d4c-aa98-179dee7a9ede-error-404-isvc-fd4b4-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-fd4b4-predictor-7694f9fcbf-n88zh\" (UID: \"4ab201cd-f34e-4d4c-aa98-179dee7a9ede\") " pod="kserve-ci-e2e-test/error-404-isvc-fd4b4-predictor-7694f9fcbf-n88zh"
May 25 12:42:21.637154 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:21.637132    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4ab201cd-f34e-4d4c-aa98-179dee7a9ede-proxy-tls\") pod \"error-404-isvc-fd4b4-predictor-7694f9fcbf-n88zh\" (UID: \"4ab201cd-f34e-4d4c-aa98-179dee7a9ede\") " pod="kserve-ci-e2e-test/error-404-isvc-fd4b4-predictor-7694f9fcbf-n88zh"
May 25 12:42:21.645309 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:21.645275    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwtxw\" (UniqueName: \"kubernetes.io/projected/4ab201cd-f34e-4d4c-aa98-179dee7a9ede-kube-api-access-jwtxw\") pod \"error-404-isvc-fd4b4-predictor-7694f9fcbf-n88zh\" (UID: \"4ab201cd-f34e-4d4c-aa98-179dee7a9ede\") " pod="kserve-ci-e2e-test/error-404-isvc-fd4b4-predictor-7694f9fcbf-n88zh"
May 25 12:42:21.657100 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:21.657069    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-fd4b4-predictor-7694f9fcbf-n88zh"
May 25 12:42:21.801317 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:21.801289    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-fd4b4-predictor-7694f9fcbf-n88zh"]
May 25 12:42:21.803260 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:42:21.803233    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ab201cd_f34e_4d4c_aa98_179dee7a9ede.slice/crio-c6561f98e36992f43cc6e864e3f68c6f26bd2cbfe06f0094784a81d0a9eb0bfc WatchSource:0}: Error finding container c6561f98e36992f43cc6e864e3f68c6f26bd2cbfe06f0094784a81d0a9eb0bfc: Status 404 returned error can't find the container with id c6561f98e36992f43cc6e864e3f68c6f26bd2cbfe06f0094784a81d0a9eb0bfc
May 25 12:42:21.805042 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:21.805025    2549 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 25 12:42:22.038720 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:22.038661    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/90636a58-53a5-4c10-864f-cdf6647bd677-proxy-tls\") pod \"success-200-isvc-fd4b4-predictor-59d7bccbf8-pl6qv\" (UID: \"90636a58-53a5-4c10-864f-cdf6647bd677\") " pod="kserve-ci-e2e-test/success-200-isvc-fd4b4-predictor-59d7bccbf8-pl6qv"
May 25 12:42:22.041105 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:22.041077    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/90636a58-53a5-4c10-864f-cdf6647bd677-proxy-tls\") pod \"success-200-isvc-fd4b4-predictor-59d7bccbf8-pl6qv\" (UID: \"90636a58-53a5-4c10-864f-cdf6647bd677\") " pod="kserve-ci-e2e-test/success-200-isvc-fd4b4-predictor-59d7bccbf8-pl6qv"
May 25 12:42:22.189531 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:22.189484    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-fd4b4-predictor-59d7bccbf8-pl6qv"
May 25 12:42:22.329429 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:22.329400    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-fd4b4-predictor-59d7bccbf8-pl6qv"]
May 25 12:42:22.331474 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:42:22.331441    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90636a58_53a5_4c10_864f_cdf6647bd677.slice/crio-d2fe48e34e138cda82f963808119b3328d002457d20cd1379d35013976a49241 WatchSource:0}: Error finding container d2fe48e34e138cda82f963808119b3328d002457d20cd1379d35013976a49241: Status 404 returned error can't find the container with id d2fe48e34e138cda82f963808119b3328d002457d20cd1379d35013976a49241
May 25 12:42:22.375834 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:22.375803    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-fd4b4-predictor-7694f9fcbf-n88zh" event={"ID":"4ab201cd-f34e-4d4c-aa98-179dee7a9ede","Type":"ContainerStarted","Data":"1381cf1891a17ff8634ea9fadd2649a636c0c9cdb9b2f6073ce2385969fceeeb"}
May 25 12:42:22.375991 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:22.375846    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-fd4b4-predictor-7694f9fcbf-n88zh" event={"ID":"4ab201cd-f34e-4d4c-aa98-179dee7a9ede","Type":"ContainerStarted","Data":"90a698ddb71598c0742e56099de7a64d97eedf2103ce534cff37a4de4b7ecdf3"}
May 25 12:42:22.375991 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:22.375858    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-fd4b4-predictor-7694f9fcbf-n88zh" event={"ID":"4ab201cd-f34e-4d4c-aa98-179dee7a9ede","Type":"ContainerStarted","Data":"c6561f98e36992f43cc6e864e3f68c6f26bd2cbfe06f0094784a81d0a9eb0bfc"}
May 25 12:42:22.375991 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:22.375916    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-fd4b4-predictor-7694f9fcbf-n88zh"
May 25 12:42:22.377446 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:22.377420    2549 generic.go:358] "Generic (PLEG): container finished" podID="3f43e011-dd0d-4717-8d72-3e2882bb8650" containerID="b234d4a0306944df8548d229741e8cbfa6101f667f0b33781f772e33652514fd" exitCode=2
May 25 12:42:22.377552 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:22.377484    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-155ee-predictor-57c99b6f99-mtr6n" event={"ID":"3f43e011-dd0d-4717-8d72-3e2882bb8650","Type":"ContainerDied","Data":"b234d4a0306944df8548d229741e8cbfa6101f667f0b33781f772e33652514fd"}
May 25 12:42:22.378553 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:22.378529    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-fd4b4-predictor-59d7bccbf8-pl6qv" event={"ID":"90636a58-53a5-4c10-864f-cdf6647bd677","Type":"ContainerStarted","Data":"d2fe48e34e138cda82f963808119b3328d002457d20cd1379d35013976a49241"}
May 25 12:42:22.397124 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:22.397075    2549 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-fd4b4-predictor-7694f9fcbf-n88zh" podStartSLOduration=1.397057385 podStartE2EDuration="1.397057385s" podCreationTimestamp="2026-05-25 12:42:21 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 12:42:22.394908042 +0000 UTC m=+1746.110723330" watchObservedRunningTime="2026-05-25 12:42:22.397057385 +0000 UTC m=+1746.112872749"
May 25 12:42:23.385049 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:23.385002    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-fd4b4-predictor-59d7bccbf8-pl6qv" event={"ID":"90636a58-53a5-4c10-864f-cdf6647bd677","Type":"ContainerStarted","Data":"adbe33e7faa9651120a9b1d2516e315b69fdb91e5837809a23b9a6c9fe64ff9e"}
May 25 12:42:23.385049 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:23.385057    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-fd4b4-predictor-59d7bccbf8-pl6qv" event={"ID":"90636a58-53a5-4c10-864f-cdf6647bd677","Type":"ContainerStarted","Data":"c0d259710e691108fc4e62a1f77b759332965e59359d57ef9a0334c7941c916a"}
May 25 12:42:23.385588 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:23.385080    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-fd4b4-predictor-59d7bccbf8-pl6qv"
May 25 12:42:23.385588 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:23.385096    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-fd4b4-predictor-59d7bccbf8-pl6qv"
May 25 12:42:23.385588 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:23.385542    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-fd4b4-predictor-7694f9fcbf-n88zh"
May 25 12:42:23.387421 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:23.387387    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-fd4b4-predictor-7694f9fcbf-n88zh" podUID="4ab201cd-f34e-4d4c-aa98-179dee7a9ede" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
May 25 12:42:23.387421 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:23.387407    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-fd4b4-predictor-59d7bccbf8-pl6qv" podUID="90636a58-53a5-4c10-864f-cdf6647bd677" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
May 25 12:42:23.405284 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:23.405223    2549 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-fd4b4-predictor-59d7bccbf8-pl6qv" podStartSLOduration=2.405208795 podStartE2EDuration="2.405208795s" podCreationTimestamp="2026-05-25 12:42:21 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 12:42:23.404616099 +0000 UTC m=+1747.120431388" watchObservedRunningTime="2026-05-25 12:42:23.405208795 +0000 UTC m=+1747.121024076"
May 25 12:42:24.253738 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:24.253693    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-155ee-predictor-57c99b6f99-mtr6n" podUID="3f43e011-dd0d-4717-8d72-3e2882bb8650" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.45:8643/healthz\": dial tcp 10.134.0.45:8643: connect: connection refused"
May 25 12:42:24.253910 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:24.253763    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-155ee-predictor-6b9f6fd99b-qvjql" podUID="efcb96a7-a292-4595-880f-e7670a2d9928" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.44:8643/healthz\": dial tcp 10.134.0.44:8643: connect: connection refused"
May 25 12:42:24.257342 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:24.257314    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-155ee-predictor-6b9f6fd99b-qvjql" podUID="efcb96a7-a292-4595-880f-e7670a2d9928" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
May 25 12:42:24.259662 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:24.259638    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-155ee-predictor-57c99b6f99-mtr6n" podUID="3f43e011-dd0d-4717-8d72-3e2882bb8650" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
May 25 12:42:24.391108 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:24.390754    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-fd4b4-predictor-59d7bccbf8-pl6qv" podUID="90636a58-53a5-4c10-864f-cdf6647bd677" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
May 25 12:42:24.391108 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:24.390821    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-fd4b4-predictor-7694f9fcbf-n88zh" podUID="4ab201cd-f34e-4d4c-aa98-179dee7a9ede" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
May 25 12:42:24.687741 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:24.687710    2549 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-155ee-predictor-6b9f6fd99b-qvjql"
May 25 12:42:24.763991 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:24.763903    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"success-200-isvc-155ee-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/efcb96a7-a292-4595-880f-e7670a2d9928-success-200-isvc-155ee-kube-rbac-proxy-sar-config\") pod \"efcb96a7-a292-4595-880f-e7670a2d9928\" (UID: \"efcb96a7-a292-4595-880f-e7670a2d9928\") "
May 25 12:42:24.763991 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:24.763974    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sqxnk\" (UniqueName: \"kubernetes.io/projected/efcb96a7-a292-4595-880f-e7670a2d9928-kube-api-access-sqxnk\") pod \"efcb96a7-a292-4595-880f-e7670a2d9928\" (UID: \"efcb96a7-a292-4595-880f-e7670a2d9928\") "
May 25 12:42:24.764235 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:24.764029    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/efcb96a7-a292-4595-880f-e7670a2d9928-proxy-tls\") pod \"efcb96a7-a292-4595-880f-e7670a2d9928\" (UID: \"efcb96a7-a292-4595-880f-e7670a2d9928\") "
May 25 12:42:24.764367 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:24.764339    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/efcb96a7-a292-4595-880f-e7670a2d9928-success-200-isvc-155ee-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "success-200-isvc-155ee-kube-rbac-proxy-sar-config") pod "efcb96a7-a292-4595-880f-e7670a2d9928" (UID: "efcb96a7-a292-4595-880f-e7670a2d9928"). InnerVolumeSpecName "success-200-isvc-155ee-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 12:42:24.766170 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:24.766144    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/efcb96a7-a292-4595-880f-e7670a2d9928-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "efcb96a7-a292-4595-880f-e7670a2d9928" (UID: "efcb96a7-a292-4595-880f-e7670a2d9928"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 12:42:24.766348 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:24.766324    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efcb96a7-a292-4595-880f-e7670a2d9928-kube-api-access-sqxnk" (OuterVolumeSpecName: "kube-api-access-sqxnk") pod "efcb96a7-a292-4595-880f-e7670a2d9928" (UID: "efcb96a7-a292-4595-880f-e7670a2d9928"). InnerVolumeSpecName "kube-api-access-sqxnk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 12:42:24.804524 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:24.804500    2549 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-155ee-predictor-57c99b6f99-mtr6n"
May 25 12:42:24.864539 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:24.864516    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dxjlh\" (UniqueName: \"kubernetes.io/projected/3f43e011-dd0d-4717-8d72-3e2882bb8650-kube-api-access-dxjlh\") pod \"3f43e011-dd0d-4717-8d72-3e2882bb8650\" (UID: \"3f43e011-dd0d-4717-8d72-3e2882bb8650\") "
May 25 12:42:24.864664 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:24.864552    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"error-404-isvc-155ee-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3f43e011-dd0d-4717-8d72-3e2882bb8650-error-404-isvc-155ee-kube-rbac-proxy-sar-config\") pod \"3f43e011-dd0d-4717-8d72-3e2882bb8650\" (UID: \"3f43e011-dd0d-4717-8d72-3e2882bb8650\") "
May 25 12:42:24.864664 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:24.864589    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3f43e011-dd0d-4717-8d72-3e2882bb8650-proxy-tls\") pod \"3f43e011-dd0d-4717-8d72-3e2882bb8650\" (UID: \"3f43e011-dd0d-4717-8d72-3e2882bb8650\") "
May 25 12:42:24.865002 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:24.864968    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f43e011-dd0d-4717-8d72-3e2882bb8650-error-404-isvc-155ee-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "error-404-isvc-155ee-kube-rbac-proxy-sar-config") pod "3f43e011-dd0d-4717-8d72-3e2882bb8650" (UID: "3f43e011-dd0d-4717-8d72-3e2882bb8650"). InnerVolumeSpecName "error-404-isvc-155ee-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 12:42:24.865133 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:24.865108    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sqxnk\" (UniqueName: \"kubernetes.io/projected/efcb96a7-a292-4595-880f-e7670a2d9928-kube-api-access-sqxnk\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
May 25 12:42:24.865232 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:24.865130    2549 reconciler_common.go:299] "Volume detached for volume \"error-404-isvc-155ee-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3f43e011-dd0d-4717-8d72-3e2882bb8650-error-404-isvc-155ee-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
May 25 12:42:24.865232 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:24.865148    2549 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/efcb96a7-a292-4595-880f-e7670a2d9928-proxy-tls\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
May 25 12:42:24.865232 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:24.865167    2549 reconciler_common.go:299] "Volume detached for volume \"success-200-isvc-155ee-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/efcb96a7-a292-4595-880f-e7670a2d9928-success-200-isvc-155ee-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
May 25 12:42:24.866760 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:24.866734    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f43e011-dd0d-4717-8d72-3e2882bb8650-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3f43e011-dd0d-4717-8d72-3e2882bb8650" (UID: "3f43e011-dd0d-4717-8d72-3e2882bb8650"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 12:42:24.866855 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:24.866810    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f43e011-dd0d-4717-8d72-3e2882bb8650-kube-api-access-dxjlh" (OuterVolumeSpecName: "kube-api-access-dxjlh") pod "3f43e011-dd0d-4717-8d72-3e2882bb8650" (UID: "3f43e011-dd0d-4717-8d72-3e2882bb8650"). InnerVolumeSpecName "kube-api-access-dxjlh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 12:42:24.966335 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:24.966302    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dxjlh\" (UniqueName: \"kubernetes.io/projected/3f43e011-dd0d-4717-8d72-3e2882bb8650-kube-api-access-dxjlh\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
May 25 12:42:24.966335 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:24.966332    2549 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3f43e011-dd0d-4717-8d72-3e2882bb8650-proxy-tls\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
May 25 12:42:25.395978 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:25.395942    2549 generic.go:358] "Generic (PLEG): container finished" podID="efcb96a7-a292-4595-880f-e7670a2d9928" containerID="705521478faa138c09b52e9f4f8bc7ae915575fc304e739b0110b0137fedee21" exitCode=0
May 25 12:42:25.396426 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:25.396010    2549 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-155ee-predictor-6b9f6fd99b-qvjql"
May 25 12:42:25.396426 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:25.396021    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-155ee-predictor-6b9f6fd99b-qvjql" event={"ID":"efcb96a7-a292-4595-880f-e7670a2d9928","Type":"ContainerDied","Data":"705521478faa138c09b52e9f4f8bc7ae915575fc304e739b0110b0137fedee21"}
May 25 12:42:25.396426 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:25.396061    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-155ee-predictor-6b9f6fd99b-qvjql" event={"ID":"efcb96a7-a292-4595-880f-e7670a2d9928","Type":"ContainerDied","Data":"a0224c025846d15e1d902708e01ebfd4bde6e06915bbc7ca35d02d369c221582"}
May 25 12:42:25.396426 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:25.396077    2549 scope.go:117] "RemoveContainer" containerID="35cba0cdaa0151f32daa59468dfafa895877d26e02ea9dae0e2bebe8e6b2c36a"
May 25 12:42:25.397690 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:25.397667    2549 generic.go:358] "Generic (PLEG): container finished" podID="3f43e011-dd0d-4717-8d72-3e2882bb8650" containerID="a6bc0a84961219878a56ca53f42d73fb8f31d3a695ce162918376d483eb1a9d1" exitCode=0
May 25 12:42:25.397794 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:25.397721    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-155ee-predictor-57c99b6f99-mtr6n" event={"ID":"3f43e011-dd0d-4717-8d72-3e2882bb8650","Type":"ContainerDied","Data":"a6bc0a84961219878a56ca53f42d73fb8f31d3a695ce162918376d483eb1a9d1"}
May 25 12:42:25.397794 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:25.397747    2549 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-155ee-predictor-57c99b6f99-mtr6n"
May 25 12:42:25.397872 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:25.397749    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-155ee-predictor-57c99b6f99-mtr6n" event={"ID":"3f43e011-dd0d-4717-8d72-3e2882bb8650","Type":"ContainerDied","Data":"30076f522f43210a25ca0e03f272eb59d2dfd223c31e20a5f3dcac7cc3ba3ee9"}
May 25 12:42:25.406926 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:25.406903    2549 scope.go:117] "RemoveContainer" containerID="705521478faa138c09b52e9f4f8bc7ae915575fc304e739b0110b0137fedee21"
May 25 12:42:25.415023 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:25.414998    2549 scope.go:117] "RemoveContainer" containerID="35cba0cdaa0151f32daa59468dfafa895877d26e02ea9dae0e2bebe8e6b2c36a"
May 25 12:42:25.415416 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:42:25.415392    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35cba0cdaa0151f32daa59468dfafa895877d26e02ea9dae0e2bebe8e6b2c36a\": container with ID starting with 35cba0cdaa0151f32daa59468dfafa895877d26e02ea9dae0e2bebe8e6b2c36a not found: ID does not exist" containerID="35cba0cdaa0151f32daa59468dfafa895877d26e02ea9dae0e2bebe8e6b2c36a"
May 25 12:42:25.415519 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:25.415429    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35cba0cdaa0151f32daa59468dfafa895877d26e02ea9dae0e2bebe8e6b2c36a"} err="failed to get container status \"35cba0cdaa0151f32daa59468dfafa895877d26e02ea9dae0e2bebe8e6b2c36a\": rpc error: code = NotFound desc = could not find container \"35cba0cdaa0151f32daa59468dfafa895877d26e02ea9dae0e2bebe8e6b2c36a\": container with ID starting with 35cba0cdaa0151f32daa59468dfafa895877d26e02ea9dae0e2bebe8e6b2c36a not found: ID does not exist"
May 25 12:42:25.415519 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:25.415455    2549 scope.go:117] "RemoveContainer" containerID="705521478faa138c09b52e9f4f8bc7ae915575fc304e739b0110b0137fedee21"
May 25 12:42:25.415894 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:42:25.415865    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"705521478faa138c09b52e9f4f8bc7ae915575fc304e739b0110b0137fedee21\": container with ID starting with 705521478faa138c09b52e9f4f8bc7ae915575fc304e739b0110b0137fedee21 not found: ID does not exist" containerID="705521478faa138c09b52e9f4f8bc7ae915575fc304e739b0110b0137fedee21"
May 25 12:42:25.415980 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:25.415900    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"705521478faa138c09b52e9f4f8bc7ae915575fc304e739b0110b0137fedee21"} err="failed to get container status \"705521478faa138c09b52e9f4f8bc7ae915575fc304e739b0110b0137fedee21\": rpc error: code = NotFound desc = could not find container \"705521478faa138c09b52e9f4f8bc7ae915575fc304e739b0110b0137fedee21\": container with ID starting with 705521478faa138c09b52e9f4f8bc7ae915575fc304e739b0110b0137fedee21 not found: ID does not exist"
May 25 12:42:25.415980 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:25.415922    2549 scope.go:117] "RemoveContainer" containerID="b234d4a0306944df8548d229741e8cbfa6101f667f0b33781f772e33652514fd"
May 25 12:42:25.417708 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:25.417681    2549 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-155ee-predictor-6b9f6fd99b-qvjql"]
May 25 12:42:25.422539 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:25.422507    2549 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-155ee-predictor-6b9f6fd99b-qvjql"]
May 25 12:42:25.424981 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:25.424965    2549 scope.go:117] "RemoveContainer" containerID="a6bc0a84961219878a56ca53f42d73fb8f31d3a695ce162918376d483eb1a9d1"
May 25 12:42:25.432766 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:25.432732    2549 scope.go:117] "RemoveContainer" containerID="b234d4a0306944df8548d229741e8cbfa6101f667f0b33781f772e33652514fd"
May 25 12:42:25.433007 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:42:25.432988    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b234d4a0306944df8548d229741e8cbfa6101f667f0b33781f772e33652514fd\": container with ID starting with b234d4a0306944df8548d229741e8cbfa6101f667f0b33781f772e33652514fd not found: ID does not exist" containerID="b234d4a0306944df8548d229741e8cbfa6101f667f0b33781f772e33652514fd"
May 25 12:42:25.433082 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:25.433019    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b234d4a0306944df8548d229741e8cbfa6101f667f0b33781f772e33652514fd"} err="failed to get container status \"b234d4a0306944df8548d229741e8cbfa6101f667f0b33781f772e33652514fd\": rpc error: code = NotFound desc = could not find container \"b234d4a0306944df8548d229741e8cbfa6101f667f0b33781f772e33652514fd\": container with ID starting with b234d4a0306944df8548d229741e8cbfa6101f667f0b33781f772e33652514fd not found: ID does not exist"
May 25 12:42:25.433082 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:25.433046    2549 scope.go:117] "RemoveContainer" containerID="a6bc0a84961219878a56ca53f42d73fb8f31d3a695ce162918376d483eb1a9d1"
May 25 12:42:25.433334 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:42:25.433317    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6bc0a84961219878a56ca53f42d73fb8f31d3a695ce162918376d483eb1a9d1\": container with ID starting with a6bc0a84961219878a56ca53f42d73fb8f31d3a695ce162918376d483eb1a9d1 not found: ID does not exist" containerID="a6bc0a84961219878a56ca53f42d73fb8f31d3a695ce162918376d483eb1a9d1"
May 25 12:42:25.433381 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:25.433341    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6bc0a84961219878a56ca53f42d73fb8f31d3a695ce162918376d483eb1a9d1"} err="failed to get container status \"a6bc0a84961219878a56ca53f42d73fb8f31d3a695ce162918376d483eb1a9d1\": rpc error: code = NotFound desc = could not find container \"a6bc0a84961219878a56ca53f42d73fb8f31d3a695ce162918376d483eb1a9d1\": container with ID starting with a6bc0a84961219878a56ca53f42d73fb8f31d3a695ce162918376d483eb1a9d1 not found: ID does not exist"
May 25 12:42:25.438926 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:25.438904    2549 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-155ee-predictor-57c99b6f99-mtr6n"]
May 25 12:42:25.443427 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:25.443404    2549 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-155ee-predictor-57c99b6f99-mtr6n"]
May 25 12:42:26.848882 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:26.848846    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3f43e011-dd0d-4717-8d72-3e2882bb8650" path="/var/lib/kubelet/pods/3f43e011-dd0d-4717-8d72-3e2882bb8650/volumes"
May 25 12:42:26.849306 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:26.849291    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="efcb96a7-a292-4595-880f-e7670a2d9928" path="/var/lib/kubelet/pods/efcb96a7-a292-4595-880f-e7670a2d9928/volumes"
May 25 12:42:29.395896 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:29.395867    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-fd4b4-predictor-7694f9fcbf-n88zh"
May 25 12:42:29.396412 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:29.396383    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-fd4b4-predictor-59d7bccbf8-pl6qv"
May 25 12:42:29.396494 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:29.396431    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-fd4b4-predictor-7694f9fcbf-n88zh" podUID="4ab201cd-f34e-4d4c-aa98-179dee7a9ede" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
May 25 12:42:29.396788 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:29.396764    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-fd4b4-predictor-59d7bccbf8-pl6qv" podUID="90636a58-53a5-4c10-864f-cdf6647bd677" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
May 25 12:42:39.396786 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:39.396742    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-fd4b4-predictor-7694f9fcbf-n88zh" podUID="4ab201cd-f34e-4d4c-aa98-179dee7a9ede" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
May 25 12:42:39.397232 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:39.396880    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-fd4b4-predictor-59d7bccbf8-pl6qv" podUID="90636a58-53a5-4c10-864f-cdf6647bd677" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
May 25 12:42:49.396769 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:49.396727    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-fd4b4-predictor-7694f9fcbf-n88zh" podUID="4ab201cd-f34e-4d4c-aa98-179dee7a9ede" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
May 25 12:42:49.397184 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:49.396867    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-fd4b4-predictor-59d7bccbf8-pl6qv" podUID="90636a58-53a5-4c10-864f-cdf6647bd677" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
May 25 12:42:51.163581 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:51.163542    2549 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-4d75e-predictor-98c496fb-m8ddw"]
May 25 12:42:51.164063 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:51.163834    2549 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-4d75e-predictor-98c496fb-m8ddw" podUID="b5232f59-357c-4a8a-a062-13d1161f8a81" containerName="kserve-container" containerID="cri-o://298328194cbed0020fcfc4bd1bc4dfcace27fc89e79fb5abe29c99a22b4f01c8" gracePeriod=30
May 25 12:42:51.164063 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:51.163897    2549 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-4d75e-predictor-98c496fb-m8ddw" podUID="b5232f59-357c-4a8a-a062-13d1161f8a81" containerName="kube-rbac-proxy" containerID="cri-o://0e589999a9931208d6901d9f586a935980ebc01727722bebd4a22a3a3f2cb81e" gracePeriod=30
May 25 12:42:51.187467 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:51.187435    2549 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-09598-predictor-587794bf84-zl6dc"]
May 25 12:42:51.187881 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:51.187867    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f43e011-dd0d-4717-8d72-3e2882bb8650" containerName="kube-rbac-proxy"
May 25 12:42:51.187939 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:51.187882    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f43e011-dd0d-4717-8d72-3e2882bb8650" containerName="kube-rbac-proxy"
May 25 12:42:51.187939 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:51.187900    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="efcb96a7-a292-4595-880f-e7670a2d9928" containerName="kserve-container"
May 25 12:42:51.187939 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:51.187906    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="efcb96a7-a292-4595-880f-e7670a2d9928" containerName="kserve-container"
May 25 12:42:51.187939 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:51.187912    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f43e011-dd0d-4717-8d72-3e2882bb8650" containerName="kserve-container"
May 25 12:42:51.187939 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:51.187919    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f43e011-dd0d-4717-8d72-3e2882bb8650" containerName="kserve-container"
May 25 12:42:51.187939 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:51.187930    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="efcb96a7-a292-4595-880f-e7670a2d9928" containerName="kube-rbac-proxy"
May 25 12:42:51.187939 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:51.187935    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="efcb96a7-a292-4595-880f-e7670a2d9928" containerName="kube-rbac-proxy"
May 25 12:42:51.188160 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:51.188000    2549 memory_manager.go:356] "RemoveStaleState removing state" podUID="efcb96a7-a292-4595-880f-e7670a2d9928" containerName="kube-rbac-proxy"
May 25 12:42:51.188160 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:51.188007    2549 memory_manager.go:356] "RemoveStaleState removing state" podUID="3f43e011-dd0d-4717-8d72-3e2882bb8650" containerName="kube-rbac-proxy"
May 25 12:42:51.188160 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:51.188016    2549 memory_manager.go:356] "RemoveStaleState removing state" podUID="3f43e011-dd0d-4717-8d72-3e2882bb8650" containerName="kserve-container"
May 25 12:42:51.188160 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:51.188025    2549 memory_manager.go:356] "RemoveStaleState removing state" podUID="efcb96a7-a292-4595-880f-e7670a2d9928" containerName="kserve-container"
May 25 12:42:51.191585 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:51.191560    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-09598-predictor-587794bf84-zl6dc"
May 25 12:42:51.194634 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:51.194606    2549 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-09598-kube-rbac-proxy-sar-config\""
May 25 12:42:51.194932 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:51.194871    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-09598-predictor-serving-cert\""
May 25 12:42:51.204095 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:51.204070    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-09598-predictor-587794bf84-zl6dc"]
May 25 12:42:51.224065 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:51.224036    2549 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-4d75e-predictor-5c8bb955c5-mqnb5"]
May 25 12:42:51.224358 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:51.224326    2549 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-4d75e-predictor-5c8bb955c5-mqnb5" podUID="d0a4c463-e906-4147-b266-b3d9b2fd6c29" containerName="kserve-container" containerID="cri-o://f28b53db05c8eb0d7ccd2e1105707864e565f141a219a60d1d01e58e48298b4d" gracePeriod=30
May 25 12:42:51.224504 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:51.224369    2549 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-4d75e-predictor-5c8bb955c5-mqnb5" podUID="d0a4c463-e906-4147-b266-b3d9b2fd6c29" containerName="kube-rbac-proxy" containerID="cri-o://34dca45b8b17a4dbf5731d8a197ac7f86e9338dcc48dac93149c4a73cdc4f4c3" gracePeriod=30
May 25 12:42:51.305964 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:51.305917    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"success-200-isvc-09598-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7b0bcced-f9ba-4d60-a892-a604b11c68af-success-200-isvc-09598-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-09598-predictor-587794bf84-zl6dc\" (UID: \"7b0bcced-f9ba-4d60-a892-a604b11c68af\") " pod="kserve-ci-e2e-test/success-200-isvc-09598-predictor-587794bf84-zl6dc"
May 25 12:42:51.306159 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:51.306004    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7b0bcced-f9ba-4d60-a892-a604b11c68af-proxy-tls\") pod \"success-200-isvc-09598-predictor-587794bf84-zl6dc\" (UID: \"7b0bcced-f9ba-4d60-a892-a604b11c68af\") " pod="kserve-ci-e2e-test/success-200-isvc-09598-predictor-587794bf84-zl6dc"
May 25 12:42:51.306159 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:51.306126    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g24bq\" (UniqueName: \"kubernetes.io/projected/7b0bcced-f9ba-4d60-a892-a604b11c68af-kube-api-access-g24bq\") pod \"success-200-isvc-09598-predictor-587794bf84-zl6dc\" (UID: \"7b0bcced-f9ba-4d60-a892-a604b11c68af\") " pod="kserve-ci-e2e-test/success-200-isvc-09598-predictor-587794bf84-zl6dc"
May 25 12:42:51.327570 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:51.327102    2549 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-09598-predictor-7c4d94fb5f-hqnbq"]
May 25 12:42:51.331436 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:51.331406    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-09598-predictor-7c4d94fb5f-hqnbq"
May 25 12:42:51.335049 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:51.334730    2549 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-09598-kube-rbac-proxy-sar-config\""
May 25 12:42:51.338389 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:51.338338    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-09598-predictor-serving-cert\""
May 25 12:42:51.339530 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:51.339504    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-09598-predictor-7c4d94fb5f-hqnbq"]
May 25 12:42:51.407309 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:51.407275    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"error-404-isvc-09598-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/017fafb4-c256-47d2-b271-a6c2feff2636-error-404-isvc-09598-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-09598-predictor-7c4d94fb5f-hqnbq\" (UID: \"017fafb4-c256-47d2-b271-a6c2feff2636\") " pod="kserve-ci-e2e-test/error-404-isvc-09598-predictor-7c4d94fb5f-hqnbq"
May 25 12:42:51.407468 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:51.407331    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-g24bq\" (UniqueName: \"kubernetes.io/projected/7b0bcced-f9ba-4d60-a892-a604b11c68af-kube-api-access-g24bq\") pod \"success-200-isvc-09598-predictor-587794bf84-zl6dc\" (UID: \"7b0bcced-f9ba-4d60-a892-a604b11c68af\") " pod="kserve-ci-e2e-test/success-200-isvc-09598-predictor-587794bf84-zl6dc"
May 25 12:42:51.407468 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:51.407449    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"success-200-isvc-09598-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7b0bcced-f9ba-4d60-a892-a604b11c68af-success-200-isvc-09598-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-09598-predictor-587794bf84-zl6dc\" (UID: \"7b0bcced-f9ba-4d60-a892-a604b11c68af\") " pod="kserve-ci-e2e-test/success-200-isvc-09598-predictor-587794bf84-zl6dc"
May 25 12:42:51.407546 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:51.407520    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7b0bcced-f9ba-4d60-a892-a604b11c68af-proxy-tls\") pod \"success-200-isvc-09598-predictor-587794bf84-zl6dc\" (UID: \"7b0bcced-f9ba-4d60-a892-a604b11c68af\") " pod="kserve-ci-e2e-test/success-200-isvc-09598-predictor-587794bf84-zl6dc"
May 25 12:42:51.407605 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:51.407569    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gmd8\" (UniqueName: \"kubernetes.io/projected/017fafb4-c256-47d2-b271-a6c2feff2636-kube-api-access-5gmd8\") pod \"error-404-isvc-09598-predictor-7c4d94fb5f-hqnbq\" (UID: \"017fafb4-c256-47d2-b271-a6c2feff2636\") " pod="kserve-ci-e2e-test/error-404-isvc-09598-predictor-7c4d94fb5f-hqnbq"
May 25 12:42:51.407652 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:51.407610    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/017fafb4-c256-47d2-b271-a6c2feff2636-proxy-tls\") pod \"error-404-isvc-09598-predictor-7c4d94fb5f-hqnbq\" (UID: \"017fafb4-c256-47d2-b271-a6c2feff2636\") " pod="kserve-ci-e2e-test/error-404-isvc-09598-predictor-7c4d94fb5f-hqnbq"
May 25 12:42:51.407701 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:42:51.407667    2549 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-09598-predictor-serving-cert: secret "success-200-isvc-09598-predictor-serving-cert" not found
May 25 12:42:51.407755 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:42:51.407746    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7b0bcced-f9ba-4d60-a892-a604b11c68af-proxy-tls podName:7b0bcced-f9ba-4d60-a892-a604b11c68af nodeName:}" failed. No retries permitted until 2026-05-25 12:42:51.907721719 +0000 UTC m=+1775.623536984 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/7b0bcced-f9ba-4d60-a892-a604b11c68af-proxy-tls") pod "success-200-isvc-09598-predictor-587794bf84-zl6dc" (UID: "7b0bcced-f9ba-4d60-a892-a604b11c68af") : secret "success-200-isvc-09598-predictor-serving-cert" not found
May 25 12:42:51.408284 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:51.408254    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"success-200-isvc-09598-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7b0bcced-f9ba-4d60-a892-a604b11c68af-success-200-isvc-09598-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-09598-predictor-587794bf84-zl6dc\" (UID: \"7b0bcced-f9ba-4d60-a892-a604b11c68af\") " pod="kserve-ci-e2e-test/success-200-isvc-09598-predictor-587794bf84-zl6dc"
May 25 12:42:51.416989 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:51.416924    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g24bq\" (UniqueName: \"kubernetes.io/projected/7b0bcced-f9ba-4d60-a892-a604b11c68af-kube-api-access-g24bq\") pod \"success-200-isvc-09598-predictor-587794bf84-zl6dc\" (UID: \"7b0bcced-f9ba-4d60-a892-a604b11c68af\") " pod="kserve-ci-e2e-test/success-200-isvc-09598-predictor-587794bf84-zl6dc"
May 25 12:42:51.498490 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:51.498456    2549 generic.go:358] "Generic (PLEG): container finished" podID="d0a4c463-e906-4147-b266-b3d9b2fd6c29" containerID="34dca45b8b17a4dbf5731d8a197ac7f86e9338dcc48dac93149c4a73cdc4f4c3" exitCode=2
May 25 12:42:51.498674 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:51.498536    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-4d75e-predictor-5c8bb955c5-mqnb5" event={"ID":"d0a4c463-e906-4147-b266-b3d9b2fd6c29","Type":"ContainerDied","Data":"34dca45b8b17a4dbf5731d8a197ac7f86e9338dcc48dac93149c4a73cdc4f4c3"}
May 25 12:42:51.500175 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:51.500153    2549 generic.go:358] "Generic (PLEG): container finished" podID="b5232f59-357c-4a8a-a062-13d1161f8a81" containerID="0e589999a9931208d6901d9f586a935980ebc01727722bebd4a22a3a3f2cb81e" exitCode=2
May 25 12:42:51.500288 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:51.500205    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-4d75e-predictor-98c496fb-m8ddw" event={"ID":"b5232f59-357c-4a8a-a062-13d1161f8a81","Type":"ContainerDied","Data":"0e589999a9931208d6901d9f586a935980ebc01727722bebd4a22a3a3f2cb81e"}
May 25 12:42:51.508823 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:51.508799    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-5gmd8\" (UniqueName: \"kubernetes.io/projected/017fafb4-c256-47d2-b271-a6c2feff2636-kube-api-access-5gmd8\") pod \"error-404-isvc-09598-predictor-7c4d94fb5f-hqnbq\" (UID: \"017fafb4-c256-47d2-b271-a6c2feff2636\") " pod="kserve-ci-e2e-test/error-404-isvc-09598-predictor-7c4d94fb5f-hqnbq"
May 25 12:42:51.508907 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:51.508833    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/017fafb4-c256-47d2-b271-a6c2feff2636-proxy-tls\") pod \"error-404-isvc-09598-predictor-7c4d94fb5f-hqnbq\" (UID: \"017fafb4-c256-47d2-b271-a6c2feff2636\") " pod="kserve-ci-e2e-test/error-404-isvc-09598-predictor-7c4d94fb5f-hqnbq"
May 25 12:42:51.508907 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:51.508872    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"error-404-isvc-09598-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/017fafb4-c256-47d2-b271-a6c2feff2636-error-404-isvc-09598-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-09598-predictor-7c4d94fb5f-hqnbq\" (UID: \"017fafb4-c256-47d2-b271-a6c2feff2636\") " pod="kserve-ci-e2e-test/error-404-isvc-09598-predictor-7c4d94fb5f-hqnbq"
May 25 12:42:51.509545 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:51.509523    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"error-404-isvc-09598-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/017fafb4-c256-47d2-b271-a6c2feff2636-error-404-isvc-09598-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-09598-predictor-7c4d94fb5f-hqnbq\" (UID: \"017fafb4-c256-47d2-b271-a6c2feff2636\") " pod="kserve-ci-e2e-test/error-404-isvc-09598-predictor-7c4d94fb5f-hqnbq"
May 25 12:42:51.511359 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:51.511337    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/017fafb4-c256-47d2-b271-a6c2feff2636-proxy-tls\") pod \"error-404-isvc-09598-predictor-7c4d94fb5f-hqnbq\" (UID: \"017fafb4-c256-47d2-b271-a6c2feff2636\") " pod="kserve-ci-e2e-test/error-404-isvc-09598-predictor-7c4d94fb5f-hqnbq"
May 25 12:42:51.518337 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:51.518314    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gmd8\" (UniqueName: \"kubernetes.io/projected/017fafb4-c256-47d2-b271-a6c2feff2636-kube-api-access-5gmd8\") pod \"error-404-isvc-09598-predictor-7c4d94fb5f-hqnbq\" (UID: \"017fafb4-c256-47d2-b271-a6c2feff2636\") " pod="kserve-ci-e2e-test/error-404-isvc-09598-predictor-7c4d94fb5f-hqnbq"
May 25 12:42:51.646764 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:51.646722    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-09598-predictor-7c4d94fb5f-hqnbq"
May 25 12:42:51.788235 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:51.788183    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-09598-predictor-7c4d94fb5f-hqnbq"]
May 25 12:42:51.790753 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:42:51.790722    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod017fafb4_c256_47d2_b271_a6c2feff2636.slice/crio-ab3b94ac3ac1cac71e95287117815d2eb28ef880185bf72256e971ba30c7b6f7 WatchSource:0}: Error finding container ab3b94ac3ac1cac71e95287117815d2eb28ef880185bf72256e971ba30c7b6f7: Status 404 returned error can't find the container with id ab3b94ac3ac1cac71e95287117815d2eb28ef880185bf72256e971ba30c7b6f7
May 25 12:42:51.913495 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:51.913451    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7b0bcced-f9ba-4d60-a892-a604b11c68af-proxy-tls\") pod \"success-200-isvc-09598-predictor-587794bf84-zl6dc\" (UID: \"7b0bcced-f9ba-4d60-a892-a604b11c68af\") " pod="kserve-ci-e2e-test/success-200-isvc-09598-predictor-587794bf84-zl6dc"
May 25 12:42:51.915894 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:51.915867    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7b0bcced-f9ba-4d60-a892-a604b11c68af-proxy-tls\") pod \"success-200-isvc-09598-predictor-587794bf84-zl6dc\" (UID: \"7b0bcced-f9ba-4d60-a892-a604b11c68af\") " pod="kserve-ci-e2e-test/success-200-isvc-09598-predictor-587794bf84-zl6dc"
May 25 12:42:52.106044 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:52.106011    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-09598-predictor-587794bf84-zl6dc"
May 25 12:42:52.255301 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:52.255268    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-09598-predictor-587794bf84-zl6dc"]
May 25 12:42:52.258610 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:42:52.258576    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b0bcced_f9ba_4d60_a892_a604b11c68af.slice/crio-9361707e236acf1ddd81c2a17f24853c33ca280d293cef92f56feb77593f582c WatchSource:0}: Error finding container 9361707e236acf1ddd81c2a17f24853c33ca280d293cef92f56feb77593f582c: Status 404 returned error can't find the container with id 9361707e236acf1ddd81c2a17f24853c33ca280d293cef92f56feb77593f582c
May 25 12:42:52.505630 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:52.505580    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-09598-predictor-7c4d94fb5f-hqnbq" event={"ID":"017fafb4-c256-47d2-b271-a6c2feff2636","Type":"ContainerStarted","Data":"c068f158d0fa76499f94bf39f2a656298d469fe3b0e411307e8b12a4db215002"}
May 25 12:42:52.505630 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:52.505627    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-09598-predictor-7c4d94fb5f-hqnbq" event={"ID":"017fafb4-c256-47d2-b271-a6c2feff2636","Type":"ContainerStarted","Data":"16addaa2fdd22777687d5c31501caea4ece0bdb18f98d82c4f946e7741cc221d"}
May 25 12:42:52.505864 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:52.505643    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-09598-predictor-7c4d94fb5f-hqnbq" event={"ID":"017fafb4-c256-47d2-b271-a6c2feff2636","Type":"ContainerStarted","Data":"ab3b94ac3ac1cac71e95287117815d2eb28ef880185bf72256e971ba30c7b6f7"}
May 25 12:42:52.505864 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:52.505707    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-09598-predictor-7c4d94fb5f-hqnbq"
May 25 12:42:52.507236 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:52.507213    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-09598-predictor-587794bf84-zl6dc" event={"ID":"7b0bcced-f9ba-4d60-a892-a604b11c68af","Type":"ContainerStarted","Data":"1db7e497da00c556f0d947e3c02c176f5827d2be67a1240b42e3986cd2b6ba41"}
May 25 12:42:52.507236 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:52.507237    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-09598-predictor-587794bf84-zl6dc" event={"ID":"7b0bcced-f9ba-4d60-a892-a604b11c68af","Type":"ContainerStarted","Data":"f8b3ae24913bc9ec0dd89022b47c218c91b785bd62b7e4830af85279caa9c4da"}
May 25 12:42:52.507389 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:52.507246    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-09598-predictor-587794bf84-zl6dc" event={"ID":"7b0bcced-f9ba-4d60-a892-a604b11c68af","Type":"ContainerStarted","Data":"9361707e236acf1ddd81c2a17f24853c33ca280d293cef92f56feb77593f582c"}
May 25 12:42:52.507389 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:52.507370    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-09598-predictor-587794bf84-zl6dc"
May 25 12:42:52.527770 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:52.527727    2549 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-09598-predictor-7c4d94fb5f-hqnbq" podStartSLOduration=1.52771116 podStartE2EDuration="1.52771116s" podCreationTimestamp="2026-05-25 12:42:51 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 12:42:52.525519361 +0000 UTC m=+1776.241334648" watchObservedRunningTime="2026-05-25 12:42:52.52771116 +0000 UTC m=+1776.243526446"
May 25 12:42:52.544501 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:52.544454    2549 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-09598-predictor-587794bf84-zl6dc" podStartSLOduration=1.544439264 podStartE2EDuration="1.544439264s" podCreationTimestamp="2026-05-25 12:42:51 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 12:42:52.543523027 +0000 UTC m=+1776.259338315" watchObservedRunningTime="2026-05-25 12:42:52.544439264 +0000 UTC m=+1776.260254550"
May 25 12:42:53.511791 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:53.511765    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-09598-predictor-7c4d94fb5f-hqnbq"
May 25 12:42:53.512304 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:53.511995    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-09598-predictor-587794bf84-zl6dc"
May 25 12:42:53.513021 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:53.512993    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-09598-predictor-587794bf84-zl6dc" podUID="7b0bcced-f9ba-4d60-a892-a604b11c68af" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
May 25 12:42:53.513105 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:53.513085    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-09598-predictor-7c4d94fb5f-hqnbq" podUID="017fafb4-c256-47d2-b271-a6c2feff2636" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
May 25 12:42:54.438950 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:54.438903    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-4d75e-predictor-5c8bb955c5-mqnb5" podUID="d0a4c463-e906-4147-b266-b3d9b2fd6c29" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.47:8643/healthz\": dial tcp 10.134.0.47:8643: connect: connection refused"
May 25 12:42:54.443148 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:54.443125    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-4d75e-predictor-5c8bb955c5-mqnb5" podUID="d0a4c463-e906-4147-b266-b3d9b2fd6c29" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
May 25 12:42:54.516280 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:54.516238    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-09598-predictor-7c4d94fb5f-hqnbq" podUID="017fafb4-c256-47d2-b271-a6c2feff2636" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
May 25 12:42:54.516653 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:54.516362    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-09598-predictor-587794bf84-zl6dc" podUID="7b0bcced-f9ba-4d60-a892-a604b11c68af" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
May 25 12:42:54.987042 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:54.987013    2549 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-4d75e-predictor-5c8bb955c5-mqnb5"
May 25 12:42:55.128246 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:55.128224    2549 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-4d75e-predictor-98c496fb-m8ddw"
May 25 12:42:55.143794 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:55.143771    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d0a4c463-e906-4147-b266-b3d9b2fd6c29-proxy-tls\") pod \"d0a4c463-e906-4147-b266-b3d9b2fd6c29\" (UID: \"d0a4c463-e906-4147-b266-b3d9b2fd6c29\") "
May 25 12:42:55.143937 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:55.143837    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"error-404-isvc-4d75e-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d0a4c463-e906-4147-b266-b3d9b2fd6c29-error-404-isvc-4d75e-kube-rbac-proxy-sar-config\") pod \"d0a4c463-e906-4147-b266-b3d9b2fd6c29\" (UID: \"d0a4c463-e906-4147-b266-b3d9b2fd6c29\") "
May 25 12:42:55.143937 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:55.143903    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-twxq9\" (UniqueName: \"kubernetes.io/projected/d0a4c463-e906-4147-b266-b3d9b2fd6c29-kube-api-access-twxq9\") pod \"d0a4c463-e906-4147-b266-b3d9b2fd6c29\" (UID: \"d0a4c463-e906-4147-b266-b3d9b2fd6c29\") "
May 25 12:42:55.144148 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:55.144124    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0a4c463-e906-4147-b266-b3d9b2fd6c29-error-404-isvc-4d75e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "error-404-isvc-4d75e-kube-rbac-proxy-sar-config") pod "d0a4c463-e906-4147-b266-b3d9b2fd6c29" (UID: "d0a4c463-e906-4147-b266-b3d9b2fd6c29"). InnerVolumeSpecName "error-404-isvc-4d75e-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 12:42:55.145903 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:55.145876    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0a4c463-e906-4147-b266-b3d9b2fd6c29-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d0a4c463-e906-4147-b266-b3d9b2fd6c29" (UID: "d0a4c463-e906-4147-b266-b3d9b2fd6c29"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 12:42:55.145990 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:55.145909    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0a4c463-e906-4147-b266-b3d9b2fd6c29-kube-api-access-twxq9" (OuterVolumeSpecName: "kube-api-access-twxq9") pod "d0a4c463-e906-4147-b266-b3d9b2fd6c29" (UID: "d0a4c463-e906-4147-b266-b3d9b2fd6c29"). InnerVolumeSpecName "kube-api-access-twxq9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 12:42:55.245474 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:55.245421    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b5232f59-357c-4a8a-a062-13d1161f8a81-proxy-tls\") pod \"b5232f59-357c-4a8a-a062-13d1161f8a81\" (UID: \"b5232f59-357c-4a8a-a062-13d1161f8a81\") "
May 25 12:42:55.245639 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:55.245537    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"success-200-isvc-4d75e-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b5232f59-357c-4a8a-a062-13d1161f8a81-success-200-isvc-4d75e-kube-rbac-proxy-sar-config\") pod \"b5232f59-357c-4a8a-a062-13d1161f8a81\" (UID: \"b5232f59-357c-4a8a-a062-13d1161f8a81\") "
May 25 12:42:55.245697 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:55.245640    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lrdhv\" (UniqueName: \"kubernetes.io/projected/b5232f59-357c-4a8a-a062-13d1161f8a81-kube-api-access-lrdhv\") pod \"b5232f59-357c-4a8a-a062-13d1161f8a81\" (UID: \"b5232f59-357c-4a8a-a062-13d1161f8a81\") "
May 25 12:42:55.245900 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:55.245871    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5232f59-357c-4a8a-a062-13d1161f8a81-success-200-isvc-4d75e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "success-200-isvc-4d75e-kube-rbac-proxy-sar-config") pod "b5232f59-357c-4a8a-a062-13d1161f8a81" (UID: "b5232f59-357c-4a8a-a062-13d1161f8a81"). InnerVolumeSpecName "success-200-isvc-4d75e-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 12:42:55.246047 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:55.245973    2549 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d0a4c463-e906-4147-b266-b3d9b2fd6c29-proxy-tls\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
May 25 12:42:55.246047 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:55.245996    2549 reconciler_common.go:299] "Volume detached for volume \"error-404-isvc-4d75e-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d0a4c463-e906-4147-b266-b3d9b2fd6c29-error-404-isvc-4d75e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
May 25 12:42:55.246047 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:55.246011    2549 reconciler_common.go:299] "Volume detached for volume \"success-200-isvc-4d75e-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b5232f59-357c-4a8a-a062-13d1161f8a81-success-200-isvc-4d75e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
May 25 12:42:55.246047 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:55.246026    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-twxq9\" (UniqueName: \"kubernetes.io/projected/d0a4c463-e906-4147-b266-b3d9b2fd6c29-kube-api-access-twxq9\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
May 25 12:42:55.247680 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:55.247658    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5232f59-357c-4a8a-a062-13d1161f8a81-kube-api-access-lrdhv" (OuterVolumeSpecName: "kube-api-access-lrdhv") pod "b5232f59-357c-4a8a-a062-13d1161f8a81" (UID: "b5232f59-357c-4a8a-a062-13d1161f8a81"). InnerVolumeSpecName "kube-api-access-lrdhv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 12:42:55.247680 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:55.247670    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5232f59-357c-4a8a-a062-13d1161f8a81-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b5232f59-357c-4a8a-a062-13d1161f8a81" (UID: "b5232f59-357c-4a8a-a062-13d1161f8a81"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 12:42:55.346739 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:55.346705    2549 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b5232f59-357c-4a8a-a062-13d1161f8a81-proxy-tls\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
May 25 12:42:55.346739 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:55.346740    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lrdhv\" (UniqueName: \"kubernetes.io/projected/b5232f59-357c-4a8a-a062-13d1161f8a81-kube-api-access-lrdhv\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
May 25 12:42:55.522287 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:55.522249    2549 generic.go:358] "Generic (PLEG): container finished" podID="b5232f59-357c-4a8a-a062-13d1161f8a81" containerID="298328194cbed0020fcfc4bd1bc4dfcace27fc89e79fb5abe29c99a22b4f01c8" exitCode=0
May 25 12:42:55.522725 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:55.522323    2549 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-4d75e-predictor-98c496fb-m8ddw"
May 25 12:42:55.522725 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:55.522329    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-4d75e-predictor-98c496fb-m8ddw" event={"ID":"b5232f59-357c-4a8a-a062-13d1161f8a81","Type":"ContainerDied","Data":"298328194cbed0020fcfc4bd1bc4dfcace27fc89e79fb5abe29c99a22b4f01c8"}
May 25 12:42:55.522725 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:55.522366    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-4d75e-predictor-98c496fb-m8ddw" event={"ID":"b5232f59-357c-4a8a-a062-13d1161f8a81","Type":"ContainerDied","Data":"debff1ab0c64f9a45e7c889864b30238945d4e7faf79b9c2b9b90efdecc47677"}
May 25 12:42:55.522725 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:55.522387    2549 scope.go:117] "RemoveContainer" containerID="0e589999a9931208d6901d9f586a935980ebc01727722bebd4a22a3a3f2cb81e"
May 25 12:42:55.523876 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:55.523851    2549 generic.go:358] "Generic (PLEG): container finished" podID="d0a4c463-e906-4147-b266-b3d9b2fd6c29" containerID="f28b53db05c8eb0d7ccd2e1105707864e565f141a219a60d1d01e58e48298b4d" exitCode=0
May 25 12:42:55.524005 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:55.523962    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-4d75e-predictor-5c8bb955c5-mqnb5" event={"ID":"d0a4c463-e906-4147-b266-b3d9b2fd6c29","Type":"ContainerDied","Data":"f28b53db05c8eb0d7ccd2e1105707864e565f141a219a60d1d01e58e48298b4d"}
May 25 12:42:55.524005 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:55.523988    2549 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-4d75e-predictor-5c8bb955c5-mqnb5"
May 25 12:42:55.524144 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:55.523992    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-4d75e-predictor-5c8bb955c5-mqnb5" event={"ID":"d0a4c463-e906-4147-b266-b3d9b2fd6c29","Type":"ContainerDied","Data":"d377bd8704559f64a7eeb934ac9b84c97c007b6ef6d07b3c409deb7d094a5f70"}
May 25 12:42:55.531725 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:55.531695    2549 scope.go:117] "RemoveContainer" containerID="298328194cbed0020fcfc4bd1bc4dfcace27fc89e79fb5abe29c99a22b4f01c8"
May 25 12:42:55.539794 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:55.539773    2549 scope.go:117] "RemoveContainer" containerID="0e589999a9931208d6901d9f586a935980ebc01727722bebd4a22a3a3f2cb81e"
May 25 12:42:55.540053 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:42:55.540036    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e589999a9931208d6901d9f586a935980ebc01727722bebd4a22a3a3f2cb81e\": container with ID starting with 0e589999a9931208d6901d9f586a935980ebc01727722bebd4a22a3a3f2cb81e not found: ID does not exist" containerID="0e589999a9931208d6901d9f586a935980ebc01727722bebd4a22a3a3f2cb81e"
May 25 12:42:55.540121 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:55.540074    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e589999a9931208d6901d9f586a935980ebc01727722bebd4a22a3a3f2cb81e"} err="failed to get container status \"0e589999a9931208d6901d9f586a935980ebc01727722bebd4a22a3a3f2cb81e\": rpc error: code = NotFound desc = could not find container \"0e589999a9931208d6901d9f586a935980ebc01727722bebd4a22a3a3f2cb81e\": container with ID starting with 0e589999a9931208d6901d9f586a935980ebc01727722bebd4a22a3a3f2cb81e not found: ID does not exist"
May 25 12:42:55.540121 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:55.540091    2549 scope.go:117] "RemoveContainer" containerID="298328194cbed0020fcfc4bd1bc4dfcace27fc89e79fb5abe29c99a22b4f01c8"
May 25 12:42:55.540333 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:42:55.540312    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"298328194cbed0020fcfc4bd1bc4dfcace27fc89e79fb5abe29c99a22b4f01c8\": container with ID starting with 298328194cbed0020fcfc4bd1bc4dfcace27fc89e79fb5abe29c99a22b4f01c8 not found: ID does not exist" containerID="298328194cbed0020fcfc4bd1bc4dfcace27fc89e79fb5abe29c99a22b4f01c8"
May 25 12:42:55.540401 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:55.540344    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"298328194cbed0020fcfc4bd1bc4dfcace27fc89e79fb5abe29c99a22b4f01c8"} err="failed to get container status \"298328194cbed0020fcfc4bd1bc4dfcace27fc89e79fb5abe29c99a22b4f01c8\": rpc error: code = NotFound desc = could not find container \"298328194cbed0020fcfc4bd1bc4dfcace27fc89e79fb5abe29c99a22b4f01c8\": container with ID starting with 298328194cbed0020fcfc4bd1bc4dfcace27fc89e79fb5abe29c99a22b4f01c8 not found: ID does not exist"
May 25 12:42:55.540401 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:55.540366    2549 scope.go:117] "RemoveContainer" containerID="34dca45b8b17a4dbf5731d8a197ac7f86e9338dcc48dac93149c4a73cdc4f4c3"
May 25 12:42:55.550930 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:55.550539    2549 scope.go:117] "RemoveContainer" containerID="f28b53db05c8eb0d7ccd2e1105707864e565f141a219a60d1d01e58e48298b4d"
May 25 12:42:55.553173 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:55.553149    2549 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-4d75e-predictor-5c8bb955c5-mqnb5"]
May 25 12:42:55.557295 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:55.557275    2549 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-4d75e-predictor-5c8bb955c5-mqnb5"]
May 25 12:42:55.559921 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:55.559906    2549 scope.go:117] "RemoveContainer" containerID="34dca45b8b17a4dbf5731d8a197ac7f86e9338dcc48dac93149c4a73cdc4f4c3"
May 25 12:42:55.560220 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:42:55.560188    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34dca45b8b17a4dbf5731d8a197ac7f86e9338dcc48dac93149c4a73cdc4f4c3\": container with ID starting with 34dca45b8b17a4dbf5731d8a197ac7f86e9338dcc48dac93149c4a73cdc4f4c3 not found: ID does not exist" containerID="34dca45b8b17a4dbf5731d8a197ac7f86e9338dcc48dac93149c4a73cdc4f4c3"
May 25 12:42:55.560276 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:55.560231    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34dca45b8b17a4dbf5731d8a197ac7f86e9338dcc48dac93149c4a73cdc4f4c3"} err="failed to get container status \"34dca45b8b17a4dbf5731d8a197ac7f86e9338dcc48dac93149c4a73cdc4f4c3\": rpc error: code = NotFound desc = could not find container \"34dca45b8b17a4dbf5731d8a197ac7f86e9338dcc48dac93149c4a73cdc4f4c3\": container with ID starting with 34dca45b8b17a4dbf5731d8a197ac7f86e9338dcc48dac93149c4a73cdc4f4c3 not found: ID does not exist"
May 25 12:42:55.560276 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:55.560249    2549 scope.go:117] "RemoveContainer" containerID="f28b53db05c8eb0d7ccd2e1105707864e565f141a219a60d1d01e58e48298b4d"
May 25 12:42:55.560468 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:42:55.560454    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f28b53db05c8eb0d7ccd2e1105707864e565f141a219a60d1d01e58e48298b4d\": container with ID starting with f28b53db05c8eb0d7ccd2e1105707864e565f141a219a60d1d01e58e48298b4d not found: ID does not exist" containerID="f28b53db05c8eb0d7ccd2e1105707864e565f141a219a60d1d01e58e48298b4d"
May 25 12:42:55.560506 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:55.560472    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f28b53db05c8eb0d7ccd2e1105707864e565f141a219a60d1d01e58e48298b4d"} err="failed to get container status \"f28b53db05c8eb0d7ccd2e1105707864e565f141a219a60d1d01e58e48298b4d\": rpc error: code = NotFound desc = could not find container \"f28b53db05c8eb0d7ccd2e1105707864e565f141a219a60d1d01e58e48298b4d\": container with ID starting with f28b53db05c8eb0d7ccd2e1105707864e565f141a219a60d1d01e58e48298b4d not found: ID does not exist"
May 25 12:42:55.569736 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:55.569711    2549 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-4d75e-predictor-98c496fb-m8ddw"]
May 25 12:42:55.574146 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:55.574125    2549 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-4d75e-predictor-98c496fb-m8ddw"]
May 25 12:42:56.848888 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:56.848845    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b5232f59-357c-4a8a-a062-13d1161f8a81" path="/var/lib/kubelet/pods/b5232f59-357c-4a8a-a062-13d1161f8a81/volumes"
May 25 12:42:56.849496 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:56.849474    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d0a4c463-e906-4147-b266-b3d9b2fd6c29" path="/var/lib/kubelet/pods/d0a4c463-e906-4147-b266-b3d9b2fd6c29/volumes"
May 25 12:42:59.396714 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:59.396670    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-fd4b4-predictor-7694f9fcbf-n88zh" podUID="4ab201cd-f34e-4d4c-aa98-179dee7a9ede" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
May 25 12:42:59.397115 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:59.396869    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-fd4b4-predictor-59d7bccbf8-pl6qv" podUID="90636a58-53a5-4c10-864f-cdf6647bd677" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
May 25 12:42:59.521002 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:59.520967    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-09598-predictor-7c4d94fb5f-hqnbq"
May 25 12:42:59.521178 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:59.521164    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-09598-predictor-587794bf84-zl6dc"
May 25 12:42:59.521391 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:59.521364    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-09598-predictor-7c4d94fb5f-hqnbq" podUID="017fafb4-c256-47d2-b271-a6c2feff2636" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
May 25 12:42:59.521670 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:42:59.521651    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-09598-predictor-587794bf84-zl6dc" podUID="7b0bcced-f9ba-4d60-a892-a604b11c68af" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
May 25 12:43:09.398037 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:43:09.397961    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-fd4b4-predictor-7694f9fcbf-n88zh"
May 25 12:43:09.398405 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:43:09.398151    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-fd4b4-predictor-59d7bccbf8-pl6qv"
May 25 12:43:09.521630 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:43:09.521581    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-09598-predictor-7c4d94fb5f-hqnbq" podUID="017fafb4-c256-47d2-b271-a6c2feff2636" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
May 25 12:43:09.521807 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:43:09.521778    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-09598-predictor-587794bf84-zl6dc" podUID="7b0bcced-f9ba-4d60-a892-a604b11c68af" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
May 25 12:43:19.521391 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:43:19.521351    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-09598-predictor-7c4d94fb5f-hqnbq" podUID="017fafb4-c256-47d2-b271-a6c2feff2636" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
May 25 12:43:19.521778 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:43:19.521757    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-09598-predictor-587794bf84-zl6dc" podUID="7b0bcced-f9ba-4d60-a892-a604b11c68af" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
May 25 12:43:29.521371 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:43:29.521332    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-09598-predictor-7c4d94fb5f-hqnbq" podUID="017fafb4-c256-47d2-b271-a6c2feff2636" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
May 25 12:43:29.521891 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:43:29.521743    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-09598-predictor-587794bf84-zl6dc" podUID="7b0bcced-f9ba-4d60-a892-a604b11c68af" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
May 25 12:43:31.507092 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:43:31.507055    2549 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-fd4b4-predictor-7694f9fcbf-n88zh"]
May 25 12:43:31.507545 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:43:31.507454    2549 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-fd4b4-predictor-7694f9fcbf-n88zh" podUID="4ab201cd-f34e-4d4c-aa98-179dee7a9ede" containerName="kserve-container" containerID="cri-o://90a698ddb71598c0742e56099de7a64d97eedf2103ce534cff37a4de4b7ecdf3" gracePeriod=30
May 25 12:43:31.507615 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:43:31.507510    2549 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-fd4b4-predictor-7694f9fcbf-n88zh" podUID="4ab201cd-f34e-4d4c-aa98-179dee7a9ede" containerName="kube-rbac-proxy" containerID="cri-o://1381cf1891a17ff8634ea9fadd2649a636c0c9cdb9b2f6073ce2385969fceeeb" gracePeriod=30
May 25 12:43:31.562218 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:43:31.561716    2549 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-fd4b4-predictor-59d7bccbf8-pl6qv"]
May 25 12:43:31.562218 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:43:31.562073    2549 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-fd4b4-predictor-59d7bccbf8-pl6qv" podUID="90636a58-53a5-4c10-864f-cdf6647bd677" containerName="kserve-container" containerID="cri-o://c0d259710e691108fc4e62a1f77b759332965e59359d57ef9a0334c7941c916a" gracePeriod=30
May 25 12:43:31.562476 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:43:31.562428    2549 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-fd4b4-predictor-59d7bccbf8-pl6qv" podUID="90636a58-53a5-4c10-864f-cdf6647bd677" containerName="kube-rbac-proxy" containerID="cri-o://adbe33e7faa9651120a9b1d2516e315b69fdb91e5837809a23b9a6c9fe64ff9e" gracePeriod=30
May 25 12:43:31.577368 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:43:31.577340    2549 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-8902c-predictor-b6f848b6b-rmjnr"]
May 25 12:43:31.577801 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:43:31.577784    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0a4c463-e906-4147-b266-b3d9b2fd6c29" containerName="kube-rbac-proxy"
May 25 12:43:31.577881 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:43:31.577804    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0a4c463-e906-4147-b266-b3d9b2fd6c29" containerName="kube-rbac-proxy"
May 25 12:43:31.577881 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:43:31.577822    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0a4c463-e906-4147-b266-b3d9b2fd6c29" containerName="kserve-container"
May 25 12:43:31.577881 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:43:31.577830    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0a4c463-e906-4147-b266-b3d9b2fd6c29" containerName="kserve-container"
May 25 12:43:31.577881 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:43:31.577847    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5232f59-357c-4a8a-a062-13d1161f8a81" containerName="kserve-container"
May 25 12:43:31.577881 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:43:31.577855    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5232f59-357c-4a8a-a062-13d1161f8a81" containerName="kserve-container"
May 25 12:43:31.577881 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:43:31.577877    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5232f59-357c-4a8a-a062-13d1161f8a81" containerName="kube-rbac-proxy"
May 25 12:43:31.578177 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:43:31.577885    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5232f59-357c-4a8a-a062-13d1161f8a81" containerName="kube-rbac-proxy"
May 25 12:43:31.578177 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:43:31.577981    2549 memory_manager.go:356] "RemoveStaleState removing state" podUID="d0a4c463-e906-4147-b266-b3d9b2fd6c29" containerName="kube-rbac-proxy"
May 25 12:43:31.578177 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:43:31.577995    2549 memory_manager.go:356] "RemoveStaleState removing state" podUID="b5232f59-357c-4a8a-a062-13d1161f8a81" containerName="kserve-container"
May 25 12:43:31.578177 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:43:31.578008    2549 memory_manager.go:356] "RemoveStaleState removing state" podUID="d0a4c463-e906-4147-b266-b3d9b2fd6c29" containerName="kserve-container"
May 25 12:43:31.578177 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:43:31.578021    2549 memory_manager.go:356] "RemoveStaleState removing state" podUID="b5232f59-357c-4a8a-a062-13d1161f8a81" containerName="kube-rbac-proxy"
May 25 12:43:31.582726 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:43:31.582709    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-8902c-predictor-b6f848b6b-rmjnr"
May 25 12:43:31.586077 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:43:31.586057    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-8902c-predictor-serving-cert\""
May 25 12:43:31.586722 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:43:31.586702    2549 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-8902c-kube-rbac-proxy-sar-config\""
May 25 12:43:31.595041 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:43:31.595014    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-8902c-predictor-b6f848b6b-rmjnr"]
May 25 12:43:31.669570 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:43:31.669514    2549 generic.go:358] "Generic (PLEG): container finished" podID="4ab201cd-f34e-4d4c-aa98-179dee7a9ede" containerID="1381cf1891a17ff8634ea9fadd2649a636c0c9cdb9b2f6073ce2385969fceeeb" exitCode=2
May 25 12:43:31.669748 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:43:31.669584    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-fd4b4-predictor-7694f9fcbf-n88zh" event={"ID":"4ab201cd-f34e-4d4c-aa98-179dee7a9ede","Type":"ContainerDied","Data":"1381cf1891a17ff8634ea9fadd2649a636c0c9cdb9b2f6073ce2385969fceeeb"}
May 25 12:43:31.676765 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:43:31.676735    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbj52\" (UniqueName: \"kubernetes.io/projected/7a6774c9-5000-4f2c-9ece-20506dde280f-kube-api-access-mbj52\") pod \"success-200-isvc-8902c-predictor-b6f848b6b-rmjnr\" (UID: \"7a6774c9-5000-4f2c-9ece-20506dde280f\") " pod="kserve-ci-e2e-test/success-200-isvc-8902c-predictor-b6f848b6b-rmjnr"
May 25 12:43:31.676886 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:43:31.676810    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7a6774c9-5000-4f2c-9ece-20506dde280f-proxy-tls\") pod \"success-200-isvc-8902c-predictor-b6f848b6b-rmjnr\" (UID: \"7a6774c9-5000-4f2c-9ece-20506dde280f\") " pod="kserve-ci-e2e-test/success-200-isvc-8902c-predictor-b6f848b6b-rmjnr"
May 25 12:43:31.676949 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:43:31.676895    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"success-200-isvc-8902c-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7a6774c9-5000-4f2c-9ece-20506dde280f-success-200-isvc-8902c-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-8902c-predictor-b6f848b6b-rmjnr\" (UID: \"7a6774c9-5000-4f2c-9ece-20506dde280f\") " pod="kserve-ci-e2e-test/success-200-isvc-8902c-predictor-b6f848b6b-rmjnr"
May 25 12:43:31.710728 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:43:31.710687    2549 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-8902c-predictor-86c999b5df-f6nxh"]
May 25 12:43:31.714568 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:43:31.714542    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-8902c-predictor-86c999b5df-f6nxh"
May 25 12:43:31.717541 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:43:31.717516    2549 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-8902c-kube-rbac-proxy-sar-config\""
May 25 12:43:31.717668 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:43:31.717516    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-8902c-predictor-serving-cert\""
May 25 12:43:31.722952 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:43:31.722606    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-8902c-predictor-86c999b5df-f6nxh"]
May 25 12:43:31.778327 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:43:31.778230    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76jb9\" (UniqueName: \"kubernetes.io/projected/8caf44b4-889f-4d31-b6d0-1ae3beafd3b7-kube-api-access-76jb9\") pod \"error-404-isvc-8902c-predictor-86c999b5df-f6nxh\" (UID: \"8caf44b4-889f-4d31-b6d0-1ae3beafd3b7\") " pod="kserve-ci-e2e-test/error-404-isvc-8902c-predictor-86c999b5df-f6nxh"
May 25 12:43:31.778327 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:43:31.778282    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-mbj52\" (UniqueName: \"kubernetes.io/projected/7a6774c9-5000-4f2c-9ece-20506dde280f-kube-api-access-mbj52\") pod \"success-200-isvc-8902c-predictor-b6f848b6b-rmjnr\" (UID: \"7a6774c9-5000-4f2c-9ece-20506dde280f\") " pod="kserve-ci-e2e-test/success-200-isvc-8902c-predictor-b6f848b6b-rmjnr"
May 25 12:43:31.778554 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:43:31.778367    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8caf44b4-889f-4d31-b6d0-1ae3beafd3b7-proxy-tls\") pod \"error-404-isvc-8902c-predictor-86c999b5df-f6nxh\" (UID: \"8caf44b4-889f-4d31-b6d0-1ae3beafd3b7\") " pod="kserve-ci-e2e-test/error-404-isvc-8902c-predictor-86c999b5df-f6nxh"
May 25 12:43:31.778554 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:43:31.778410    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7a6774c9-5000-4f2c-9ece-20506dde280f-proxy-tls\") pod \"success-200-isvc-8902c-predictor-b6f848b6b-rmjnr\" (UID: \"7a6774c9-5000-4f2c-9ece-20506dde280f\") " pod="kserve-ci-e2e-test/success-200-isvc-8902c-predictor-b6f848b6b-rmjnr"
May 25 12:43:31.778554 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:43:31.778447    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"success-200-isvc-8902c-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7a6774c9-5000-4f2c-9ece-20506dde280f-success-200-isvc-8902c-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-8902c-predictor-b6f848b6b-rmjnr\" (UID: \"7a6774c9-5000-4f2c-9ece-20506dde280f\") " pod="kserve-ci-e2e-test/success-200-isvc-8902c-predictor-b6f848b6b-rmjnr"
May 25 12:43:31.778554 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:43:31.778486    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"error-404-isvc-8902c-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8caf44b4-889f-4d31-b6d0-1ae3beafd3b7-error-404-isvc-8902c-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-8902c-predictor-86c999b5df-f6nxh\" (UID: \"8caf44b4-889f-4d31-b6d0-1ae3beafd3b7\") " pod="kserve-ci-e2e-test/error-404-isvc-8902c-predictor-86c999b5df-f6nxh"
May 25 12:43:31.778554 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:43:31.778545    2549 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-8902c-predictor-serving-cert: secret "success-200-isvc-8902c-predictor-serving-cert" not found
May 25 12:43:31.778803 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:43:31.778616    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7a6774c9-5000-4f2c-9ece-20506dde280f-proxy-tls podName:7a6774c9-5000-4f2c-9ece-20506dde280f nodeName:}" failed. No retries permitted until 2026-05-25 12:43:32.278594196 +0000 UTC m=+1815.994409474 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/7a6774c9-5000-4f2c-9ece-20506dde280f-proxy-tls") pod "success-200-isvc-8902c-predictor-b6f848b6b-rmjnr" (UID: "7a6774c9-5000-4f2c-9ece-20506dde280f") : secret "success-200-isvc-8902c-predictor-serving-cert" not found
May 25 12:43:31.779224 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:43:31.779184    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"success-200-isvc-8902c-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7a6774c9-5000-4f2c-9ece-20506dde280f-success-200-isvc-8902c-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-8902c-predictor-b6f848b6b-rmjnr\" (UID: \"7a6774c9-5000-4f2c-9ece-20506dde280f\") " pod="kserve-ci-e2e-test/success-200-isvc-8902c-predictor-b6f848b6b-rmjnr"
May 25 12:43:31.793686 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:43:31.793657    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbj52\" (UniqueName: \"kubernetes.io/projected/7a6774c9-5000-4f2c-9ece-20506dde280f-kube-api-access-mbj52\") pod \"success-200-isvc-8902c-predictor-b6f848b6b-rmjnr\" (UID: \"7a6774c9-5000-4f2c-9ece-20506dde280f\") " pod="kserve-ci-e2e-test/success-200-isvc-8902c-predictor-b6f848b6b-rmjnr"
May 25 12:43:31.879703 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:43:31.879670    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"error-404-isvc-8902c-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8caf44b4-889f-4d31-b6d0-1ae3beafd3b7-error-404-isvc-8902c-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-8902c-predictor-86c999b5df-f6nxh\" (UID: \"8caf44b4-889f-4d31-b6d0-1ae3beafd3b7\") " pod="kserve-ci-e2e-test/error-404-isvc-8902c-predictor-86c999b5df-f6nxh"
May 25 12:43:31.879876 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:43:31.879773    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-76jb9\" (UniqueName: \"kubernetes.io/projected/8caf44b4-889f-4d31-b6d0-1ae3beafd3b7-kube-api-access-76jb9\") pod \"error-404-isvc-8902c-predictor-86c999b5df-f6nxh\" (UID: \"8caf44b4-889f-4d31-b6d0-1ae3beafd3b7\") " pod="kserve-ci-e2e-test/error-404-isvc-8902c-predictor-86c999b5df-f6nxh"
May 25 12:43:31.879876 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:43:31.879808    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8caf44b4-889f-4d31-b6d0-1ae3beafd3b7-proxy-tls\") pod \"error-404-isvc-8902c-predictor-86c999b5df-f6nxh\" (UID: \"8caf44b4-889f-4d31-b6d0-1ae3beafd3b7\") " pod="kserve-ci-e2e-test/error-404-isvc-8902c-predictor-86c999b5df-f6nxh"
May 25 12:43:31.879971 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:43:31.879925    2549 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-8902c-predictor-serving-cert: secret "error-404-isvc-8902c-predictor-serving-cert" not found
May 25 12:43:31.880017 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:43:31.879982    2549 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8caf44b4-889f-4d31-b6d0-1ae3beafd3b7-proxy-tls podName:8caf44b4-889f-4d31-b6d0-1ae3beafd3b7 nodeName:}" failed. No retries permitted until 2026-05-25 12:43:32.37996432 +0000 UTC m=+1816.095779585 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/8caf44b4-889f-4d31-b6d0-1ae3beafd3b7-proxy-tls") pod "error-404-isvc-8902c-predictor-86c999b5df-f6nxh" (UID: "8caf44b4-889f-4d31-b6d0-1ae3beafd3b7") : secret "error-404-isvc-8902c-predictor-serving-cert" not found
May 25 12:43:31.880325 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:43:31.880304    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"error-404-isvc-8902c-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8caf44b4-889f-4d31-b6d0-1ae3beafd3b7-error-404-isvc-8902c-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-8902c-predictor-86c999b5df-f6nxh\" (UID: \"8caf44b4-889f-4d31-b6d0-1ae3beafd3b7\") " pod="kserve-ci-e2e-test/error-404-isvc-8902c-predictor-86c999b5df-f6nxh"
May 25 12:43:31.893887 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:43:31.893853    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-76jb9\" (UniqueName: \"kubernetes.io/projected/8caf44b4-889f-4d31-b6d0-1ae3beafd3b7-kube-api-access-76jb9\") pod \"error-404-isvc-8902c-predictor-86c999b5df-f6nxh\" (UID: \"8caf44b4-889f-4d31-b6d0-1ae3beafd3b7\") " pod="kserve-ci-e2e-test/error-404-isvc-8902c-predictor-86c999b5df-f6nxh"
May 25 12:43:32.284830 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:43:32.284778    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7a6774c9-5000-4f2c-9ece-20506dde280f-proxy-tls\") pod \"success-200-isvc-8902c-predictor-b6f848b6b-rmjnr\" (UID: \"7a6774c9-5000-4f2c-9ece-20506dde280f\") " pod="kserve-ci-e2e-test/success-200-isvc-8902c-predictor-b6f848b6b-rmjnr"
May 25 12:43:32.287601 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:43:32.287567    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7a6774c9-5000-4f2c-9ece-20506dde280f-proxy-tls\") pod \"success-200-isvc-8902c-predictor-b6f848b6b-rmjnr\" (UID: \"7a6774c9-5000-4f2c-9ece-20506dde280f\") " pod="kserve-ci-e2e-test/success-200-isvc-8902c-predictor-b6f848b6b-rmjnr"
May 25 12:43:32.386110 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:43:32.386051    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8caf44b4-889f-4d31-b6d0-1ae3beafd3b7-proxy-tls\") pod \"error-404-isvc-8902c-predictor-86c999b5df-f6nxh\" (UID: \"8caf44b4-889f-4d31-b6d0-1ae3beafd3b7\") " pod="kserve-ci-e2e-test/error-404-isvc-8902c-predictor-86c999b5df-f6nxh"
May 25 12:43:32.389415 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:43:32.389381    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8caf44b4-889f-4d31-b6d0-1ae3beafd3b7-proxy-tls\") pod \"error-404-isvc-8902c-predictor-86c999b5df-f6nxh\" (UID: \"8caf44b4-889f-4d31-b6d0-1ae3beafd3b7\") " pod="kserve-ci-e2e-test/error-404-isvc-8902c-predictor-86c999b5df-f6nxh"
May 25 12:43:32.498624 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:43:32.498583    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-8902c-predictor-b6f848b6b-rmjnr"
May 25 12:43:32.627346 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:43:32.627312    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-8902c-predictor-86c999b5df-f6nxh"
May 25 12:43:32.640220 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:43:32.640165    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-8902c-predictor-b6f848b6b-rmjnr"]
May 25 12:43:32.643347 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:43:32.643315    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a6774c9_5000_4f2c_9ece_20506dde280f.slice/crio-73e4e0e5bc0f2beb62f6e2330be8303711df04675f0d8eeb9b21bd39498923c8 WatchSource:0}: Error finding container 73e4e0e5bc0f2beb62f6e2330be8303711df04675f0d8eeb9b21bd39498923c8: Status 404 returned error can't find the container with id 73e4e0e5bc0f2beb62f6e2330be8303711df04675f0d8eeb9b21bd39498923c8
May 25 12:43:32.677127 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:43:32.677095    2549 generic.go:358] "Generic (PLEG): container finished" podID="90636a58-53a5-4c10-864f-cdf6647bd677" containerID="adbe33e7faa9651120a9b1d2516e315b69fdb91e5837809a23b9a6c9fe64ff9e" exitCode=2
May 25 12:43:32.677300 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:43:32.677160    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-fd4b4-predictor-59d7bccbf8-pl6qv" event={"ID":"90636a58-53a5-4c10-864f-cdf6647bd677","Type":"ContainerDied","Data":"adbe33e7faa9651120a9b1d2516e315b69fdb91e5837809a23b9a6c9fe64ff9e"}
May 25 12:43:32.678388 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:43:32.678365    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-8902c-predictor-b6f848b6b-rmjnr" event={"ID":"7a6774c9-5000-4f2c-9ece-20506dde280f","Type":"ContainerStarted","Data":"73e4e0e5bc0f2beb62f6e2330be8303711df04675f0d8eeb9b21bd39498923c8"}
May 25 12:43:32.776644 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:43:32.776609    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-8902c-predictor-86c999b5df-f6nxh"]
May 25 12:43:32.779433 ip-10-0-142-53 kubenswrapper[2549]: W0525 12:43:32.779403    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8caf44b4_889f_4d31_b6d0_1ae3beafd3b7.slice/crio-99002cb6140aaf943f86cb69222db77d148ef00968857a948b0f9c66aa4f0cfd WatchSource:0}: Error finding container 99002cb6140aaf943f86cb69222db77d148ef00968857a948b0f9c66aa4f0cfd: Status 404 returned error can't find the container with id 99002cb6140aaf943f86cb69222db77d148ef00968857a948b0f9c66aa4f0cfd
May 25 12:43:33.685005 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:43:33.684945    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-8902c-predictor-86c999b5df-f6nxh" event={"ID":"8caf44b4-889f-4d31-b6d0-1ae3beafd3b7","Type":"ContainerStarted","Data":"7b275ab500091e9061847ad286cd666f0018cabf1f481d42cdfd4e2c285fefff"}
May 25 12:43:33.685005 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:43:33.685005    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-8902c-predictor-86c999b5df-f6nxh" event={"ID":"8caf44b4-889f-4d31-b6d0-1ae3beafd3b7","Type":"ContainerStarted","Data":"1ab865509867941b74155b31eabd03e9e7de7ebf7afe15689a97e227ab38e0a8"}
May 25 12:43:33.685565 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:43:33.685020    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-8902c-predictor-86c999b5df-f6nxh" event={"ID":"8caf44b4-889f-4d31-b6d0-1ae3beafd3b7","Type":"ContainerStarted","Data":"99002cb6140aaf943f86cb69222db77d148ef00968857a948b0f9c66aa4f0cfd"}
May 25 12:43:33.686150 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:43:33.686085    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-8902c-predictor-86c999b5df-f6nxh"
May 25 12:43:33.686150 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:43:33.686121    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-8902c-predictor-86c999b5df-f6nxh"
May 25 12:43:33.687881 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:43:33.687582    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-8902c-predictor-86c999b5df-f6nxh" podUID="8caf44b4-889f-4d31-b6d0-1ae3beafd3b7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.53:8080: connect: connection refused"
May 25 12:43:33.689822 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:43:33.689796    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-8902c-predictor-b6f848b6b-rmjnr" event={"ID":"7a6774c9-5000-4f2c-9ece-20506dde280f","Type":"ContainerStarted","Data":"6649543b215dec0c04127283177ec45ce668cd9db73c0649c6c82b691af03639"}
May 25 12:43:33.689974 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:43:33.689957    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-8902c-predictor-b6f848b6b-rmjnr" event={"ID":"7a6774c9-5000-4f2c-9ece-20506dde280f","Type":"ContainerStarted","Data":"0f19969d428fcd94c4ad0437ca124a47c5c85204d16c4fc3105bc526043d5da4"}
May 25 12:43:33.690431 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:43:33.690407    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-8902c-predictor-b6f848b6b-rmjnr"
May 25 12:43:33.691145 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:43:33.691125    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-8902c-predictor-b6f848b6b-rmjnr"
May 25 12:43:33.691780 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:43:33.691751    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-8902c-predictor-b6f848b6b-rmjnr" podUID="7a6774c9-5000-4f2c-9ece-20506dde280f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
May 25 12:43:33.708561 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:43:33.708522    2549 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-8902c-predictor-86c999b5df-f6nxh" podStartSLOduration=2.7085104810000002 podStartE2EDuration="2.708510481s" podCreationTimestamp="2026-05-25 12:43:31 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 12:43:33.705888572 +0000 UTC m=+1817.421703860" watchObservedRunningTime="2026-05-25 12:43:33.708510481 +0000 UTC m=+1817.424325766"
May 25 12:43:33.724713 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:43:33.724671    2549 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-8902c-predictor-b6f848b6b-rmjnr" podStartSLOduration=2.724660075 podStartE2EDuration="2.724660075s" podCreationTimestamp="2026-05-25 12:43:31 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 12:43:33.723076519 +0000 UTC m=+1817.438891806" watchObservedRunningTime="2026-05-25 12:43:33.724660075 +0000 UTC m=+1817.440475360"
May 25 12:43:34.392026 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:43:34.391738    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-fd4b4-predictor-7694f9fcbf-n88zh" podUID="4ab201cd-f34e-4d4c-aa98-179dee7a9ede" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.49:8643/healthz\": dial tcp 10.134.0.49:8643: connect: connection refused"
May 25 12:43:34.392026 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:43:34.391738    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-fd4b4-predictor-59d7bccbf8-pl6qv" podUID="90636a58-53a5-4c10-864f-cdf6647bd677" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.48:8643/healthz\": dial tcp 10.134.0.48:8643: connect: connection refused"
May 25 12:43:34.695453 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:43:34.695362    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-8902c-predictor-86c999b5df-f6nxh" podUID="8caf44b4-889f-4d31-b6d0-1ae3beafd3b7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.53:8080: connect: connection refused"
May 25 12:43:34.695822 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:43:34.695464    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-8902c-predictor-b6f848b6b-rmjnr" podUID="7a6774c9-5000-4f2c-9ece-20506dde280f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
May 25 12:43:35.550267 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:43:35.550239    2549 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-fd4b4-predictor-7694f9fcbf-n88zh"
May 25 12:43:35.618827 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:43:35.618799    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4ab201cd-f34e-4d4c-aa98-179dee7a9ede-proxy-tls\") pod \"4ab201cd-f34e-4d4c-aa98-179dee7a9ede\" (UID: \"4ab201cd-f34e-4d4c-aa98-179dee7a9ede\") "
May 25 12:43:35.618984 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:43:35.618876    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jwtxw\" (UniqueName: \"kubernetes.io/projected/4ab201cd-f34e-4d4c-aa98-179dee7a9ede-kube-api-access-jwtxw\") pod \"4ab201cd-f34e-4d4c-aa98-179dee7a9ede\" (UID: \"4ab201cd-f34e-4d4c-aa98-179dee7a9ede\") "
May 25 12:43:35.618984 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:43:35.618931    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"error-404-isvc-fd4b4-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4ab201cd-f34e-4d4c-aa98-179dee7a9ede-error-404-isvc-fd4b4-kube-rbac-proxy-sar-config\") pod \"4ab201cd-f34e-4d4c-aa98-179dee7a9ede\" (UID: \"4ab201cd-f34e-4d4c-aa98-179dee7a9ede\") "
May 25 12:43:35.619317 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:43:35.619293    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ab201cd-f34e-4d4c-aa98-179dee7a9ede-error-404-isvc-fd4b4-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "error-404-isvc-fd4b4-kube-rbac-proxy-sar-config") pod "4ab201cd-f34e-4d4c-aa98-179dee7a9ede" (UID: "4ab201cd-f34e-4d4c-aa98-179dee7a9ede"). InnerVolumeSpecName "error-404-isvc-fd4b4-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 12:43:35.620936 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:43:35.620918    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ab201cd-f34e-4d4c-aa98-179dee7a9ede-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4ab201cd-f34e-4d4c-aa98-179dee7a9ede" (UID: "4ab201cd-f34e-4d4c-aa98-179dee7a9ede"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 12:43:35.620936 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:43:35.620927    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ab201cd-f34e-4d4c-aa98-179dee7a9ede-kube-api-access-jwtxw" (OuterVolumeSpecName: "kube-api-access-jwtxw") pod "4ab201cd-f34e-4d4c-aa98-179dee7a9ede" (UID: "4ab201cd-f34e-4d4c-aa98-179dee7a9ede"). InnerVolumeSpecName "kube-api-access-jwtxw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 12:43:35.700739 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:43:35.700706    2549 generic.go:358] "Generic (PLEG): container finished" podID="90636a58-53a5-4c10-864f-cdf6647bd677" containerID="c0d259710e691108fc4e62a1f77b759332965e59359d57ef9a0334c7941c916a" exitCode=0
May 25 12:43:35.701130 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:43:35.700774    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-fd4b4-predictor-59d7bccbf8-pl6qv" event={"ID":"90636a58-53a5-4c10-864f-cdf6647bd677","Type":"ContainerDied","Data":"c0d259710e691108fc4e62a1f77b759332965e59359d57ef9a0334c7941c916a"}
May 25 12:43:35.702327 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:43:35.702297    2549 generic.go:358] "Generic (PLEG): container finished" podID="4ab201cd-f34e-4d4c-aa98-179dee7a9ede" containerID="90a698ddb71598c0742e56099de7a64d97eedf2103ce534cff37a4de4b7ecdf3" exitCode=0
May 25 12:43:35.702471 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:43:35.702382    2549 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-fd4b4-predictor-7694f9fcbf-n88zh"
May 25 12:43:35.702471 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:43:35.702394    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-fd4b4-predictor-7694f9fcbf-n88zh" event={"ID":"4ab201cd-f34e-4d4c-aa98-179dee7a9ede","Type":"ContainerDied","Data":"90a698ddb71598c0742e56099de7a64d97eedf2103ce534cff37a4de4b7ecdf3"}
May 25 12:43:35.702471 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:43:35.702441    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-fd4b4-predictor-7694f9fcbf-n88zh" event={"ID":"4ab201cd-f34e-4d4c-aa98-179dee7a9ede","Type":"ContainerDied","Data":"c6561f98e36992f43cc6e864e3f68c6f26bd2cbfe06f0094784a81d0a9eb0bfc"}
May 25 12:43:35.702471 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:43:35.702463    2549 scope.go:117] "RemoveContainer" containerID="1381cf1891a17ff8634ea9fadd2649a636c0c9cdb9b2f6073ce2385969fceeeb"
May 25 12:43:35.702863 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:43:35.702823    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-8902c-predictor-86c999b5df-f6nxh" podUID="8caf44b4-889f-4d31-b6d0-1ae3beafd3b7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.53:8080: connect: connection refused"
May 25 12:43:35.703366 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:43:35.703272    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-8902c-predictor-b6f848b6b-rmjnr" podUID="7a6774c9-5000-4f2c-9ece-20506dde280f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
May 25 12:43:35.721085 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:43:35.721051    2549 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4ab201cd-f34e-4d4c-aa98-179dee7a9ede-proxy-tls\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
May 25 12:43:35.721085 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:43:35.721082    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jwtxw\" (UniqueName: \"kubernetes.io/projected/4ab201cd-f34e-4d4c-aa98-179dee7a9ede-kube-api-access-jwtxw\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
May 25 12:43:35.721085 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:43:35.721093    2549 reconciler_common.go:299] "Volume detached for volume \"error-404-isvc-fd4b4-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4ab201cd-f34e-4d4c-aa98-179dee7a9ede-error-404-isvc-fd4b4-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
May 25 12:43:35.722768 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:43:35.722744    2549 scope.go:117] "RemoveContainer" containerID="90a698ddb71598c0742e56099de7a64d97eedf2103ce534cff37a4de4b7ecdf3"
May 25 12:43:35.732414 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:43:35.732384    2549 scope.go:117] "RemoveContainer" containerID="1381cf1891a17ff8634ea9fadd2649a636c0c9cdb9b2f6073ce2385969fceeeb"
May 25 12:43:35.732790 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:43:35.732770    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1381cf1891a17ff8634ea9fadd2649a636c0c9cdb9b2f6073ce2385969fceeeb\": container with ID starting with 1381cf1891a17ff8634ea9fadd2649a636c0c9cdb9b2f6073ce2385969fceeeb not found: ID does not exist" containerID="1381cf1891a17ff8634ea9fadd2649a636c0c9cdb9b2f6073ce2385969fceeeb"
May 25 12:43:35.732837 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:43:35.732800    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1381cf1891a17ff8634ea9fadd2649a636c0c9cdb9b2f6073ce2385969fceeeb"} err="failed to get container status \"1381cf1891a17ff8634ea9fadd2649a636c0c9cdb9b2f6073ce2385969fceeeb\": rpc error: code = NotFound desc = could not find container \"1381cf1891a17ff8634ea9fadd2649a636c0c9cdb9b2f6073ce2385969fceeeb\": container with ID starting with 1381cf1891a17ff8634ea9fadd2649a636c0c9cdb9b2f6073ce2385969fceeeb not found: ID does not exist"
May 25 12:43:35.732837 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:43:35.732820    2549 scope.go:117] "RemoveContainer" containerID="90a698ddb71598c0742e56099de7a64d97eedf2103ce534cff37a4de4b7ecdf3"
May 25 12:43:35.733090 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:43:35.733072    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90a698ddb71598c0742e56099de7a64d97eedf2103ce534cff37a4de4b7ecdf3\": container with ID starting with 90a698ddb71598c0742e56099de7a64d97eedf2103ce534cff37a4de4b7ecdf3 not found: ID does not exist" containerID="90a698ddb71598c0742e56099de7a64d97eedf2103ce534cff37a4de4b7ecdf3"
May 25 12:43:35.733134 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:43:35.733098    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90a698ddb71598c0742e56099de7a64d97eedf2103ce534cff37a4de4b7ecdf3"} err="failed to get container status \"90a698ddb71598c0742e56099de7a64d97eedf2103ce534cff37a4de4b7ecdf3\": rpc error: code = NotFound desc = could not find container \"90a698ddb71598c0742e56099de7a64d97eedf2103ce534cff37a4de4b7ecdf3\": container with ID starting with 90a698ddb71598c0742e56099de7a64d97eedf2103ce534cff37a4de4b7ecdf3 not found: ID does not exist"
May 25 12:43:35.733171 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:43:35.733131    2549 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-fd4b4-predictor-7694f9fcbf-n88zh"]
May 25 12:43:35.740436 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:43:35.740410    2549 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-fd4b4-predictor-7694f9fcbf-n88zh"]
May 25 12:43:35.828058 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:43:35.828033    2549 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-fd4b4-predictor-59d7bccbf8-pl6qv"
May 25 12:43:35.923807 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:43:35.923713    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xnlz8\" (UniqueName: \"kubernetes.io/projected/90636a58-53a5-4c10-864f-cdf6647bd677-kube-api-access-xnlz8\") pod \"90636a58-53a5-4c10-864f-cdf6647bd677\" (UID: \"90636a58-53a5-4c10-864f-cdf6647bd677\") "
May 25 12:43:35.923807 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:43:35.923760    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/90636a58-53a5-4c10-864f-cdf6647bd677-proxy-tls\") pod \"90636a58-53a5-4c10-864f-cdf6647bd677\" (UID: \"90636a58-53a5-4c10-864f-cdf6647bd677\") "
May 25 12:43:35.923807 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:43:35.923795    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"success-200-isvc-fd4b4-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/90636a58-53a5-4c10-864f-cdf6647bd677-success-200-isvc-fd4b4-kube-rbac-proxy-sar-config\") pod \"90636a58-53a5-4c10-864f-cdf6647bd677\" (UID: \"90636a58-53a5-4c10-864f-cdf6647bd677\") "
May 25 12:43:35.924231 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:43:35.924185    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90636a58-53a5-4c10-864f-cdf6647bd677-success-200-isvc-fd4b4-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "success-200-isvc-fd4b4-kube-rbac-proxy-sar-config") pod "90636a58-53a5-4c10-864f-cdf6647bd677" (UID: "90636a58-53a5-4c10-864f-cdf6647bd677"). InnerVolumeSpecName "success-200-isvc-fd4b4-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 12:43:35.925849 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:43:35.925827    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90636a58-53a5-4c10-864f-cdf6647bd677-kube-api-access-xnlz8" (OuterVolumeSpecName: "kube-api-access-xnlz8") pod "90636a58-53a5-4c10-864f-cdf6647bd677" (UID: "90636a58-53a5-4c10-864f-cdf6647bd677"). InnerVolumeSpecName "kube-api-access-xnlz8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 12:43:35.925908 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:43:35.925845    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90636a58-53a5-4c10-864f-cdf6647bd677-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "90636a58-53a5-4c10-864f-cdf6647bd677" (UID: "90636a58-53a5-4c10-864f-cdf6647bd677"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 12:43:36.025252 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:43:36.025179    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xnlz8\" (UniqueName: \"kubernetes.io/projected/90636a58-53a5-4c10-864f-cdf6647bd677-kube-api-access-xnlz8\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
May 25 12:43:36.025252 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:43:36.025244    2549 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/90636a58-53a5-4c10-864f-cdf6647bd677-proxy-tls\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
May 25 12:43:36.025252 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:43:36.025257    2549 reconciler_common.go:299] "Volume detached for volume \"success-200-isvc-fd4b4-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/90636a58-53a5-4c10-864f-cdf6647bd677-success-200-isvc-fd4b4-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
May 25 12:43:36.708109 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:43:36.708018    2549 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-fd4b4-predictor-59d7bccbf8-pl6qv"
May 25 12:43:36.708109 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:43:36.708012    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-fd4b4-predictor-59d7bccbf8-pl6qv" event={"ID":"90636a58-53a5-4c10-864f-cdf6647bd677","Type":"ContainerDied","Data":"d2fe48e34e138cda82f963808119b3328d002457d20cd1379d35013976a49241"}
May 25 12:43:36.708555 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:43:36.708142    2549 scope.go:117] "RemoveContainer" containerID="adbe33e7faa9651120a9b1d2516e315b69fdb91e5837809a23b9a6c9fe64ff9e"
May 25 12:43:36.717555 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:43:36.717536    2549 scope.go:117] "RemoveContainer" containerID="c0d259710e691108fc4e62a1f77b759332965e59359d57ef9a0334c7941c916a"
May 25 12:43:36.732458 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:43:36.732426    2549 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-fd4b4-predictor-59d7bccbf8-pl6qv"]
May 25 12:43:36.736660 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:43:36.736629    2549 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-fd4b4-predictor-59d7bccbf8-pl6qv"]
May 25 12:43:36.848717 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:43:36.848685    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4ab201cd-f34e-4d4c-aa98-179dee7a9ede" path="/var/lib/kubelet/pods/4ab201cd-f34e-4d4c-aa98-179dee7a9ede/volumes"
May 25 12:43:36.849131 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:43:36.849119    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="90636a58-53a5-4c10-864f-cdf6647bd677" path="/var/lib/kubelet/pods/90636a58-53a5-4c10-864f-cdf6647bd677/volumes"
May 25 12:43:39.521997 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:43:39.521958    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-09598-predictor-587794bf84-zl6dc" podUID="7b0bcced-f9ba-4d60-a892-a604b11c68af" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
May 25 12:43:39.522522 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:43:39.522505    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-09598-predictor-7c4d94fb5f-hqnbq"
May 25 12:43:40.707327 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:43:40.707296    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-8902c-predictor-b6f848b6b-rmjnr"
May 25 12:43:40.707731 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:43:40.707689    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-8902c-predictor-86c999b5df-f6nxh"
May 25 12:43:40.707821 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:43:40.707771    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-8902c-predictor-b6f848b6b-rmjnr" podUID="7a6774c9-5000-4f2c-9ece-20506dde280f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
May 25 12:43:40.708234 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:43:40.708189    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-8902c-predictor-86c999b5df-f6nxh" podUID="8caf44b4-889f-4d31-b6d0-1ae3beafd3b7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.53:8080: connect: connection refused"
May 25 12:43:49.522406 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:43:49.522367    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-09598-predictor-587794bf84-zl6dc"
May 25 12:43:50.708127 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:43:50.708072    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-8902c-predictor-b6f848b6b-rmjnr" podUID="7a6774c9-5000-4f2c-9ece-20506dde280f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
May 25 12:43:50.708522 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:43:50.708246    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-8902c-predictor-86c999b5df-f6nxh" podUID="8caf44b4-889f-4d31-b6d0-1ae3beafd3b7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.53:8080: connect: connection refused"
May 25 12:44:00.707825 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:44:00.707783    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-8902c-predictor-b6f848b6b-rmjnr" podUID="7a6774c9-5000-4f2c-9ece-20506dde280f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
May 25 12:44:00.708319 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:44:00.708231    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-8902c-predictor-86c999b5df-f6nxh" podUID="8caf44b4-889f-4d31-b6d0-1ae3beafd3b7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.53:8080: connect: connection refused"
May 25 12:44:10.708148 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:44:10.708102    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-8902c-predictor-b6f848b6b-rmjnr" podUID="7a6774c9-5000-4f2c-9ece-20506dde280f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
May 25 12:44:10.708545 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:44:10.708113    2549 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-8902c-predictor-86c999b5df-f6nxh" podUID="8caf44b4-889f-4d31-b6d0-1ae3beafd3b7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.53:8080: connect: connection refused"
May 25 12:44:20.708357 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:44:20.708326    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-8902c-predictor-b6f848b6b-rmjnr"
May 25 12:44:20.708850 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:44:20.708778    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-8902c-predictor-86c999b5df-f6nxh"
May 25 12:52:46.569944 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:52:46.569906    2549 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-8902c-predictor-b6f848b6b-rmjnr"]
May 25 12:52:46.572449 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:52:46.570228    2549 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-8902c-predictor-b6f848b6b-rmjnr" podUID="7a6774c9-5000-4f2c-9ece-20506dde280f" containerName="kserve-container" containerID="cri-o://0f19969d428fcd94c4ad0437ca124a47c5c85204d16c4fc3105bc526043d5da4" gracePeriod=30
May 25 12:52:46.572449 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:52:46.570257    2549 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-8902c-predictor-b6f848b6b-rmjnr" podUID="7a6774c9-5000-4f2c-9ece-20506dde280f" containerName="kube-rbac-proxy" containerID="cri-o://6649543b215dec0c04127283177ec45ce668cd9db73c0649c6c82b691af03639" gracePeriod=30
May 25 12:52:46.614115 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:52:46.614075    2549 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-8902c-predictor-86c999b5df-f6nxh"]
May 25 12:52:46.614487 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:52:46.614460    2549 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-8902c-predictor-86c999b5df-f6nxh" podUID="8caf44b4-889f-4d31-b6d0-1ae3beafd3b7" containerName="kserve-container" containerID="cri-o://1ab865509867941b74155b31eabd03e9e7de7ebf7afe15689a97e227ab38e0a8" gracePeriod=30
May 25 12:52:46.614615 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:52:46.614499    2549 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-8902c-predictor-86c999b5df-f6nxh" podUID="8caf44b4-889f-4d31-b6d0-1ae3beafd3b7" containerName="kube-rbac-proxy" containerID="cri-o://7b275ab500091e9061847ad286cd666f0018cabf1f481d42cdfd4e2c285fefff" gracePeriod=30
May 25 12:52:46.719807 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:52:46.719768    2549 generic.go:358] "Generic (PLEG): container finished" podID="7a6774c9-5000-4f2c-9ece-20506dde280f" containerID="6649543b215dec0c04127283177ec45ce668cd9db73c0649c6c82b691af03639" exitCode=2
May 25 12:52:46.719975 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:52:46.719831    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-8902c-predictor-b6f848b6b-rmjnr" event={"ID":"7a6774c9-5000-4f2c-9ece-20506dde280f","Type":"ContainerDied","Data":"6649543b215dec0c04127283177ec45ce668cd9db73c0649c6c82b691af03639"}
May 25 12:52:47.725147 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:52:47.725112    2549 generic.go:358] "Generic (PLEG): container finished" podID="8caf44b4-889f-4d31-b6d0-1ae3beafd3b7" containerID="7b275ab500091e9061847ad286cd666f0018cabf1f481d42cdfd4e2c285fefff" exitCode=2
May 25 12:52:47.725560 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:52:47.725181    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-8902c-predictor-86c999b5df-f6nxh" event={"ID":"8caf44b4-889f-4d31-b6d0-1ae3beafd3b7","Type":"ContainerDied","Data":"7b275ab500091e9061847ad286cd666f0018cabf1f481d42cdfd4e2c285fefff"}
May 25 12:52:50.250285 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:52:50.250260    2549 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-8902c-predictor-86c999b5df-f6nxh"
May 25 12:52:50.253560 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:52:50.253533    2549 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-8902c-predictor-b6f848b6b-rmjnr"
May 25 12:52:50.329949 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:52:50.329914    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mbj52\" (UniqueName: \"kubernetes.io/projected/7a6774c9-5000-4f2c-9ece-20506dde280f-kube-api-access-mbj52\") pod \"7a6774c9-5000-4f2c-9ece-20506dde280f\" (UID: \"7a6774c9-5000-4f2c-9ece-20506dde280f\") "
May 25 12:52:50.329949 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:52:50.329952    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"success-200-isvc-8902c-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7a6774c9-5000-4f2c-9ece-20506dde280f-success-200-isvc-8902c-kube-rbac-proxy-sar-config\") pod \"7a6774c9-5000-4f2c-9ece-20506dde280f\" (UID: \"7a6774c9-5000-4f2c-9ece-20506dde280f\") "
May 25 12:52:50.330176 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:52:50.330007    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8caf44b4-889f-4d31-b6d0-1ae3beafd3b7-proxy-tls\") pod \"8caf44b4-889f-4d31-b6d0-1ae3beafd3b7\" (UID: \"8caf44b4-889f-4d31-b6d0-1ae3beafd3b7\") "
May 25 12:52:50.330176 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:52:50.330025    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"error-404-isvc-8902c-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8caf44b4-889f-4d31-b6d0-1ae3beafd3b7-error-404-isvc-8902c-kube-rbac-proxy-sar-config\") pod \"8caf44b4-889f-4d31-b6d0-1ae3beafd3b7\" (UID: \"8caf44b4-889f-4d31-b6d0-1ae3beafd3b7\") "
May 25 12:52:50.330176 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:52:50.330160    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7a6774c9-5000-4f2c-9ece-20506dde280f-proxy-tls\") pod \"7a6774c9-5000-4f2c-9ece-20506dde280f\" (UID: \"7a6774c9-5000-4f2c-9ece-20506dde280f\") "
May 25 12:52:50.330379 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:52:50.330279    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-76jb9\" (UniqueName: \"kubernetes.io/projected/8caf44b4-889f-4d31-b6d0-1ae3beafd3b7-kube-api-access-76jb9\") pod \"8caf44b4-889f-4d31-b6d0-1ae3beafd3b7\" (UID: \"8caf44b4-889f-4d31-b6d0-1ae3beafd3b7\") "
May 25 12:52:50.330458 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:52:50.330425    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a6774c9-5000-4f2c-9ece-20506dde280f-success-200-isvc-8902c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "success-200-isvc-8902c-kube-rbac-proxy-sar-config") pod "7a6774c9-5000-4f2c-9ece-20506dde280f" (UID: "7a6774c9-5000-4f2c-9ece-20506dde280f"). InnerVolumeSpecName "success-200-isvc-8902c-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 12:52:50.330538 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:52:50.330511    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8caf44b4-889f-4d31-b6d0-1ae3beafd3b7-error-404-isvc-8902c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "error-404-isvc-8902c-kube-rbac-proxy-sar-config") pod "8caf44b4-889f-4d31-b6d0-1ae3beafd3b7" (UID: "8caf44b4-889f-4d31-b6d0-1ae3beafd3b7"). InnerVolumeSpecName "error-404-isvc-8902c-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 12:52:50.330704 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:52:50.330682    2549 reconciler_common.go:299] "Volume detached for volume \"success-200-isvc-8902c-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7a6774c9-5000-4f2c-9ece-20506dde280f-success-200-isvc-8902c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
May 25 12:52:50.330786 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:52:50.330711    2549 reconciler_common.go:299] "Volume detached for volume \"error-404-isvc-8902c-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8caf44b4-889f-4d31-b6d0-1ae3beafd3b7-error-404-isvc-8902c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
May 25 12:52:50.332331 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:52:50.332311    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a6774c9-5000-4f2c-9ece-20506dde280f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7a6774c9-5000-4f2c-9ece-20506dde280f" (UID: "7a6774c9-5000-4f2c-9ece-20506dde280f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 12:52:50.332331 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:52:50.332316    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8caf44b4-889f-4d31-b6d0-1ae3beafd3b7-kube-api-access-76jb9" (OuterVolumeSpecName: "kube-api-access-76jb9") pod "8caf44b4-889f-4d31-b6d0-1ae3beafd3b7" (UID: "8caf44b4-889f-4d31-b6d0-1ae3beafd3b7"). InnerVolumeSpecName "kube-api-access-76jb9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 12:52:50.332599 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:52:50.332580    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8caf44b4-889f-4d31-b6d0-1ae3beafd3b7-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8caf44b4-889f-4d31-b6d0-1ae3beafd3b7" (UID: "8caf44b4-889f-4d31-b6d0-1ae3beafd3b7"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 12:52:50.332599 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:52:50.332589    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a6774c9-5000-4f2c-9ece-20506dde280f-kube-api-access-mbj52" (OuterVolumeSpecName: "kube-api-access-mbj52") pod "7a6774c9-5000-4f2c-9ece-20506dde280f" (UID: "7a6774c9-5000-4f2c-9ece-20506dde280f"). InnerVolumeSpecName "kube-api-access-mbj52". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 12:52:50.432186 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:52:50.432140    2549 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7a6774c9-5000-4f2c-9ece-20506dde280f-proxy-tls\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
May 25 12:52:50.432186 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:52:50.432180    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76jb9\" (UniqueName: \"kubernetes.io/projected/8caf44b4-889f-4d31-b6d0-1ae3beafd3b7-kube-api-access-76jb9\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
May 25 12:52:50.432186 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:52:50.432190    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mbj52\" (UniqueName: \"kubernetes.io/projected/7a6774c9-5000-4f2c-9ece-20506dde280f-kube-api-access-mbj52\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
May 25 12:52:50.432445 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:52:50.432232    2549 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8caf44b4-889f-4d31-b6d0-1ae3beafd3b7-proxy-tls\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
May 25 12:52:50.740089 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:52:50.740047    2549 generic.go:358] "Generic (PLEG): container finished" podID="8caf44b4-889f-4d31-b6d0-1ae3beafd3b7" containerID="1ab865509867941b74155b31eabd03e9e7de7ebf7afe15689a97e227ab38e0a8" exitCode=0
May 25 12:52:50.740301 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:52:50.740136    2549 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-8902c-predictor-86c999b5df-f6nxh"
May 25 12:52:50.740301 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:52:50.740131    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-8902c-predictor-86c999b5df-f6nxh" event={"ID":"8caf44b4-889f-4d31-b6d0-1ae3beafd3b7","Type":"ContainerDied","Data":"1ab865509867941b74155b31eabd03e9e7de7ebf7afe15689a97e227ab38e0a8"}
May 25 12:52:50.740301 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:52:50.740259    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-8902c-predictor-86c999b5df-f6nxh" event={"ID":"8caf44b4-889f-4d31-b6d0-1ae3beafd3b7","Type":"ContainerDied","Data":"99002cb6140aaf943f86cb69222db77d148ef00968857a948b0f9c66aa4f0cfd"}
May 25 12:52:50.740301 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:52:50.740283    2549 scope.go:117] "RemoveContainer" containerID="7b275ab500091e9061847ad286cd666f0018cabf1f481d42cdfd4e2c285fefff"
May 25 12:52:50.741451 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:52:50.741428    2549 generic.go:358] "Generic (PLEG): container finished" podID="7a6774c9-5000-4f2c-9ece-20506dde280f" containerID="0f19969d428fcd94c4ad0437ca124a47c5c85204d16c4fc3105bc526043d5da4" exitCode=0
May 25 12:52:50.741535 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:52:50.741506    2549 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-8902c-predictor-b6f848b6b-rmjnr"
May 25 12:52:50.741535 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:52:50.741510    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-8902c-predictor-b6f848b6b-rmjnr" event={"ID":"7a6774c9-5000-4f2c-9ece-20506dde280f","Type":"ContainerDied","Data":"0f19969d428fcd94c4ad0437ca124a47c5c85204d16c4fc3105bc526043d5da4"}
May 25 12:52:50.741648 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:52:50.741539    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-8902c-predictor-b6f848b6b-rmjnr" event={"ID":"7a6774c9-5000-4f2c-9ece-20506dde280f","Type":"ContainerDied","Data":"73e4e0e5bc0f2beb62f6e2330be8303711df04675f0d8eeb9b21bd39498923c8"}
May 25 12:52:50.750451 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:52:50.750268    2549 scope.go:117] "RemoveContainer" containerID="1ab865509867941b74155b31eabd03e9e7de7ebf7afe15689a97e227ab38e0a8"
May 25 12:52:50.759048 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:52:50.759027    2549 scope.go:117] "RemoveContainer" containerID="7b275ab500091e9061847ad286cd666f0018cabf1f481d42cdfd4e2c285fefff"
May 25 12:52:50.759393 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:52:50.759372    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b275ab500091e9061847ad286cd666f0018cabf1f481d42cdfd4e2c285fefff\": container with ID starting with 7b275ab500091e9061847ad286cd666f0018cabf1f481d42cdfd4e2c285fefff not found: ID does not exist" containerID="7b275ab500091e9061847ad286cd666f0018cabf1f481d42cdfd4e2c285fefff"
May 25 12:52:50.759478 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:52:50.759402    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b275ab500091e9061847ad286cd666f0018cabf1f481d42cdfd4e2c285fefff"} err="failed to get container status \"7b275ab500091e9061847ad286cd666f0018cabf1f481d42cdfd4e2c285fefff\": rpc error: code = NotFound desc = could not find container \"7b275ab500091e9061847ad286cd666f0018cabf1f481d42cdfd4e2c285fefff\": container with ID starting with 7b275ab500091e9061847ad286cd666f0018cabf1f481d42cdfd4e2c285fefff not found: ID does not exist"
May 25 12:52:50.759478 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:52:50.759422    2549 scope.go:117] "RemoveContainer" containerID="1ab865509867941b74155b31eabd03e9e7de7ebf7afe15689a97e227ab38e0a8"
May 25 12:52:50.759695 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:52:50.759677    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ab865509867941b74155b31eabd03e9e7de7ebf7afe15689a97e227ab38e0a8\": container with ID starting with 1ab865509867941b74155b31eabd03e9e7de7ebf7afe15689a97e227ab38e0a8 not found: ID does not exist" containerID="1ab865509867941b74155b31eabd03e9e7de7ebf7afe15689a97e227ab38e0a8"
May 25 12:52:50.759736 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:52:50.759701    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ab865509867941b74155b31eabd03e9e7de7ebf7afe15689a97e227ab38e0a8"} err="failed to get container status \"1ab865509867941b74155b31eabd03e9e7de7ebf7afe15689a97e227ab38e0a8\": rpc error: code = NotFound desc = could not find container \"1ab865509867941b74155b31eabd03e9e7de7ebf7afe15689a97e227ab38e0a8\": container with ID starting with 1ab865509867941b74155b31eabd03e9e7de7ebf7afe15689a97e227ab38e0a8 not found: ID does not exist"
May 25 12:52:50.759736 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:52:50.759717    2549 scope.go:117] "RemoveContainer" containerID="6649543b215dec0c04127283177ec45ce668cd9db73c0649c6c82b691af03639"
May 25 12:52:50.768733 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:52:50.768715    2549 scope.go:117] "RemoveContainer" containerID="0f19969d428fcd94c4ad0437ca124a47c5c85204d16c4fc3105bc526043d5da4"
May 25 12:52:50.769441 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:52:50.769418    2549 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-8902c-predictor-86c999b5df-f6nxh"]
May 25 12:52:50.773995 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:52:50.773967    2549 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-8902c-predictor-86c999b5df-f6nxh"]
May 25 12:52:50.777317 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:52:50.777299    2549 scope.go:117] "RemoveContainer" containerID="6649543b215dec0c04127283177ec45ce668cd9db73c0649c6c82b691af03639"
May 25 12:52:50.777606 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:52:50.777587    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6649543b215dec0c04127283177ec45ce668cd9db73c0649c6c82b691af03639\": container with ID starting with 6649543b215dec0c04127283177ec45ce668cd9db73c0649c6c82b691af03639 not found: ID does not exist" containerID="6649543b215dec0c04127283177ec45ce668cd9db73c0649c6c82b691af03639"
May 25 12:52:50.777651 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:52:50.777615    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6649543b215dec0c04127283177ec45ce668cd9db73c0649c6c82b691af03639"} err="failed to get container status \"6649543b215dec0c04127283177ec45ce668cd9db73c0649c6c82b691af03639\": rpc error: code = NotFound desc = could not find container \"6649543b215dec0c04127283177ec45ce668cd9db73c0649c6c82b691af03639\": container with ID starting with 6649543b215dec0c04127283177ec45ce668cd9db73c0649c6c82b691af03639 not found: ID does not exist"
May 25 12:52:50.777651 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:52:50.777632    2549 scope.go:117] "RemoveContainer" containerID="0f19969d428fcd94c4ad0437ca124a47c5c85204d16c4fc3105bc526043d5da4"
May 25 12:52:50.777886 ip-10-0-142-53 kubenswrapper[2549]: E0525 12:52:50.777868    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f19969d428fcd94c4ad0437ca124a47c5c85204d16c4fc3105bc526043d5da4\": container with ID starting with 0f19969d428fcd94c4ad0437ca124a47c5c85204d16c4fc3105bc526043d5da4 not found: ID does not exist" containerID="0f19969d428fcd94c4ad0437ca124a47c5c85204d16c4fc3105bc526043d5da4"
May 25 12:52:50.777929 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:52:50.777892    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f19969d428fcd94c4ad0437ca124a47c5c85204d16c4fc3105bc526043d5da4"} err="failed to get container status \"0f19969d428fcd94c4ad0437ca124a47c5c85204d16c4fc3105bc526043d5da4\": rpc error: code = NotFound desc = could not find container \"0f19969d428fcd94c4ad0437ca124a47c5c85204d16c4fc3105bc526043d5da4\": container with ID starting with 0f19969d428fcd94c4ad0437ca124a47c5c85204d16c4fc3105bc526043d5da4 not found: ID does not exist"
May 25 12:52:50.787019 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:52:50.786992    2549 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-8902c-predictor-b6f848b6b-rmjnr"]
May 25 12:52:50.791247 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:52:50.791222    2549 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-8902c-predictor-b6f848b6b-rmjnr"]
May 25 12:52:50.853130 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:52:50.853087    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7a6774c9-5000-4f2c-9ece-20506dde280f" path="/var/lib/kubelet/pods/7a6774c9-5000-4f2c-9ece-20506dde280f/volumes"
May 25 12:52:50.853593 ip-10-0-142-53 kubenswrapper[2549]: I0525 12:52:50.853577    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8caf44b4-889f-4d31-b6d0-1ae3beafd3b7" path="/var/lib/kubelet/pods/8caf44b4-889f-4d31-b6d0-1ae3beafd3b7/volumes"
May 25 13:00:20.793928 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:00:20.793891    2549 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-09598-predictor-7c4d94fb5f-hqnbq"]
May 25 13:00:20.796436 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:00:20.794183    2549 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-09598-predictor-7c4d94fb5f-hqnbq" podUID="017fafb4-c256-47d2-b271-a6c2feff2636" containerName="kserve-container" containerID="cri-o://16addaa2fdd22777687d5c31501caea4ece0bdb18f98d82c4f946e7741cc221d" gracePeriod=30
May 25 13:00:20.796436 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:00:20.794242    2549 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-09598-predictor-7c4d94fb5f-hqnbq" podUID="017fafb4-c256-47d2-b271-a6c2feff2636" containerName="kube-rbac-proxy" containerID="cri-o://c068f158d0fa76499f94bf39f2a656298d469fe3b0e411307e8b12a4db215002" gracePeriod=30
May 25 13:00:20.851923 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:00:20.851893    2549 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-09598-predictor-587794bf84-zl6dc"]
May 25 13:00:20.852147 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:00:20.852123    2549 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-09598-predictor-587794bf84-zl6dc" podUID="7b0bcced-f9ba-4d60-a892-a604b11c68af" containerName="kserve-container" containerID="cri-o://f8b3ae24913bc9ec0dd89022b47c218c91b785bd62b7e4830af85279caa9c4da" gracePeriod=30
May 25 13:00:20.852249 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:00:20.852158    2549 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-09598-predictor-587794bf84-zl6dc" podUID="7b0bcced-f9ba-4d60-a892-a604b11c68af" containerName="kube-rbac-proxy" containerID="cri-o://1db7e497da00c556f0d947e3c02c176f5827d2be67a1240b42e3986cd2b6ba41" gracePeriod=30
May 25 13:00:21.352739 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:00:21.352706    2549 generic.go:358] "Generic (PLEG): container finished" podID="017fafb4-c256-47d2-b271-a6c2feff2636" containerID="c068f158d0fa76499f94bf39f2a656298d469fe3b0e411307e8b12a4db215002" exitCode=2
May 25 13:00:21.352923 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:00:21.352776    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-09598-predictor-7c4d94fb5f-hqnbq" event={"ID":"017fafb4-c256-47d2-b271-a6c2feff2636","Type":"ContainerDied","Data":"c068f158d0fa76499f94bf39f2a656298d469fe3b0e411307e8b12a4db215002"}
May 25 13:00:21.354430 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:00:21.354409    2549 generic.go:358] "Generic (PLEG): container finished" podID="7b0bcced-f9ba-4d60-a892-a604b11c68af" containerID="1db7e497da00c556f0d947e3c02c176f5827d2be67a1240b42e3986cd2b6ba41" exitCode=2
May 25 13:00:21.354534 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:00:21.354457    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-09598-predictor-587794bf84-zl6dc" event={"ID":"7b0bcced-f9ba-4d60-a892-a604b11c68af","Type":"ContainerDied","Data":"1db7e497da00c556f0d947e3c02c176f5827d2be67a1240b42e3986cd2b6ba41"}
May 25 13:00:23.871698 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:00:23.871673    2549 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-09598-predictor-7c4d94fb5f-hqnbq"
May 25 13:00:23.964587 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:00:23.964500    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/017fafb4-c256-47d2-b271-a6c2feff2636-proxy-tls\") pod \"017fafb4-c256-47d2-b271-a6c2feff2636\" (UID: \"017fafb4-c256-47d2-b271-a6c2feff2636\") "
May 25 13:00:23.964587 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:00:23.964580    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"error-404-isvc-09598-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/017fafb4-c256-47d2-b271-a6c2feff2636-error-404-isvc-09598-kube-rbac-proxy-sar-config\") pod \"017fafb4-c256-47d2-b271-a6c2feff2636\" (UID: \"017fafb4-c256-47d2-b271-a6c2feff2636\") "
May 25 13:00:23.964791 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:00:23.964676    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5gmd8\" (UniqueName: \"kubernetes.io/projected/017fafb4-c256-47d2-b271-a6c2feff2636-kube-api-access-5gmd8\") pod \"017fafb4-c256-47d2-b271-a6c2feff2636\" (UID: \"017fafb4-c256-47d2-b271-a6c2feff2636\") "
May 25 13:00:23.964980 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:00:23.964946    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/017fafb4-c256-47d2-b271-a6c2feff2636-error-404-isvc-09598-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "error-404-isvc-09598-kube-rbac-proxy-sar-config") pod "017fafb4-c256-47d2-b271-a6c2feff2636" (UID: "017fafb4-c256-47d2-b271-a6c2feff2636"). InnerVolumeSpecName "error-404-isvc-09598-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 13:00:23.966794 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:00:23.966762    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/017fafb4-c256-47d2-b271-a6c2feff2636-kube-api-access-5gmd8" (OuterVolumeSpecName: "kube-api-access-5gmd8") pod "017fafb4-c256-47d2-b271-a6c2feff2636" (UID: "017fafb4-c256-47d2-b271-a6c2feff2636"). InnerVolumeSpecName "kube-api-access-5gmd8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 13:00:23.967101 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:00:23.967080    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/017fafb4-c256-47d2-b271-a6c2feff2636-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "017fafb4-c256-47d2-b271-a6c2feff2636" (UID: "017fafb4-c256-47d2-b271-a6c2feff2636"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 13:00:23.997834 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:00:23.997814    2549 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-09598-predictor-587794bf84-zl6dc"
May 25 13:00:24.065282 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:00:24.065244    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7b0bcced-f9ba-4d60-a892-a604b11c68af-proxy-tls\") pod \"7b0bcced-f9ba-4d60-a892-a604b11c68af\" (UID: \"7b0bcced-f9ba-4d60-a892-a604b11c68af\") "
May 25 13:00:24.065459 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:00:24.065312    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g24bq\" (UniqueName: \"kubernetes.io/projected/7b0bcced-f9ba-4d60-a892-a604b11c68af-kube-api-access-g24bq\") pod \"7b0bcced-f9ba-4d60-a892-a604b11c68af\" (UID: \"7b0bcced-f9ba-4d60-a892-a604b11c68af\") "
May 25 13:00:24.065459 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:00:24.065358    2549 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"success-200-isvc-09598-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7b0bcced-f9ba-4d60-a892-a604b11c68af-success-200-isvc-09598-kube-rbac-proxy-sar-config\") pod \"7b0bcced-f9ba-4d60-a892-a604b11c68af\" (UID: \"7b0bcced-f9ba-4d60-a892-a604b11c68af\") "
May 25 13:00:24.065586 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:00:24.065520    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5gmd8\" (UniqueName: \"kubernetes.io/projected/017fafb4-c256-47d2-b271-a6c2feff2636-kube-api-access-5gmd8\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
May 25 13:00:24.065586 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:00:24.065536    2549 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/017fafb4-c256-47d2-b271-a6c2feff2636-proxy-tls\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
May 25 13:00:24.065586 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:00:24.065550    2549 reconciler_common.go:299] "Volume detached for volume \"error-404-isvc-09598-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/017fafb4-c256-47d2-b271-a6c2feff2636-error-404-isvc-09598-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
May 25 13:00:24.065804 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:00:24.065776    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b0bcced-f9ba-4d60-a892-a604b11c68af-success-200-isvc-09598-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "success-200-isvc-09598-kube-rbac-proxy-sar-config") pod "7b0bcced-f9ba-4d60-a892-a604b11c68af" (UID: "7b0bcced-f9ba-4d60-a892-a604b11c68af"). InnerVolumeSpecName "success-200-isvc-09598-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 13:00:24.067377 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:00:24.067355    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b0bcced-f9ba-4d60-a892-a604b11c68af-kube-api-access-g24bq" (OuterVolumeSpecName: "kube-api-access-g24bq") pod "7b0bcced-f9ba-4d60-a892-a604b11c68af" (UID: "7b0bcced-f9ba-4d60-a892-a604b11c68af"). InnerVolumeSpecName "kube-api-access-g24bq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 13:00:24.067446 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:00:24.067377    2549 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b0bcced-f9ba-4d60-a892-a604b11c68af-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7b0bcced-f9ba-4d60-a892-a604b11c68af" (UID: "7b0bcced-f9ba-4d60-a892-a604b11c68af"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 13:00:24.166950 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:00:24.166908    2549 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g24bq\" (UniqueName: \"kubernetes.io/projected/7b0bcced-f9ba-4d60-a892-a604b11c68af-kube-api-access-g24bq\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
May 25 13:00:24.166950 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:00:24.166947    2549 reconciler_common.go:299] "Volume detached for volume \"success-200-isvc-09598-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7b0bcced-f9ba-4d60-a892-a604b11c68af-success-200-isvc-09598-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
May 25 13:00:24.167161 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:00:24.166961    2549 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7b0bcced-f9ba-4d60-a892-a604b11c68af-proxy-tls\") on node \"ip-10-0-142-53.ec2.internal\" DevicePath \"\""
May 25 13:00:24.366860 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:00:24.366816    2549 generic.go:358] "Generic (PLEG): container finished" podID="7b0bcced-f9ba-4d60-a892-a604b11c68af" containerID="f8b3ae24913bc9ec0dd89022b47c218c91b785bd62b7e4830af85279caa9c4da" exitCode=0
May 25 13:00:24.367061 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:00:24.366902    2549 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-09598-predictor-587794bf84-zl6dc"
May 25 13:00:24.367061 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:00:24.366917    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-09598-predictor-587794bf84-zl6dc" event={"ID":"7b0bcced-f9ba-4d60-a892-a604b11c68af","Type":"ContainerDied","Data":"f8b3ae24913bc9ec0dd89022b47c218c91b785bd62b7e4830af85279caa9c4da"}
May 25 13:00:24.367061 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:00:24.366955    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-09598-predictor-587794bf84-zl6dc" event={"ID":"7b0bcced-f9ba-4d60-a892-a604b11c68af","Type":"ContainerDied","Data":"9361707e236acf1ddd81c2a17f24853c33ca280d293cef92f56feb77593f582c"}
May 25 13:00:24.367061 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:00:24.366972    2549 scope.go:117] "RemoveContainer" containerID="1db7e497da00c556f0d947e3c02c176f5827d2be67a1240b42e3986cd2b6ba41"
May 25 13:00:24.368496 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:00:24.368458    2549 generic.go:358] "Generic (PLEG): container finished" podID="017fafb4-c256-47d2-b271-a6c2feff2636" containerID="16addaa2fdd22777687d5c31501caea4ece0bdb18f98d82c4f946e7741cc221d" exitCode=0
May 25 13:00:24.368620 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:00:24.368526    2549 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-09598-predictor-7c4d94fb5f-hqnbq"
May 25 13:00:24.368620 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:00:24.368530    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-09598-predictor-7c4d94fb5f-hqnbq" event={"ID":"017fafb4-c256-47d2-b271-a6c2feff2636","Type":"ContainerDied","Data":"16addaa2fdd22777687d5c31501caea4ece0bdb18f98d82c4f946e7741cc221d"}
May 25 13:00:24.368620 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:00:24.368556    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-09598-predictor-7c4d94fb5f-hqnbq" event={"ID":"017fafb4-c256-47d2-b271-a6c2feff2636","Type":"ContainerDied","Data":"ab3b94ac3ac1cac71e95287117815d2eb28ef880185bf72256e971ba30c7b6f7"}
May 25 13:00:24.377027 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:00:24.376998    2549 scope.go:117] "RemoveContainer" containerID="f8b3ae24913bc9ec0dd89022b47c218c91b785bd62b7e4830af85279caa9c4da"
May 25 13:00:24.385050 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:00:24.385028    2549 scope.go:117] "RemoveContainer" containerID="1db7e497da00c556f0d947e3c02c176f5827d2be67a1240b42e3986cd2b6ba41"
May 25 13:00:24.385339 ip-10-0-142-53 kubenswrapper[2549]: E0525 13:00:24.385317    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1db7e497da00c556f0d947e3c02c176f5827d2be67a1240b42e3986cd2b6ba41\": container with ID starting with 1db7e497da00c556f0d947e3c02c176f5827d2be67a1240b42e3986cd2b6ba41 not found: ID does not exist" containerID="1db7e497da00c556f0d947e3c02c176f5827d2be67a1240b42e3986cd2b6ba41"
May 25 13:00:24.385420 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:00:24.385347    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1db7e497da00c556f0d947e3c02c176f5827d2be67a1240b42e3986cd2b6ba41"} err="failed to get container status \"1db7e497da00c556f0d947e3c02c176f5827d2be67a1240b42e3986cd2b6ba41\": rpc error: code = NotFound desc = could not find container \"1db7e497da00c556f0d947e3c02c176f5827d2be67a1240b42e3986cd2b6ba41\": container with ID starting with 1db7e497da00c556f0d947e3c02c176f5827d2be67a1240b42e3986cd2b6ba41 not found: ID does not exist"
May 25 13:00:24.385420 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:00:24.385370    2549 scope.go:117] "RemoveContainer" containerID="f8b3ae24913bc9ec0dd89022b47c218c91b785bd62b7e4830af85279caa9c4da"
May 25 13:00:24.385582 ip-10-0-142-53 kubenswrapper[2549]: E0525 13:00:24.385559    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8b3ae24913bc9ec0dd89022b47c218c91b785bd62b7e4830af85279caa9c4da\": container with ID starting with f8b3ae24913bc9ec0dd89022b47c218c91b785bd62b7e4830af85279caa9c4da not found: ID does not exist" containerID="f8b3ae24913bc9ec0dd89022b47c218c91b785bd62b7e4830af85279caa9c4da"
May 25 13:00:24.385624 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:00:24.385592    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8b3ae24913bc9ec0dd89022b47c218c91b785bd62b7e4830af85279caa9c4da"} err="failed to get container status \"f8b3ae24913bc9ec0dd89022b47c218c91b785bd62b7e4830af85279caa9c4da\": rpc error: code = NotFound desc = could not find container \"f8b3ae24913bc9ec0dd89022b47c218c91b785bd62b7e4830af85279caa9c4da\": container with ID starting with f8b3ae24913bc9ec0dd89022b47c218c91b785bd62b7e4830af85279caa9c4da not found: ID does not exist"
May 25 13:00:24.385624 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:00:24.385611    2549 scope.go:117] "RemoveContainer" containerID="c068f158d0fa76499f94bf39f2a656298d469fe3b0e411307e8b12a4db215002"
May 25 13:00:24.394069 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:00:24.394050    2549 scope.go:117] "RemoveContainer" containerID="16addaa2fdd22777687d5c31501caea4ece0bdb18f98d82c4f946e7741cc221d"
May 25 13:00:24.394748 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:00:24.394724    2549 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-09598-predictor-7c4d94fb5f-hqnbq"]
May 25 13:00:24.398967 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:00:24.398948    2549 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-09598-predictor-7c4d94fb5f-hqnbq"]
May 25 13:00:24.401676 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:00:24.401661    2549 scope.go:117] "RemoveContainer" containerID="c068f158d0fa76499f94bf39f2a656298d469fe3b0e411307e8b12a4db215002"
May 25 13:00:24.401919 ip-10-0-142-53 kubenswrapper[2549]: E0525 13:00:24.401902    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c068f158d0fa76499f94bf39f2a656298d469fe3b0e411307e8b12a4db215002\": container with ID starting with c068f158d0fa76499f94bf39f2a656298d469fe3b0e411307e8b12a4db215002 not found: ID does not exist" containerID="c068f158d0fa76499f94bf39f2a656298d469fe3b0e411307e8b12a4db215002"
May 25 13:00:24.401966 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:00:24.401926    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c068f158d0fa76499f94bf39f2a656298d469fe3b0e411307e8b12a4db215002"} err="failed to get container status \"c068f158d0fa76499f94bf39f2a656298d469fe3b0e411307e8b12a4db215002\": rpc error: code = NotFound desc = could not find container \"c068f158d0fa76499f94bf39f2a656298d469fe3b0e411307e8b12a4db215002\": container with ID starting with c068f158d0fa76499f94bf39f2a656298d469fe3b0e411307e8b12a4db215002 not found: ID does not exist"
May 25 13:00:24.401966 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:00:24.401944    2549 scope.go:117] "RemoveContainer" containerID="16addaa2fdd22777687d5c31501caea4ece0bdb18f98d82c4f946e7741cc221d"
May 25 13:00:24.402157 ip-10-0-142-53 kubenswrapper[2549]: E0525 13:00:24.402139    2549 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16addaa2fdd22777687d5c31501caea4ece0bdb18f98d82c4f946e7741cc221d\": container with ID starting with 16addaa2fdd22777687d5c31501caea4ece0bdb18f98d82c4f946e7741cc221d not found: ID does not exist" containerID="16addaa2fdd22777687d5c31501caea4ece0bdb18f98d82c4f946e7741cc221d"
May 25 13:00:24.402223 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:00:24.402163    2549 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16addaa2fdd22777687d5c31501caea4ece0bdb18f98d82c4f946e7741cc221d"} err="failed to get container status \"16addaa2fdd22777687d5c31501caea4ece0bdb18f98d82c4f946e7741cc221d\": rpc error: code = NotFound desc = could not find container \"16addaa2fdd22777687d5c31501caea4ece0bdb18f98d82c4f946e7741cc221d\": container with ID starting with 16addaa2fdd22777687d5c31501caea4ece0bdb18f98d82c4f946e7741cc221d not found: ID does not exist"
May 25 13:00:24.410343 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:00:24.410302    2549 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-09598-predictor-587794bf84-zl6dc"]
May 25 13:00:24.412343 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:00:24.412324    2549 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-09598-predictor-587794bf84-zl6dc"]
May 25 13:00:24.847744 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:00:24.847712    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="017fafb4-c256-47d2-b271-a6c2feff2636" path="/var/lib/kubelet/pods/017fafb4-c256-47d2-b271-a6c2feff2636/volumes"
May 25 13:00:24.848133 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:00:24.848120    2549 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7b0bcced-f9ba-4d60-a892-a604b11c68af" path="/var/lib/kubelet/pods/7b0bcced-f9ba-4d60-a892-a604b11c68af/volumes"
May 25 13:00:52.161924 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:00:52.161885    2549 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-must-gather-9vddf/must-gather-xq6hn"]
May 25 13:00:52.162406 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:00:52.162277    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="90636a58-53a5-4c10-864f-cdf6647bd677" containerName="kube-rbac-proxy"
May 25 13:00:52.162406 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:00:52.162289    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="90636a58-53a5-4c10-864f-cdf6647bd677" containerName="kube-rbac-proxy"
May 25 13:00:52.162406 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:00:52.162299    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="017fafb4-c256-47d2-b271-a6c2feff2636" containerName="kube-rbac-proxy"
May 25 13:00:52.162406 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:00:52.162305    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="017fafb4-c256-47d2-b271-a6c2feff2636" containerName="kube-rbac-proxy"
May 25 13:00:52.162406 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:00:52.162317    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b0bcced-f9ba-4d60-a892-a604b11c68af" containerName="kserve-container"
May 25 13:00:52.162406 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:00:52.162324    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b0bcced-f9ba-4d60-a892-a604b11c68af" containerName="kserve-container"
May 25 13:00:52.162406 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:00:52.162331    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b0bcced-f9ba-4d60-a892-a604b11c68af" containerName="kube-rbac-proxy"
May 25 13:00:52.162406 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:00:52.162336    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b0bcced-f9ba-4d60-a892-a604b11c68af" containerName="kube-rbac-proxy"
May 25 13:00:52.162406 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:00:52.162343    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8caf44b4-889f-4d31-b6d0-1ae3beafd3b7" containerName="kube-rbac-proxy"
May 25 13:00:52.162406 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:00:52.162348    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="8caf44b4-889f-4d31-b6d0-1ae3beafd3b7" containerName="kube-rbac-proxy"
May 25 13:00:52.162406 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:00:52.162355    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ab201cd-f34e-4d4c-aa98-179dee7a9ede" containerName="kserve-container"
May 25 13:00:52.162406 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:00:52.162360    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ab201cd-f34e-4d4c-aa98-179dee7a9ede" containerName="kserve-container"
May 25 13:00:52.162406 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:00:52.162365    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8caf44b4-889f-4d31-b6d0-1ae3beafd3b7" containerName="kserve-container"
May 25 13:00:52.162406 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:00:52.162371    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="8caf44b4-889f-4d31-b6d0-1ae3beafd3b7" containerName="kserve-container"
May 25 13:00:52.162406 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:00:52.162380    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="017fafb4-c256-47d2-b271-a6c2feff2636" containerName="kserve-container"
May 25 13:00:52.162406 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:00:52.162386    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="017fafb4-c256-47d2-b271-a6c2feff2636" containerName="kserve-container"
May 25 13:00:52.162406 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:00:52.162392    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a6774c9-5000-4f2c-9ece-20506dde280f" containerName="kserve-container"
May 25 13:00:52.162406 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:00:52.162397    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a6774c9-5000-4f2c-9ece-20506dde280f" containerName="kserve-container"
May 25 13:00:52.162406 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:00:52.162404    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ab201cd-f34e-4d4c-aa98-179dee7a9ede" containerName="kube-rbac-proxy"
May 25 13:00:52.162406 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:00:52.162409    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ab201cd-f34e-4d4c-aa98-179dee7a9ede" containerName="kube-rbac-proxy"
May 25 13:00:52.162406 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:00:52.162417    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="90636a58-53a5-4c10-864f-cdf6647bd677" containerName="kserve-container"
May 25 13:00:52.163029 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:00:52.162422    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="90636a58-53a5-4c10-864f-cdf6647bd677" containerName="kserve-container"
May 25 13:00:52.163029 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:00:52.162428    2549 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a6774c9-5000-4f2c-9ece-20506dde280f" containerName="kube-rbac-proxy"
May 25 13:00:52.163029 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:00:52.162433    2549 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a6774c9-5000-4f2c-9ece-20506dde280f" containerName="kube-rbac-proxy"
May 25 13:00:52.163029 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:00:52.162486    2549 memory_manager.go:356] "RemoveStaleState removing state" podUID="90636a58-53a5-4c10-864f-cdf6647bd677" containerName="kube-rbac-proxy"
May 25 13:00:52.163029 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:00:52.162493    2549 memory_manager.go:356] "RemoveStaleState removing state" podUID="8caf44b4-889f-4d31-b6d0-1ae3beafd3b7" containerName="kube-rbac-proxy"
May 25 13:00:52.163029 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:00:52.162499    2549 memory_manager.go:356] "RemoveStaleState removing state" podUID="7a6774c9-5000-4f2c-9ece-20506dde280f" containerName="kube-rbac-proxy"
May 25 13:00:52.163029 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:00:52.162507    2549 memory_manager.go:356] "RemoveStaleState removing state" podUID="7a6774c9-5000-4f2c-9ece-20506dde280f" containerName="kserve-container"
May 25 13:00:52.163029 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:00:52.162513    2549 memory_manager.go:356] "RemoveStaleState removing state" podUID="4ab201cd-f34e-4d4c-aa98-179dee7a9ede" containerName="kube-rbac-proxy"
May 25 13:00:52.163029 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:00:52.162519    2549 memory_manager.go:356] "RemoveStaleState removing state" podUID="017fafb4-c256-47d2-b271-a6c2feff2636" containerName="kserve-container"
May 25 13:00:52.163029 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:00:52.162526    2549 memory_manager.go:356] "RemoveStaleState removing state" podUID="017fafb4-c256-47d2-b271-a6c2feff2636" containerName="kube-rbac-proxy"
May 25 13:00:52.163029 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:00:52.162532    2549 memory_manager.go:356] "RemoveStaleState removing state" podUID="4ab201cd-f34e-4d4c-aa98-179dee7a9ede" containerName="kserve-container"
May 25 13:00:52.163029 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:00:52.162539    2549 memory_manager.go:356] "RemoveStaleState removing state" podUID="7b0bcced-f9ba-4d60-a892-a604b11c68af" containerName="kube-rbac-proxy"
May 25 13:00:52.163029 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:00:52.162547    2549 memory_manager.go:356] "RemoveStaleState removing state" podUID="7b0bcced-f9ba-4d60-a892-a604b11c68af" containerName="kserve-container"
May 25 13:00:52.163029 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:00:52.162554    2549 memory_manager.go:356] "RemoveStaleState removing state" podUID="8caf44b4-889f-4d31-b6d0-1ae3beafd3b7" containerName="kserve-container"
May 25 13:00:52.163029 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:00:52.162559    2549 memory_manager.go:356] "RemoveStaleState removing state" podUID="90636a58-53a5-4c10-864f-cdf6647bd677" containerName="kserve-container"
May 25 13:00:52.165368 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:00:52.165352    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9vddf/must-gather-xq6hn"
May 25 13:00:52.168192 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:00:52.168170    2549 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-9vddf\"/\"kube-root-ca.crt\""
May 25 13:00:52.169269 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:00:52.169248    2549 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-9vddf\"/\"default-dockercfg-cdbvg\""
May 25 13:00:52.169387 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:00:52.169251    2549 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-9vddf\"/\"openshift-service-ca.crt\""
May 25 13:00:52.172645 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:00:52.172235    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9vddf/must-gather-xq6hn"]
May 25 13:00:52.307900 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:00:52.307864    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/50b792e2-630c-4cb8-87f9-51110f3e97ae-must-gather-output\") pod \"must-gather-xq6hn\" (UID: \"50b792e2-630c-4cb8-87f9-51110f3e97ae\") " pod="openshift-must-gather-9vddf/must-gather-xq6hn"
May 25 13:00:52.307900 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:00:52.307908    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cg968\" (UniqueName: \"kubernetes.io/projected/50b792e2-630c-4cb8-87f9-51110f3e97ae-kube-api-access-cg968\") pod \"must-gather-xq6hn\" (UID: \"50b792e2-630c-4cb8-87f9-51110f3e97ae\") " pod="openshift-must-gather-9vddf/must-gather-xq6hn"
May 25 13:00:52.409014 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:00:52.408971    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/50b792e2-630c-4cb8-87f9-51110f3e97ae-must-gather-output\") pod \"must-gather-xq6hn\" (UID: \"50b792e2-630c-4cb8-87f9-51110f3e97ae\") " pod="openshift-must-gather-9vddf/must-gather-xq6hn"
May 25 13:00:52.409014 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:00:52.409019    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-cg968\" (UniqueName: \"kubernetes.io/projected/50b792e2-630c-4cb8-87f9-51110f3e97ae-kube-api-access-cg968\") pod \"must-gather-xq6hn\" (UID: \"50b792e2-630c-4cb8-87f9-51110f3e97ae\") " pod="openshift-must-gather-9vddf/must-gather-xq6hn"
May 25 13:00:52.409347 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:00:52.409326    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/50b792e2-630c-4cb8-87f9-51110f3e97ae-must-gather-output\") pod \"must-gather-xq6hn\" (UID: \"50b792e2-630c-4cb8-87f9-51110f3e97ae\") " pod="openshift-must-gather-9vddf/must-gather-xq6hn"
May 25 13:00:52.419220 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:00:52.419167    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cg968\" (UniqueName: \"kubernetes.io/projected/50b792e2-630c-4cb8-87f9-51110f3e97ae-kube-api-access-cg968\") pod \"must-gather-xq6hn\" (UID: \"50b792e2-630c-4cb8-87f9-51110f3e97ae\") " pod="openshift-must-gather-9vddf/must-gather-xq6hn"
May 25 13:00:52.485611 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:00:52.485583    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9vddf/must-gather-xq6hn"
May 25 13:00:52.618219 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:00:52.618176    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9vddf/must-gather-xq6hn"]
May 25 13:00:52.620604 ip-10-0-142-53 kubenswrapper[2549]: W0525 13:00:52.620579    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod50b792e2_630c_4cb8_87f9_51110f3e97ae.slice/crio-eaa0114bcdbd245b3e47fcb4edf6c4448e415fa3f14bec4a4921926606ebd11a WatchSource:0}: Error finding container eaa0114bcdbd245b3e47fcb4edf6c4448e415fa3f14bec4a4921926606ebd11a: Status 404 returned error can't find the container with id eaa0114bcdbd245b3e47fcb4edf6c4448e415fa3f14bec4a4921926606ebd11a
May 25 13:00:52.622381 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:00:52.622361    2549 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 25 13:00:53.472610 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:00:53.472583    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9vddf/must-gather-xq6hn" event={"ID":"50b792e2-630c-4cb8-87f9-51110f3e97ae","Type":"ContainerStarted","Data":"eaa0114bcdbd245b3e47fcb4edf6c4448e415fa3f14bec4a4921926606ebd11a"}
May 25 13:00:54.480144 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:00:54.479463    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9vddf/must-gather-xq6hn" event={"ID":"50b792e2-630c-4cb8-87f9-51110f3e97ae","Type":"ContainerStarted","Data":"3dfe62af3cbc082d5c9b90a194eead2d4376bce8db5e96f40f0a9bb3777af0e0"}
May 25 13:00:54.480144 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:00:54.479514    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9vddf/must-gather-xq6hn" event={"ID":"50b792e2-630c-4cb8-87f9-51110f3e97ae","Type":"ContainerStarted","Data":"61d2f61773842ef9df5600fe0f18b05cb6408dcbb6c5ba77750f5889f937fe81"}
May 25 13:00:54.499102 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:00:54.499043    2549 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-must-gather-9vddf/must-gather-xq6hn" podStartSLOduration=1.7318228599999999 podStartE2EDuration="2.499027305s" podCreationTimestamp="2026-05-25 13:00:52 +0000 UTC" firstStartedPulling="2026-05-25 13:00:52.622487053 +0000 UTC m=+2856.338302319" lastFinishedPulling="2026-05-25 13:00:53.389691493 +0000 UTC m=+2857.105506764" observedRunningTime="2026-05-25 13:00:54.496932492 +0000 UTC m=+2858.212747779" watchObservedRunningTime="2026-05-25 13:00:54.499027305 +0000 UTC m=+2858.214842592"
May 25 13:00:54.991444 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:00:54.991413    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-cxn5k_271c3650-d1f0-4e2c-93f7-a2a9fd04f3b8/global-pull-secret-syncer/0.log"
May 25 13:00:55.104420 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:00:55.104380    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-tcp4j_3472ac4b-9b58-4e85-85e8-f006315d8915/konnectivity-agent/0.log"
May 25 13:00:55.159643 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:00:55.159612    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-142-53.ec2.internal_dd968f903a046947131232f9282e5635/haproxy/0.log"
May 25 13:00:58.447509 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:00:58.447476    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_0e0aaee5-af52-40f2-b0be-923d0f8467a5/alertmanager/0.log"
May 25 13:00:58.475243 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:00:58.475214    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_0e0aaee5-af52-40f2-b0be-923d0f8467a5/config-reloader/0.log"
May 25 13:00:58.497029 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:00:58.496998    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_0e0aaee5-af52-40f2-b0be-923d0f8467a5/kube-rbac-proxy-web/0.log"
May 25 13:00:58.520300 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:00:58.520264    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_0e0aaee5-af52-40f2-b0be-923d0f8467a5/kube-rbac-proxy/0.log"
May 25 13:00:58.553480 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:00:58.553451    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_0e0aaee5-af52-40f2-b0be-923d0f8467a5/kube-rbac-proxy-metric/0.log"
May 25 13:00:58.579013 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:00:58.578989    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_0e0aaee5-af52-40f2-b0be-923d0f8467a5/prom-label-proxy/0.log"
May 25 13:00:58.603040 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:00:58.602963    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_0e0aaee5-af52-40f2-b0be-923d0f8467a5/init-config-reloader/0.log"
May 25 13:00:58.664632 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:00:58.664596    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-54fd6bc5c4-c8zff_1b52309c-162a-413a-a514-801d0da5a5dd/kube-state-metrics/0.log"
May 25 13:00:58.689666 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:00:58.689598    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-54fd6bc5c4-c8zff_1b52309c-162a-413a-a514-801d0da5a5dd/kube-rbac-proxy-main/0.log"
May 25 13:00:58.717305 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:00:58.717267    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-54fd6bc5c4-c8zff_1b52309c-162a-413a-a514-801d0da5a5dd/kube-rbac-proxy-self/0.log"
May 25 13:00:58.776496 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:00:58.776464    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-665c9c8584-dpgd2_891d3439-8489-46f4-9612-dbce761a67c6/monitoring-plugin/0.log"
May 25 13:00:58.963632 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:00:58.963535    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-w4csw_8ddcb91f-25a2-4a72-b02a-19b0f85f855d/node-exporter/0.log"
May 25 13:00:58.984487 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:00:58.984456    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-w4csw_8ddcb91f-25a2-4a72-b02a-19b0f85f855d/kube-rbac-proxy/0.log"
May 25 13:00:59.006083 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:00:59.006057    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-w4csw_8ddcb91f-25a2-4a72-b02a-19b0f85f855d/init-textfile/0.log"
May 25 13:00:59.033714 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:00:59.033680    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-9c959559f-2s6h2_42ecc201-f5ef-489e-8eb3-775f91385c17/kube-rbac-proxy-main/0.log"
May 25 13:00:59.056554 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:00:59.056516    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-9c959559f-2s6h2_42ecc201-f5ef-489e-8eb3-775f91385c17/kube-rbac-proxy-self/0.log"
May 25 13:00:59.079466 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:00:59.079424    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-9c959559f-2s6h2_42ecc201-f5ef-489e-8eb3-775f91385c17/openshift-state-metrics/0.log"
May 25 13:00:59.294320 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:00:59.294253    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-7779bbdd78-mmnkt_03d5c580-b34a-4949-ac56-082edfcb0b51/prometheus-operator/0.log"
May 25 13:00:59.312219 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:00:59.312170    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-7779bbdd78-mmnkt_03d5c580-b34a-4949-ac56-082edfcb0b51/kube-rbac-proxy/0.log"
May 25 13:00:59.343688 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:00:59.343657    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6c4bbc67b7-9wkp6_46e2b2e0-2156-41bc-9ce7-3a67562ed0e5/prometheus-operator-admission-webhook/0.log"
May 25 13:00:59.373465 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:00:59.373415    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-99b5fcf8-8wb4v_048a5c2a-b40b-48c8-b889-58593849775f/telemeter-client/0.log"
May 25 13:00:59.399100 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:00:59.399068    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-99b5fcf8-8wb4v_048a5c2a-b40b-48c8-b889-58593849775f/reload/0.log"
May 25 13:00:59.421753 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:00:59.421717    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-99b5fcf8-8wb4v_048a5c2a-b40b-48c8-b889-58593849775f/kube-rbac-proxy/0.log"
May 25 13:00:59.453309 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:00:59.453271    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5b67cb8f7-zkltg_83c8cfdf-2cea-45d9-832b-b4172368763f/thanos-query/0.log"
May 25 13:00:59.475018 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:00:59.474988    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5b67cb8f7-zkltg_83c8cfdf-2cea-45d9-832b-b4172368763f/kube-rbac-proxy-web/0.log"
May 25 13:00:59.498525 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:00:59.498497    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5b67cb8f7-zkltg_83c8cfdf-2cea-45d9-832b-b4172368763f/kube-rbac-proxy/0.log"
May 25 13:00:59.520237 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:00:59.520189    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5b67cb8f7-zkltg_83c8cfdf-2cea-45d9-832b-b4172368763f/prom-label-proxy/0.log"
May 25 13:00:59.541662 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:00:59.541627    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5b67cb8f7-zkltg_83c8cfdf-2cea-45d9-832b-b4172368763f/kube-rbac-proxy-rules/0.log"
May 25 13:00:59.566861 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:00:59.566768    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5b67cb8f7-zkltg_83c8cfdf-2cea-45d9-832b-b4172368763f/kube-rbac-proxy-metrics/0.log"
May 25 13:01:00.699655 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:01:00.699623    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-d4d8f94f9-pdv7d_b7f50355-0097-4b6a-8b1a-01a3f7fd1d26/networking-console-plugin/0.log"
May 25 13:01:01.592581 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:01:01.592548    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-7c46bb4bf6-cpj2j_62d21dab-d5f6-47e9-85b8-f5e8cba9333e/console/0.log"
May 25 13:01:02.037347 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:01:02.037250    2549 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-must-gather-9vddf/perf-node-gather-daemonset-hhz8d"]
May 25 13:01:02.042353 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:01:02.042327    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9vddf/perf-node-gather-daemonset-hhz8d"
May 25 13:01:02.047727 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:01:02.047695    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9vddf/perf-node-gather-daemonset-hhz8d"]
May 25 13:01:02.104463 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:01:02.104418    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkml8\" (UniqueName: \"kubernetes.io/projected/162d549a-c138-4b9c-8c0b-9ded21c40daf-kube-api-access-kkml8\") pod \"perf-node-gather-daemonset-hhz8d\" (UID: \"162d549a-c138-4b9c-8c0b-9ded21c40daf\") " pod="openshift-must-gather-9vddf/perf-node-gather-daemonset-hhz8d"
May 25 13:01:02.104662 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:01:02.104494    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/162d549a-c138-4b9c-8c0b-9ded21c40daf-podres\") pod \"perf-node-gather-daemonset-hhz8d\" (UID: \"162d549a-c138-4b9c-8c0b-9ded21c40daf\") " pod="openshift-must-gather-9vddf/perf-node-gather-daemonset-hhz8d"
May 25 13:01:02.104662 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:01:02.104532    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/162d549a-c138-4b9c-8c0b-9ded21c40daf-proc\") pod \"perf-node-gather-daemonset-hhz8d\" (UID: \"162d549a-c138-4b9c-8c0b-9ded21c40daf\") " pod="openshift-must-gather-9vddf/perf-node-gather-daemonset-hhz8d"
May 25 13:01:02.104662 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:01:02.104620    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/162d549a-c138-4b9c-8c0b-9ded21c40daf-sys\") pod \"perf-node-gather-daemonset-hhz8d\" (UID: \"162d549a-c138-4b9c-8c0b-9ded21c40daf\") " pod="openshift-must-gather-9vddf/perf-node-gather-daemonset-hhz8d"
May 25 13:01:02.104662 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:01:02.104648    2549 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/162d549a-c138-4b9c-8c0b-9ded21c40daf-lib-modules\") pod \"perf-node-gather-daemonset-hhz8d\" (UID: \"162d549a-c138-4b9c-8c0b-9ded21c40daf\") " pod="openshift-must-gather-9vddf/perf-node-gather-daemonset-hhz8d"
May 25 13:01:02.205935 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:01:02.205891    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/162d549a-c138-4b9c-8c0b-9ded21c40daf-podres\") pod \"perf-node-gather-daemonset-hhz8d\" (UID: \"162d549a-c138-4b9c-8c0b-9ded21c40daf\") " pod="openshift-must-gather-9vddf/perf-node-gather-daemonset-hhz8d"
May 25 13:01:02.205935 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:01:02.205941    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/162d549a-c138-4b9c-8c0b-9ded21c40daf-proc\") pod \"perf-node-gather-daemonset-hhz8d\" (UID: \"162d549a-c138-4b9c-8c0b-9ded21c40daf\") " pod="openshift-must-gather-9vddf/perf-node-gather-daemonset-hhz8d"
May 25 13:01:02.206156 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:01:02.205994    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/162d549a-c138-4b9c-8c0b-9ded21c40daf-sys\") pod \"perf-node-gather-daemonset-hhz8d\" (UID: \"162d549a-c138-4b9c-8c0b-9ded21c40daf\") " pod="openshift-must-gather-9vddf/perf-node-gather-daemonset-hhz8d"
May 25 13:01:02.206156 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:01:02.206010    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/162d549a-c138-4b9c-8c0b-9ded21c40daf-lib-modules\") pod \"perf-node-gather-daemonset-hhz8d\" (UID: \"162d549a-c138-4b9c-8c0b-9ded21c40daf\") " pod="openshift-must-gather-9vddf/perf-node-gather-daemonset-hhz8d"
May 25 13:01:02.206156 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:01:02.206055    2549 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-kkml8\" (UniqueName: \"kubernetes.io/projected/162d549a-c138-4b9c-8c0b-9ded21c40daf-kube-api-access-kkml8\") pod \"perf-node-gather-daemonset-hhz8d\" (UID: \"162d549a-c138-4b9c-8c0b-9ded21c40daf\") " pod="openshift-must-gather-9vddf/perf-node-gather-daemonset-hhz8d"
May 25 13:01:02.206156 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:01:02.206076    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/162d549a-c138-4b9c-8c0b-9ded21c40daf-podres\") pod \"perf-node-gather-daemonset-hhz8d\" (UID: \"162d549a-c138-4b9c-8c0b-9ded21c40daf\") " pod="openshift-must-gather-9vddf/perf-node-gather-daemonset-hhz8d"
May 25 13:01:02.206156 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:01:02.206083    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/162d549a-c138-4b9c-8c0b-9ded21c40daf-proc\") pod \"perf-node-gather-daemonset-hhz8d\" (UID: \"162d549a-c138-4b9c-8c0b-9ded21c40daf\") " pod="openshift-must-gather-9vddf/perf-node-gather-daemonset-hhz8d"
May 25 13:01:02.206156 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:01:02.206084    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/162d549a-c138-4b9c-8c0b-9ded21c40daf-sys\") pod \"perf-node-gather-daemonset-hhz8d\" (UID: \"162d549a-c138-4b9c-8c0b-9ded21c40daf\") " pod="openshift-must-gather-9vddf/perf-node-gather-daemonset-hhz8d"
May 25 13:01:02.206405 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:01:02.206167    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/162d549a-c138-4b9c-8c0b-9ded21c40daf-lib-modules\") pod \"perf-node-gather-daemonset-hhz8d\" (UID: \"162d549a-c138-4b9c-8c0b-9ded21c40daf\") " pod="openshift-must-gather-9vddf/perf-node-gather-daemonset-hhz8d"
May 25 13:01:02.215083 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:01:02.215051    2549 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkml8\" (UniqueName: \"kubernetes.io/projected/162d549a-c138-4b9c-8c0b-9ded21c40daf-kube-api-access-kkml8\") pod \"perf-node-gather-daemonset-hhz8d\" (UID: \"162d549a-c138-4b9c-8c0b-9ded21c40daf\") " pod="openshift-must-gather-9vddf/perf-node-gather-daemonset-hhz8d"
May 25 13:01:02.356060 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:01:02.356016    2549 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9vddf/perf-node-gather-daemonset-hhz8d"
May 25 13:01:02.525857 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:01:02.525813    2549 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9vddf/perf-node-gather-daemonset-hhz8d"]
May 25 13:01:02.528858 ip-10-0-142-53 kubenswrapper[2549]: W0525 13:01:02.528825    2549 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod162d549a_c138_4b9c_8c0b_9ded21c40daf.slice/crio-acec34344108f3f9331f20f890ad15205919e0aed0ae66a954084345e3d919b7 WatchSource:0}: Error finding container acec34344108f3f9331f20f890ad15205919e0aed0ae66a954084345e3d919b7: Status 404 returned error can't find the container with id acec34344108f3f9331f20f890ad15205919e0aed0ae66a954084345e3d919b7
May 25 13:01:02.833788 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:01:02.833699    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-ml56g_acccf0fc-6d0f-4793-acb0-69227950e520/dns/0.log"
May 25 13:01:02.858882 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:01:02.858846    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-ml56g_acccf0fc-6d0f-4793-acb0-69227950e520/kube-rbac-proxy/0.log"
May 25 13:01:02.926440 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:01:02.926409    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-dv9dt_61e828df-19f1-4008-bcb9-2c3a879b0178/dns-node-resolver/0.log"
May 25 13:01:03.448602 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:01:03.448571    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-bjpxs_c13dfb5f-85fd-4f8e-9a7f-f432a01720d3/node-ca/0.log"
May 25 13:01:03.524319 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:01:03.524277    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9vddf/perf-node-gather-daemonset-hhz8d" event={"ID":"162d549a-c138-4b9c-8c0b-9ded21c40daf","Type":"ContainerStarted","Data":"962f525e0f3dd1405cc9c082d7a227229e3be49add52da58327271abbb424db1"}
May 25 13:01:03.524512 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:01:03.524339    2549 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9vddf/perf-node-gather-daemonset-hhz8d" event={"ID":"162d549a-c138-4b9c-8c0b-9ded21c40daf","Type":"ContainerStarted","Data":"acec34344108f3f9331f20f890ad15205919e0aed0ae66a954084345e3d919b7"}
May 25 13:01:03.524512 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:01:03.524380    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-9vddf/perf-node-gather-daemonset-hhz8d"
May 25 13:01:03.544474 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:01:03.544392    2549 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-must-gather-9vddf/perf-node-gather-daemonset-hhz8d" podStartSLOduration=1.544370727 podStartE2EDuration="1.544370727s" podCreationTimestamp="2026-05-25 13:01:02 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 13:01:03.54191934 +0000 UTC m=+2867.257734628" watchObservedRunningTime="2026-05-25 13:01:03.544370727 +0000 UTC m=+2867.260186012"
May 25 13:01:04.252841 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:01:04.252739    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5b7954bdbb-7n4rl_90cc9eab-ce97-480c-b971-7487792a2833/router/0.log"
May 25 13:01:04.616389 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:01:04.616340    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-fxlzt_0ea389da-cc7e-4b86-846d-e95f5a2dae7c/serve-healthcheck-canary/0.log"
May 25 13:01:05.048186 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:01:05.048103    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-78bdfb888c-nxz4w_bf795e4e-8902-4aa8-be41-2671139da816/insights-operator/0.log"
May 25 13:01:05.049946 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:01:05.049922    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-78bdfb888c-nxz4w_bf795e4e-8902-4aa8-be41-2671139da816/insights-operator/1.log"
May 25 13:01:05.072222 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:01:05.072162    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-2hm2r_69226669-8b67-4826-97c7-92970d34863a/kube-rbac-proxy/0.log"
May 25 13:01:05.099250 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:01:05.099190    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-2hm2r_69226669-8b67-4826-97c7-92970d34863a/exporter/0.log"
May 25 13:01:05.125489 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:01:05.125442    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-2hm2r_69226669-8b67-4826-97c7-92970d34863a/extractor/0.log"
May 25 13:01:07.222652 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:01:07.222616    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_kserve-controller-manager-6768985688-z8hzg_d1e2aa01-8b67-4aeb-b948-a73028a69d04/manager/0.log"
May 25 13:01:07.269179 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:01:07.269153    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-86f7b4b499-m6k77_a314e08c-f304-41f9-896b-852c9b8ca7ac/server/0.log"
May 25 13:01:09.540497 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:01:09.540468    2549 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-9vddf/perf-node-gather-daemonset-hhz8d"
May 25 13:01:12.088998 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:01:12.088951    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-565644f49-7t27f_1dc72bef-ba17-4ede-88bb-0e1a18c43bbf/kube-storage-version-migrator-operator/1.log"
May 25 13:01:12.090952 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:01:12.090926    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-565644f49-7t27f_1dc72bef-ba17-4ede-88bb-0e1a18c43bbf/kube-storage-version-migrator-operator/0.log"
May 25 13:01:13.039427 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:01:13.039396    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-29966_2d7e9aa3-9ae5-4e93-8e85-11d5efe2dd99/kube-multus/0.log"
May 25 13:01:13.236564 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:01:13.236536    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-h49mm_7153f197-fa4f-4da4-ba82-73f29e4dd2ac/kube-multus-additional-cni-plugins/0.log"
May 25 13:01:13.258189 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:01:13.258140    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-h49mm_7153f197-fa4f-4da4-ba82-73f29e4dd2ac/egress-router-binary-copy/0.log"
May 25 13:01:13.280788 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:01:13.280704    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-h49mm_7153f197-fa4f-4da4-ba82-73f29e4dd2ac/cni-plugins/0.log"
May 25 13:01:13.301293 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:01:13.301209    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-h49mm_7153f197-fa4f-4da4-ba82-73f29e4dd2ac/bond-cni-plugin/0.log"
May 25 13:01:13.326181 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:01:13.326148    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-h49mm_7153f197-fa4f-4da4-ba82-73f29e4dd2ac/routeoverride-cni/0.log"
May 25 13:01:13.349976 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:01:13.349939    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-h49mm_7153f197-fa4f-4da4-ba82-73f29e4dd2ac/whereabouts-cni-bincopy/0.log"
May 25 13:01:13.378564 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:01:13.378536    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-h49mm_7153f197-fa4f-4da4-ba82-73f29e4dd2ac/whereabouts-cni/0.log"
May 25 13:01:13.729084 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:01:13.729049    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-zf62d_1124f1f1-fdc7-4a81-b28a-620db95e2ff6/network-metrics-daemon/0.log"
May 25 13:01:13.747742 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:01:13.747710    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-zf62d_1124f1f1-fdc7-4a81-b28a-620db95e2ff6/kube-rbac-proxy/0.log"
May 25 13:01:14.506615 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:01:14.506585    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-79j85_0b693280-ce91-46e7-b8c8-37d81a9b8fe1/ovn-controller/0.log"
May 25 13:01:14.554452 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:01:14.554421    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-79j85_0b693280-ce91-46e7-b8c8-37d81a9b8fe1/ovn-acl-logging/0.log"
May 25 13:01:14.574990 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:01:14.574966    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-79j85_0b693280-ce91-46e7-b8c8-37d81a9b8fe1/kube-rbac-proxy-node/0.log"
May 25 13:01:14.596581 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:01:14.596552    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-79j85_0b693280-ce91-46e7-b8c8-37d81a9b8fe1/kube-rbac-proxy-ovn-metrics/0.log"
May 25 13:01:14.614210 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:01:14.614180    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-79j85_0b693280-ce91-46e7-b8c8-37d81a9b8fe1/northd/0.log"
May 25 13:01:14.634247 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:01:14.634228    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-79j85_0b693280-ce91-46e7-b8c8-37d81a9b8fe1/nbdb/0.log"
May 25 13:01:14.656727 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:01:14.656704    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-79j85_0b693280-ce91-46e7-b8c8-37d81a9b8fe1/sbdb/0.log"
May 25 13:01:14.865158 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:01:14.865122    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-79j85_0b693280-ce91-46e7-b8c8-37d81a9b8fe1/ovnkube-controller/0.log"
May 25 13:01:16.509635 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:01:16.509602    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-tp68k_d37b4fc9-9e8b-4e07-9672-b6bffc0f46d8/network-check-target-container/0.log"
May 25 13:01:17.473569 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:01:17.473541    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-g4949_69c409ea-f028-429e-8a53-0f25010483bd/iptables-alerter/0.log"
May 25 13:01:18.204777 ip-10-0-142-53 kubenswrapper[2549]: I0525 13:01:18.204746    2549 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-xp8d2_c66cda37-4f3e-4058-a58c-1b168db3edd7/tuned/0.log"