May 26 15:58:02.699026 ip-10-0-134-19 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
May 26 15:58:02.699039 ip-10-0-134-19 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
May 26 15:58:02.699049 ip-10-0-134-19 systemd[1]: kubelet.service: Failed with result 'resources'.
May 26 15:58:02.699371 ip-10-0-134-19 systemd[1]: Failed to start Kubernetes Kubelet.
May 26 15:58:12.834002 ip-10-0-134-19 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
May 26 15:58:12.834022 ip-10-0-134-19 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 44aa37251d3749b8ad4d1d6a05fbe638 --
May 26 16:00:38.053372 ip-10-0-134-19 systemd[1]: Starting Kubernetes Kubelet...
May 26 16:00:38.493815 ip-10-0-134-19 kubenswrapper[2556]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
May 26 16:00:38.493815 ip-10-0-134-19 kubenswrapper[2556]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
May 26 16:00:38.493815 ip-10-0-134-19 kubenswrapper[2556]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
May 26 16:00:38.493815 ip-10-0-134-19 kubenswrapper[2556]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
May 26 16:00:38.493815 ip-10-0-134-19 kubenswrapper[2556]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
May 26 16:00:38.494848 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.494768    2556 server.go:212] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
May 26 16:00:38.501424 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.501403    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 26 16:00:38.501424 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.501421    2556 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 26 16:00:38.501424 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.501425    2556 feature_gate.go:328] unrecognized feature gate: NewOLM
May 26 16:00:38.501524 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.501430    2556 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 26 16:00:38.501524 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.501434    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 26 16:00:38.501524 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.501437    2556 feature_gate.go:328] unrecognized feature gate: DualReplica
May 26 16:00:38.501524 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.501440    2556 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 26 16:00:38.501524 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.501443    2556 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 26 16:00:38.501524 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.501445    2556 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 26 16:00:38.501524 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.501448    2556 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 26 16:00:38.501524 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.501451    2556 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 26 16:00:38.501524 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.501454    2556 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 26 16:00:38.501524 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.501464    2556 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 26 16:00:38.501524 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.501467    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 26 16:00:38.501524 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.501470    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 26 16:00:38.501524 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.501472    2556 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 26 16:00:38.501524 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.501475    2556 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 26 16:00:38.501524 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.501478    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 26 16:00:38.501524 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.501480    2556 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 26 16:00:38.501524 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.501483    2556 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 26 16:00:38.501524 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.501485    2556 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 26 16:00:38.501524 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.501488    2556 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 26 16:00:38.502066 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.501491    2556 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 26 16:00:38.502066 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.501494    2556 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 26 16:00:38.502066 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.501496    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 26 16:00:38.502066 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.501499    2556 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 26 16:00:38.502066 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.501501    2556 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 26 16:00:38.502066 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.501504    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 26 16:00:38.502066 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.501507    2556 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 26 16:00:38.502066 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.501509    2556 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 26 16:00:38.502066 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.501512    2556 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 26 16:00:38.502066 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.501515    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 26 16:00:38.502066 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.501517    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 26 16:00:38.502066 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.501520    2556 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 26 16:00:38.502066 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.501523    2556 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 26 16:00:38.502066 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.501525    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 26 16:00:38.502066 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.501529    2556 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 26 16:00:38.502066 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.501532    2556 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 26 16:00:38.502066 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.501534    2556 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 26 16:00:38.502066 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.501539    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 26 16:00:38.502066 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.501543    2556 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 26 16:00:38.502780 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.501547    2556 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 26 16:00:38.502780 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.501550    2556 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 26 16:00:38.502780 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.501553    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 26 16:00:38.502780 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.501556    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 26 16:00:38.502780 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.501559    2556 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 26 16:00:38.502780 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.501561    2556 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 26 16:00:38.502780 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.501564    2556 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 26 16:00:38.502780 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.501566    2556 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 26 16:00:38.502780 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.501569    2556 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 26 16:00:38.502780 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.501571    2556 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 26 16:00:38.502780 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.501574    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 26 16:00:38.502780 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.501577    2556 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 26 16:00:38.502780 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.501579    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 26 16:00:38.502780 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.501581    2556 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 26 16:00:38.502780 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.501585    2556 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 26 16:00:38.502780 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.501587    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 26 16:00:38.502780 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.501590    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 26 16:00:38.502780 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.501594    2556 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 26 16:00:38.502780 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.501598    2556 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 26 16:00:38.502780 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.501601    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 26 16:00:38.503306 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.501604    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 26 16:00:38.503306 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.501606    2556 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 26 16:00:38.503306 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.501609    2556 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 26 16:00:38.503306 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.501611    2556 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 26 16:00:38.503306 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.501614    2556 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 26 16:00:38.503306 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.501617    2556 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 26 16:00:38.503306 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.501619    2556 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 26 16:00:38.503306 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.501622    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 26 16:00:38.503306 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.501624    2556 feature_gate.go:328] unrecognized feature gate: Example
May 26 16:00:38.503306 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.501627    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 26 16:00:38.503306 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.501629    2556 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 26 16:00:38.503306 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.501632    2556 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 26 16:00:38.503306 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.501635    2556 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 26 16:00:38.503306 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.501637    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 26 16:00:38.503306 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.501639    2556 feature_gate.go:328] unrecognized feature gate: Example2
May 26 16:00:38.503306 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.501642    2556 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 26 16:00:38.503306 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.501644    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 26 16:00:38.503306 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.501647    2556 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 26 16:00:38.503306 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.501650    2556 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 26 16:00:38.503306 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.501652    2556 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 26 16:00:38.503790 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.501655    2556 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 26 16:00:38.503790 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.501658    2556 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 26 16:00:38.503790 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.501660    2556 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 26 16:00:38.503790 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.501663    2556 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 26 16:00:38.503790 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.501665    2556 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 26 16:00:38.503790 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.503623    2556 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 26 16:00:38.503790 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.503636    2556 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 26 16:00:38.503790 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.503641    2556 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 26 16:00:38.503790 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.503644    2556 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 26 16:00:38.503790 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.503647    2556 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 26 16:00:38.503790 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.503650    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 26 16:00:38.503790 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.503653    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 26 16:00:38.503790 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.503656    2556 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 26 16:00:38.503790 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.503659    2556 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 26 16:00:38.503790 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.503662    2556 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 26 16:00:38.503790 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.503664    2556 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 26 16:00:38.503790 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.503667    2556 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 26 16:00:38.503790 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.503670    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 26 16:00:38.503790 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.503672    2556 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 26 16:00:38.503790 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.503675    2556 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 26 16:00:38.504295 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.503678    2556 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 26 16:00:38.504295 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.503680    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 26 16:00:38.504295 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.503683    2556 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 26 16:00:38.504295 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.503685    2556 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 26 16:00:38.504295 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.503688    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 26 16:00:38.504295 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.503690    2556 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 26 16:00:38.504295 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.503692    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 26 16:00:38.504295 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.503695    2556 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 26 16:00:38.504295 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.503697    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 26 16:00:38.504295 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.503700    2556 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 26 16:00:38.504295 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.503702    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 26 16:00:38.504295 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.503705    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 26 16:00:38.504295 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.503707    2556 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 26 16:00:38.504295 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.503711    2556 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 26 16:00:38.504295 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.503715    2556 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 26 16:00:38.504295 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.503717    2556 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 26 16:00:38.504295 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.503720    2556 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 26 16:00:38.504295 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.503725    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 26 16:00:38.504295 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.503728    2556 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 26 16:00:38.504295 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.503731    2556 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 26 16:00:38.504802 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.503733    2556 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 26 16:00:38.504802 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.503736    2556 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 26 16:00:38.504802 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.503738    2556 feature_gate.go:328] unrecognized feature gate: NewOLM
May 26 16:00:38.504802 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.503740    2556 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 26 16:00:38.504802 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.503743    2556 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 26 16:00:38.504802 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.503745    2556 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 26 16:00:38.504802 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.503749    2556 feature_gate.go:328] unrecognized feature gate: Example2
May 26 16:00:38.504802 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.503751    2556 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 26 16:00:38.504802 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.503754    2556 feature_gate.go:328] unrecognized feature gate: DualReplica
May 26 16:00:38.504802 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.503757    2556 feature_gate.go:328] unrecognized feature gate: Example
May 26 16:00:38.504802 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.503759    2556 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 26 16:00:38.504802 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.503762    2556 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 26 16:00:38.504802 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.503764    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 26 16:00:38.504802 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.503767    2556 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 26 16:00:38.504802 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.503770    2556 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 26 16:00:38.504802 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.503772    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 26 16:00:38.504802 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.503774    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 26 16:00:38.504802 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.503777    2556 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 26 16:00:38.504802 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.503780    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 26 16:00:38.504802 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.503783    2556 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 26 16:00:38.505310 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.503787    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 26 16:00:38.505310 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.503790    2556 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 26 16:00:38.505310 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.503792    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 26 16:00:38.505310 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.503795    2556 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 26 16:00:38.505310 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.503799    2556 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 26 16:00:38.505310 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.503802    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 26 16:00:38.505310 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.503805    2556 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 26 16:00:38.505310 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.503808    2556 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 26 16:00:38.505310 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.503810    2556 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 26 16:00:38.505310 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.503814    2556 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 26 16:00:38.505310 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.503817    2556 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 26 16:00:38.505310 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.503819    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 26 16:00:38.505310 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.503822    2556 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 26 16:00:38.505310 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.503824    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 26 16:00:38.505310 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.503827    2556 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 26 16:00:38.505310 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.503830    2556 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 26 16:00:38.505310 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.503832    2556 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 26 16:00:38.505310 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.503835    2556 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 26 16:00:38.505310 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.503837    2556 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 26 16:00:38.505773 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.503839    2556 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 26 16:00:38.505773 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.503842    2556 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 26 16:00:38.505773 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.503844    2556 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 26 16:00:38.505773 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.503846    2556 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 26 16:00:38.505773 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.503849    2556 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 26 16:00:38.505773 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.503851    2556 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 26 16:00:38.505773 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.503854    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 26 16:00:38.505773 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.503856    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 26 16:00:38.505773 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.503859    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 26 16:00:38.505773 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.503861    2556 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 26 16:00:38.505773 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.503864    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 26 16:00:38.505773 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.503867    2556 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 26 16:00:38.505773 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.504665    2556 flags.go:64] FLAG: --address="0.0.0.0"
May 26 16:00:38.505773 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.504674    2556 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
May 26 16:00:38.505773 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.504681    2556 flags.go:64] FLAG: --anonymous-auth="true"
May 26 16:00:38.505773 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.504686    2556 flags.go:64] FLAG: --application-metrics-count-limit="100"
May 26 16:00:38.505773 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.504691    2556 flags.go:64] FLAG: --authentication-token-webhook="false"
May 26 16:00:38.505773 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.504694    2556 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
May 26 16:00:38.505773 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.504700    2556 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
May 26 16:00:38.505773 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.504704    2556 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
May 26 16:00:38.505773 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.504707    2556 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
May 26 16:00:38.506328 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.504711    2556 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
May 26 16:00:38.506328 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.504714    2556 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
May 26 16:00:38.506328 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.504718    2556 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
May 26 16:00:38.506328 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.504721    2556 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
May 26 16:00:38.506328 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.504724    2556 flags.go:64] FLAG: --cgroup-root=""
May 26 16:00:38.506328 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.504727    2556 flags.go:64] FLAG: --cgroups-per-qos="true"
May 26 16:00:38.506328 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.504730    2556 flags.go:64] FLAG: --client-ca-file=""
May 26 16:00:38.506328 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.504733    2556 flags.go:64] FLAG: --cloud-config=""
May 26 16:00:38.506328 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.504735    2556 flags.go:64] FLAG: --cloud-provider="external"
May 26 16:00:38.506328 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.504738    2556 flags.go:64] FLAG: --cluster-dns="[]"
May 26 16:00:38.506328 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.504744    2556 flags.go:64] FLAG: --cluster-domain=""
May 26 16:00:38.506328 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.504747    2556 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
May 26 16:00:38.506328 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.504750    2556 flags.go:64] FLAG: --config-dir=""
May 26 16:00:38.506328 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.504753    2556 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
May 26 16:00:38.506328 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.504756    2556 flags.go:64] FLAG: --container-log-max-files="5"
May 26 16:00:38.506328 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.504760    2556 flags.go:64] FLAG: --container-log-max-size="10Mi"
May 26 16:00:38.506328 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.504763    2556 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
May 26 16:00:38.506328 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.504767    2556 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
May 26 16:00:38.506328 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.504770    2556 flags.go:64] FLAG: --containerd-namespace="k8s.io"
May 26 16:00:38.506328 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.504773    2556 flags.go:64] FLAG: --contention-profiling="false"
May 26 16:00:38.506328 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.504776    2556 flags.go:64] FLAG: --cpu-cfs-quota="true"
May 26 16:00:38.506328 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.504778    2556 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
May 26 16:00:38.506328 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.504782    2556 flags.go:64] FLAG: --cpu-manager-policy="none"
May 26 16:00:38.506328 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.504785    2556 flags.go:64] FLAG: --cpu-manager-policy-options=""
May 26 16:00:38.506328 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.504789    2556 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
May 26 16:00:38.506919 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.504792    2556 flags.go:64] FLAG: --enable-controller-attach-detach="true"
May 26 16:00:38.506919 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.504794    2556 flags.go:64] FLAG: --enable-debugging-handlers="true"
May 26 16:00:38.506919 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.504797    2556 flags.go:64] FLAG: --enable-load-reader="false"
May 26 16:00:38.506919 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.504800    2556 flags.go:64] FLAG: --enable-server="true"
May 26 16:00:38.506919 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.504804    2556 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
May 26 16:00:38.506919 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.504808    2556 flags.go:64] FLAG: --event-burst="100"
May 26 16:00:38.506919 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.504812    2556 flags.go:64] FLAG: --event-qps="50"
May 26 16:00:38.506919 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.504815    2556 flags.go:64] FLAG: --event-storage-age-limit="default=0"
May 26 16:00:38.506919 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.504818    2556 flags.go:64] FLAG: --event-storage-event-limit="default=0"
May 26 16:00:38.506919 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.504821    2556 flags.go:64] FLAG: --eviction-hard=""
May 26 16:00:38.506919 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.504825    2556 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
May 26 16:00:38.506919 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.504828    2556 flags.go:64] FLAG: --eviction-minimum-reclaim=""
May 26 16:00:38.506919 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.504831    2556 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
May 26 16:00:38.506919 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.504834    2556 flags.go:64] FLAG: --eviction-soft=""
May 26 16:00:38.506919 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.504837    2556 flags.go:64] FLAG: --eviction-soft-grace-period=""
May 26 16:00:38.506919 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.504840    2556 flags.go:64] FLAG: --exit-on-lock-contention="false"
May 26 16:00:38.506919 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.504843    2556 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
May 26 16:00:38.506919 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.504846    2556 flags.go:64] FLAG: --experimental-mounter-path=""
May 26 16:00:38.506919 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.504849    2556 flags.go:64] FLAG: --fail-cgroupv1="false"
May 26 16:00:38.506919 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.504852    2556 flags.go:64] FLAG: --fail-swap-on="true"
May 26 16:00:38.506919 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.504855    2556 flags.go:64] FLAG: --feature-gates=""
May 26 16:00:38.506919 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.504859    2556 flags.go:64] FLAG: --file-check-frequency="20s"
May 26 16:00:38.506919 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.504862    2556 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
May 26 16:00:38.506919 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.504866    2556 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
May 26 16:00:38.506919 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.504869    2556 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
May 26 16:00:38.507562 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.504872    2556 flags.go:64] FLAG: --healthz-port="10248"
May 26 16:00:38.507562 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.504875    2556 flags.go:64] FLAG: --help="false"
May 26 16:00:38.507562 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.504878    2556 flags.go:64] FLAG: --hostname-override="ip-10-0-134-19.ec2.internal"
May 26 16:00:38.507562 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.504882    2556 flags.go:64] FLAG: --housekeeping-interval="10s"
May 26 16:00:38.507562 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.504885    2556 flags.go:64] FLAG: --http-check-frequency="20s"
May 26 16:00:38.507562 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.504888    2556 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
May 26 16:00:38.507562 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.504891    2556 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
May 26 16:00:38.507562 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.504895    2556 flags.go:64] FLAG: --image-gc-high-threshold="85"
May 26 16:00:38.507562 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.504898    2556 flags.go:64] FLAG: --image-gc-low-threshold="80"
May 26 16:00:38.507562 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.504901    2556 flags.go:64] FLAG: --image-service-endpoint=""
May 26 16:00:38.507562 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.504904    2556 flags.go:64] FLAG: --kernel-memcg-notification="false"
May 26 16:00:38.507562 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.504908    2556 flags.go:64] FLAG: --kube-api-burst="100"
May 26 16:00:38.507562 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.504911    2556 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
May 26 16:00:38.507562 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.504914    2556 flags.go:64] FLAG: --kube-api-qps="50"
May 26 16:00:38.507562 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.504917    2556 flags.go:64] FLAG: --kube-reserved=""
May 26 16:00:38.507562 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.504919    2556 flags.go:64] FLAG: --kube-reserved-cgroup=""
May 26 16:00:38.507562 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.504923    2556 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
May 26 16:00:38.507562 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.504926    2556 flags.go:64] FLAG: --kubelet-cgroups=""
May 26 16:00:38.507562 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.504928    2556 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
May 26 16:00:38.507562 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.504931    2556 flags.go:64] FLAG: --lock-file=""
May 26 16:00:38.507562 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.504934    2556 flags.go:64] FLAG: --log-cadvisor-usage="false"
May 26 16:00:38.507562 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.504936    2556 flags.go:64] FLAG: --log-flush-frequency="5s"
May 26 16:00:38.507562 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.504940    2556 flags.go:64] FLAG: --log-json-info-buffer-size="0"
May 26 16:00:38.507562 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.504945    2556 flags.go:64] FLAG: --log-json-split-stream="false"
May 26 16:00:38.508194 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.504948    2556 flags.go:64] FLAG: --log-text-info-buffer-size="0"
May 26 16:00:38.508194 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.504951    2556 flags.go:64] FLAG: --log-text-split-stream="false"
May 26 16:00:38.508194 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.504954    2556 flags.go:64] FLAG: --logging-format="text"
May 26 16:00:38.508194 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.504957    2556 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
May 26 16:00:38.508194 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.504960    2556 flags.go:64] FLAG: --make-iptables-util-chains="true"
May 26 16:00:38.508194 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.504963    2556 flags.go:64] FLAG: --manifest-url=""
May 26 16:00:38.508194 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.504966    2556 flags.go:64] FLAG: --manifest-url-header=""
May 26 16:00:38.508194 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.504975    2556 flags.go:64] FLAG: --max-housekeeping-interval="15s"
May 26 16:00:38.508194 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.504978    2556 flags.go:64] FLAG: --max-open-files="1000000"
May 26 16:00:38.508194 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.504983    2556 flags.go:64] FLAG: --max-pods="110"
May 26 16:00:38.508194 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.504986    2556 flags.go:64] FLAG: --maximum-dead-containers="-1"
May 26 16:00:38.508194 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.504989    2556 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
May 26 16:00:38.508194 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.504992    2556 flags.go:64] FLAG: --memory-manager-policy="None"
May 26 16:00:38.508194 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.504994    2556 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
May 26 16:00:38.508194 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.504997    2556 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
May 26 16:00:38.508194 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.505000    2556 flags.go:64] FLAG: --node-ip="0.0.0.0"
May 26 16:00:38.508194 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.505003    2556 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
May 26 16:00:38.508194 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.505010    2556 flags.go:64] FLAG: --node-status-max-images="50"
May 26 16:00:38.508194 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.505014    2556 flags.go:64] FLAG: --node-status-update-frequency="10s"
May 26 16:00:38.508194 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.505017    2556 flags.go:64] FLAG: --oom-score-adj="-999"
May 26 16:00:38.508194 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.505020    2556 flags.go:64] FLAG: --pod-cidr=""
May 26 16:00:38.508194 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.505023    2556 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:afe88671bf163c42b467d159dc6f0b784dbe28b1ed44e171edc9c06fd99cc6fc"
May 26 16:00:38.508194 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.505028    2556 flags.go:64] FLAG: --pod-manifest-path=""
May 26 16:00:38.508755 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.505031    2556 flags.go:64] FLAG: --pod-max-pids="-1"
May 26 16:00:38.508755 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.505034    2556 flags.go:64] FLAG: --pods-per-core="0"
May 26 16:00:38.508755 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.505037    2556 flags.go:64] FLAG: --port="10250"
May 26 16:00:38.508755 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.505040    2556 flags.go:64] FLAG: --protect-kernel-defaults="false"
May 26 16:00:38.508755 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.505043    2556 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0de5d7f74ea7e4c41"
May 26 16:00:38.508755 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.505046    2556 flags.go:64] FLAG: --qos-reserved=""
May 26 16:00:38.508755 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.505049    2556 flags.go:64] FLAG: --read-only-port="10255"
May 26 16:00:38.508755 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.505056    2556 flags.go:64] FLAG: --register-node="true"
May 26 16:00:38.508755 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.505059    2556 flags.go:64] FLAG: --register-schedulable="true"
May 26 16:00:38.508755 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.505061    2556 flags.go:64] FLAG: --register-with-taints=""
May 26 16:00:38.508755 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.505065    2556 flags.go:64] FLAG: --registry-burst="10"
May 26 16:00:38.508755 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.505068    2556 flags.go:64] FLAG: --registry-qps="5"
May 26 16:00:38.508755 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.505071    2556 flags.go:64] FLAG: --reserved-cpus=""
May 26 16:00:38.508755 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.505074    2556 flags.go:64] FLAG: --reserved-memory=""
May 26 16:00:38.508755 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.505078    2556 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
May 26 16:00:38.508755 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.505081    2556 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
May 26 16:00:38.508755 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.505099    2556 flags.go:64] FLAG: --rotate-certificates="false"
May 26 16:00:38.508755 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.505103    2556 flags.go:64] FLAG: --rotate-server-certificates="false"
May 26 16:00:38.508755 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.505106    2556 flags.go:64] FLAG: --runonce="false"
May 26 16:00:38.508755 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.505109    2556 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
May 26 16:00:38.508755 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.505113    2556 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
May 26 16:00:38.508755 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.505115    2556 flags.go:64] FLAG: --seccomp-default="false"
May 26 16:00:38.508755 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.505119    2556 flags.go:64] FLAG: --serialize-image-pulls="true"
May 26 16:00:38.508755 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.505122    2556 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
May 26 16:00:38.508755 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.505125    2556 flags.go:64] FLAG: --storage-driver-db="cadvisor"
May 26 16:00:38.508755 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.505128    2556 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
May 26 16:00:38.509438 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.505131    2556 flags.go:64] FLAG: --storage-driver-password="root"
May 26 16:00:38.509438 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.505134    2556 flags.go:64] FLAG: --storage-driver-secure="false"
May 26 16:00:38.509438 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.505137    2556 flags.go:64] FLAG: --storage-driver-table="stats"
May 26 16:00:38.509438 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.505139    2556 flags.go:64] FLAG: --storage-driver-user="root"
May 26 16:00:38.509438 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.505142    2556 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
May 26 16:00:38.509438 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.505145    2556 flags.go:64] FLAG: --sync-frequency="1m0s"
May 26 16:00:38.509438 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.505148    2556 flags.go:64] FLAG: --system-cgroups=""
May 26 16:00:38.509438 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.505150    2556 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
May 26 16:00:38.509438 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.505156    2556 flags.go:64] FLAG: --system-reserved-cgroup=""
May 26 16:00:38.509438 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.505159    2556 flags.go:64] FLAG: --tls-cert-file=""
May 26 16:00:38.509438 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.505162    2556 flags.go:64] FLAG: --tls-cipher-suites="[]"
May 26 16:00:38.509438 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.505166    2556 flags.go:64] FLAG: --tls-min-version=""
May 26 16:00:38.509438 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.505169    2556 flags.go:64] FLAG: --tls-private-key-file=""
May 26 16:00:38.509438 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.505174    2556 flags.go:64] FLAG: --topology-manager-policy="none"
May 26 16:00:38.509438 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.505177    2556 flags.go:64] FLAG: --topology-manager-policy-options=""
May 26 16:00:38.509438 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.505180    2556 flags.go:64] FLAG: --topology-manager-scope="container"
May 26 16:00:38.509438 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.505183    2556 flags.go:64] FLAG: --v="2"
May 26 16:00:38.509438 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.505188    2556 flags.go:64] FLAG: --version="false"
May 26 16:00:38.509438 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.505192    2556 flags.go:64] FLAG: --vmodule=""
May 26 16:00:38.509438 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.505196    2556 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
May 26 16:00:38.509438 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.505199    2556 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
May 26 16:00:38.509438 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.505292    2556 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 26 16:00:38.509438 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.505296    2556 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 26 16:00:38.509438 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.505300    2556 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 26 16:00:38.510114 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.505304    2556 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 26 16:00:38.510114 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.505310    2556 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 26 16:00:38.510114 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.505313    2556 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 26 16:00:38.510114 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.505316    2556 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 26 16:00:38.510114 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.505319    2556 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 26 16:00:38.510114 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.505322    2556 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 26 16:00:38.510114 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.505325    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 26 16:00:38.510114 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.505327    2556 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 26 16:00:38.510114 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.505330    2556 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 26 16:00:38.510114 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.505333    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 26 16:00:38.510114 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.505335    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 26 16:00:38.510114 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.505338    2556 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 26 16:00:38.510114 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.505341    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 26 16:00:38.510114 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.505344    2556 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 26 16:00:38.510114 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.505346    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 26 16:00:38.510114 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.505350    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 26 16:00:38.510114 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.505352    2556 feature_gate.go:328] unrecognized feature gate: NewOLM
May 26 16:00:38.510114 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.505355    2556 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 26 16:00:38.510114 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.505357    2556 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 26 16:00:38.510597 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.505360    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 26 16:00:38.510597 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.505363    2556 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 26 16:00:38.510597 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.505366    2556 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 26 16:00:38.510597 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.505368    2556 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 26 16:00:38.510597 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.505371    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 26 16:00:38.510597 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.505374    2556 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 26 16:00:38.510597 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.505376    2556 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 26 16:00:38.510597 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.505379    2556 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 26 16:00:38.510597 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.505381    2556 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 26 16:00:38.510597 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.505384    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 26 16:00:38.510597 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.505387    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 26 16:00:38.510597 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.505389    2556 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 26 16:00:38.510597 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.505392    2556 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 26 16:00:38.510597 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.505394    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 26 16:00:38.510597 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.505398    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 26 16:00:38.510597 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.505402    2556 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 26 16:00:38.510597 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.505405    2556 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 26 16:00:38.510597 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.505408    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 26 16:00:38.510597 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.505410    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 26 16:00:38.511064 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.505413    2556 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 26 16:00:38.511064 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.505415    2556 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 26 16:00:38.511064 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.505417    2556 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 26 16:00:38.511064 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.505420    2556 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 26 16:00:38.511064 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.505422    2556 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 26 16:00:38.511064 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.505424    2556 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 26 16:00:38.511064 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.505427    2556 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 26 16:00:38.511064 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.505429    2556 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 26 16:00:38.511064 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.505432    2556 feature_gate.go:328] unrecognized feature gate: Example
May 26 16:00:38.511064 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.505434    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 26 16:00:38.511064 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.505437    2556 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 26 16:00:38.511064 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.505439    2556 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 26 16:00:38.511064 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.505442    2556 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 26 16:00:38.511064 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.505444    2556 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 26 16:00:38.511064 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.505447    2556 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 26 16:00:38.511064 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.505450    2556 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 26 16:00:38.511064 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.505453    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 26 16:00:38.511064 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.505455    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 26 16:00:38.511064 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.505458    2556 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 26 16:00:38.511064 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.505460    2556 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 26 16:00:38.511580 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.505463    2556 feature_gate.go:328] unrecognized feature gate: Example2
May 26 16:00:38.511580 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.505465    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 26 16:00:38.511580 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.505468    2556 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 26 16:00:38.511580 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.505470    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 26 16:00:38.511580 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.505474    2556 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 26 16:00:38.511580 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.505477    2556 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 26 16:00:38.511580 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.505480    2556 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 26 16:00:38.511580 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.505483    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 26 16:00:38.511580 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.505487    2556 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 26 16:00:38.511580 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.505489    2556 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 26 16:00:38.511580 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.505492    2556 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 26 16:00:38.511580 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.505494    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 26 16:00:38.511580 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.505496    2556 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 26 16:00:38.511580 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.505499    2556 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 26 16:00:38.511580 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.505501    2556 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 26 16:00:38.511580 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.505504    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 26 16:00:38.511580 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.505506    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 26 16:00:38.511580 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.505509    2556 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 26 16:00:38.511580 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.505511    2556 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 26 16:00:38.511580 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.505514    2556 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 26 16:00:38.512070 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.505516    2556 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 26 16:00:38.512070 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.505518    2556 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 26 16:00:38.512070 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.505521    2556 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 26 16:00:38.512070 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.505523    2556 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 26 16:00:38.512070 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.505526    2556 feature_gate.go:328] unrecognized feature gate: DualReplica
May 26 16:00:38.512070 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.506301    2556 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]}
May 26 16:00:38.513053 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.513031    2556 server.go:530] "Kubelet version" kubeletVersion="v1.33.11"
May 26 16:00:38.513105 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.513055    2556 server.go:532] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
May 26 16:00:38.513134 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.513126    2556 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 26 16:00:38.513134 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.513132    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 26 16:00:38.513183 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.513136    2556 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 26 16:00:38.513183 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.513139    2556 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 26 16:00:38.513183 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.513143    2556 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 26 16:00:38.513183 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.513145    2556 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 26 16:00:38.513183 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.513148    2556 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 26 16:00:38.513183 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.513151    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 26 16:00:38.513183 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.513154    2556 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 26 16:00:38.513183 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.513156    2556 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 26 16:00:38.513183 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.513159    2556 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 26 16:00:38.513183 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.513162    2556 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 26 16:00:38.513183 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.513164    2556 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 26 16:00:38.513183 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.513167    2556 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 26 16:00:38.513183 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.513169    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 26 16:00:38.513183 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.513172    2556 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 26 16:00:38.513183 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.513175    2556 feature_gate.go:328] unrecognized feature gate: Example2
May 26 16:00:38.513183 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.513177    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 26 16:00:38.513183 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.513181    2556 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 26 16:00:38.513183 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.513183    2556 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 26 16:00:38.513183 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.513186    2556 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 26 16:00:38.513677 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.513189    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 26 16:00:38.513677 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.513192    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 26 16:00:38.513677 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.513195    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 26 16:00:38.513677 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.513198    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 26 16:00:38.513677 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.513200    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 26 16:00:38.513677 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.513203    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 26 16:00:38.513677 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.513206    2556 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 26 16:00:38.513677 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.513208    2556 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 26 16:00:38.513677 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.513211    2556 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 26 16:00:38.513677 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.513213    2556 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 26 16:00:38.513677 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.513217    2556 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 26 16:00:38.513677 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.513219    2556 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 26 16:00:38.513677 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.513222    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 26 16:00:38.513677 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.513225    2556 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 26 16:00:38.513677 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.513228    2556 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 26 16:00:38.513677 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.513230    2556 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 26 16:00:38.513677 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.513233    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 26 16:00:38.513677 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.513235    2556 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 26 16:00:38.513677 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.513238    2556 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 26 16:00:38.513677 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.513240    2556 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 26 16:00:38.514211 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.513243    2556 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 26 16:00:38.514211 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.513245    2556 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 26 16:00:38.514211 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.513248    2556 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 26 16:00:38.514211 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.513250    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 26 16:00:38.514211 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.513253    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 26 16:00:38.514211 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.513255    2556 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 26 16:00:38.514211 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.513258    2556 feature_gate.go:328] unrecognized feature gate: NewOLM
May 26 16:00:38.514211 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.513260    2556 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 26 16:00:38.514211 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.513263    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 26 16:00:38.514211 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.513266    2556 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 26 16:00:38.514211 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.513268    2556 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 26 16:00:38.514211 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.513271    2556 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 26 16:00:38.514211 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.513273    2556 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 26 16:00:38.514211 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.513276    2556 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 26 16:00:38.514211 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.513279    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 26 16:00:38.514211 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.513281    2556 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 26 16:00:38.514211 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.513284    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 26 16:00:38.514211 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.513287    2556 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 26 16:00:38.514211 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.513289    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 26 16:00:38.514211 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.513292    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 26 16:00:38.514713 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.513294    2556 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 26 16:00:38.514713 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.513297    2556 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 26 16:00:38.514713 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.513299    2556 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 26 16:00:38.514713 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.513303    2556 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 26 16:00:38.514713 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.513306    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 26 16:00:38.514713 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.513308    2556 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 26 16:00:38.514713 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.513311    2556 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 26 16:00:38.514713 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.513313    2556 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 26 16:00:38.514713 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.513316    2556 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 26 16:00:38.514713 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.513318    2556 feature_gate.go:328] unrecognized feature gate: DualReplica
May 26 16:00:38.514713 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.513321    2556 feature_gate.go:328] unrecognized feature gate: Example
May 26 16:00:38.514713 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.513324    2556 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 26 16:00:38.514713 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.513326    2556 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 26 16:00:38.514713 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.513329    2556 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 26 16:00:38.514713 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.513333    2556 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 26 16:00:38.514713 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.513337    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 26 16:00:38.514713 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.513340    2556 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 26 16:00:38.514713 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.513343    2556 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 26 16:00:38.514713 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.513346    2556 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 26 16:00:38.514713 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.513348    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 26 16:00:38.515221 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.513351    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 26 16:00:38.515221 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.513354    2556 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 26 16:00:38.515221 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.513358    2556 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 26 16:00:38.515221 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.513362    2556 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 26 16:00:38.515221 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.513366    2556 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 26 16:00:38.515221 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.513371    2556 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]}
May 26 16:00:38.515221 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.513466    2556 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 26 16:00:38.515221 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.513471    2556 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 26 16:00:38.515221 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.513474    2556 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 26 16:00:38.515221 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.513477    2556 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 26 16:00:38.515221 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.513480    2556 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 26 16:00:38.515221 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.513483    2556 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 26 16:00:38.515221 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.513486    2556 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 26 16:00:38.515221 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.513488    2556 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 26 16:00:38.515221 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.513491    2556 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 26 16:00:38.515599 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.513494    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 26 16:00:38.515599 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.513497    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 26 16:00:38.515599 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.513499    2556 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 26 16:00:38.515599 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.513502    2556 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 26 16:00:38.515599 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.513504    2556 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 26 16:00:38.515599 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.513507    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 26 16:00:38.515599 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.513509    2556 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 26 16:00:38.515599 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.513512    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 26 16:00:38.515599 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.513515    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 26 16:00:38.515599 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.513517    2556 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 26 16:00:38.515599 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.513521    2556 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 26 16:00:38.515599 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.513525    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 26 16:00:38.515599 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.513528    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 26 16:00:38.515599 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.513530    2556 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 26 16:00:38.515599 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.513533    2556 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 26 16:00:38.515599 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.513536    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 26 16:00:38.515599 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.513538    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 26 16:00:38.515599 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.513541    2556 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 26 16:00:38.515599 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.513543    2556 feature_gate.go:328] unrecognized feature gate: DualReplica
May 26 16:00:38.516076 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.513546    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 26 16:00:38.516076 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.513548    2556 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 26 16:00:38.516076 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.513551    2556 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 26 16:00:38.516076 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.513554    2556 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 26 16:00:38.516076 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.513556    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 26 16:00:38.516076 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.513559    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 26 16:00:38.516076 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.513561    2556 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 26 16:00:38.516076 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.513564    2556 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 26 16:00:38.516076 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.513567    2556 feature_gate.go:328] unrecognized feature gate: Example2
May 26 16:00:38.516076 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.513570    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 26 16:00:38.516076 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.513572    2556 feature_gate.go:328] unrecognized feature gate: NewOLM
May 26 16:00:38.516076 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.513575    2556 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 26 16:00:38.516076 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.513577    2556 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 26 16:00:38.516076 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.513580    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 26 16:00:38.516076 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.513582    2556 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 26 16:00:38.516076 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.513586    2556 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 26 16:00:38.516076 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.513588    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 26 16:00:38.516076 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.513591    2556 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 26 16:00:38.516076 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.513593    2556 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 26 16:00:38.516076 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.513596    2556 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 26 16:00:38.516587 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.513598    2556 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 26 16:00:38.516587 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.513601    2556 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 26 16:00:38.516587 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.513604    2556 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 26 16:00:38.516587 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.513606    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 26 16:00:38.516587 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.513609    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 26 16:00:38.516587 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.513611    2556 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 26 16:00:38.516587 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.513614    2556 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 26 16:00:38.516587 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.513616    2556 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 26 16:00:38.516587 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.513618    2556 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 26 16:00:38.516587 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.513621    2556 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 26 16:00:38.516587 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.513624    2556 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 26 16:00:38.516587 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.513626    2556 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 26 16:00:38.516587 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.513629    2556 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 26 16:00:38.516587 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.513631    2556 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 26 16:00:38.516587 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.513634    2556 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 26 16:00:38.516587 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.513637    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 26 16:00:38.516587 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.513639    2556 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 26 16:00:38.516587 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.513642    2556 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 26 16:00:38.516587 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.513644    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 26 16:00:38.516587 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.513646    2556 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 26 16:00:38.517136 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.513649    2556 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 26 16:00:38.517136 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.513651    2556 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 26 16:00:38.517136 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.513654    2556 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 26 16:00:38.517136 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.513656    2556 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 26 16:00:38.517136 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.513659    2556 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 26 16:00:38.517136 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.513661    2556 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 26 16:00:38.517136 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.513663    2556 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 26 16:00:38.517136 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.513666    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 26 16:00:38.517136 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.513669    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 26 16:00:38.517136 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.513671    2556 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 26 16:00:38.517136 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.513673    2556 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 26 16:00:38.517136 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.513676    2556 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 26 16:00:38.517136 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.513679    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 26 16:00:38.517136 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.513682    2556 feature_gate.go:328] unrecognized feature gate: Example
May 26 16:00:38.517136 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.513684    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 26 16:00:38.517136 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.513688    2556 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 26 16:00:38.517136 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.513690    2556 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 26 16:00:38.517136 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:38.513693    2556 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 26 16:00:38.517572 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.513697    2556 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]}
May 26 16:00:38.517572 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.514597    2556 server.go:962] "Client rotation is on, will bootstrap in background"
May 26 16:00:38.517572 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.517073    2556 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
May 26 16:00:38.518203 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.518192    2556 server.go:1019] "Starting client certificate rotation"
May 26 16:00:38.518304 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.518289    2556 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
May 26 16:00:38.518341 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.518330    2556 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
May 26 16:00:38.544140 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.544120    2556 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
May 26 16:00:38.550832 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.550817    2556 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
May 26 16:00:38.565317 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.565295    2556 log.go:25] "Validated CRI v1 runtime API"
May 26 16:00:38.571495 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.571481    2556 log.go:25] "Validated CRI v1 image API"
May 26 16:00:38.574734 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.574717    2556 server.go:1452] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
May 26 16:00:38.580077 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.580045    2556 fs.go:135] Filesystem UUIDs: map[6985f284-14fb-4133-9248-312430f71419:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 7f9d462a-b0b3-41dc-8f99-87f617e9a2d5:/dev/nvme0n1p3]
May 26 16:00:38.580178 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.580067    2556 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
May 26 16:00:38.582449 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.582430    2556 reflector.go:430] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
May 26 16:00:38.584303 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.584195    2556 manager.go:217] Machine: {Timestamp:2026-05-26 16:00:38.583694082 +0000 UTC m=+0.428311067 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3103034 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec29642cca22a96bb314d6c0f97eded8 SystemUUID:ec29642c-ca22-a96b-b314-d6c0f97eded8 BootID:44aa3725-1d37-49b8-ad4d-1d6a05fbe638 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6103040 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:2a:cb:6b:94:77 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:2a:cb:6b:94:77 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:32:f8:36:35:03:b7 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
May 26 16:00:38.585026 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.585015    2556 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
May 26 16:00:38.585172 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.585159    2556 manager.go:233] Version: {KernelVersion:5.14.0-570.114.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260512-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
May 26 16:00:38.587267 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.587240    2556 container_manager_linux.go:270] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
May 26 16:00:38.587416 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.587270    2556 container_manager_linux.go:275] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-134-19.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
May 26 16:00:38.587458 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.587426    2556 topology_manager.go:138] "Creating topology manager with none policy"
May 26 16:00:38.587458 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.587434    2556 container_manager_linux.go:306] "Creating device plugin manager"
May 26 16:00:38.587458 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.587442    2556 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
May 26 16:00:38.588289 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.588279    2556 server.go:72] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
May 26 16:00:38.590359 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.590348    2556 state_mem.go:36] "Initialized new in-memory state store"
May 26 16:00:38.590468 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.590459    2556 server.go:1267] "Using root directory" path="/var/lib/kubelet"
May 26 16:00:38.593287 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.593278    2556 kubelet.go:491] "Attempting to sync node with API server"
May 26 16:00:38.593322 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.593292    2556 kubelet.go:386] "Adding static pod path" path="/etc/kubernetes/manifests"
May 26 16:00:38.593322 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.593303    2556 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
May 26 16:00:38.593322 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.593312    2556 kubelet.go:397] "Adding apiserver pod source"
May 26 16:00:38.593322 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.593321    2556 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
May 26 16:00:38.594568 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.594558    2556 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
May 26 16:00:38.594601 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.594576    2556 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
May 26 16:00:38.598012 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.597995    2556 kuberuntime_manager.go:279] "Container runtime initialized" containerRuntime="cri-o" version="1.33.11-2.rhaos4.20.gitb2a8320.el9" apiVersion="v1"
May 26 16:00:38.600528 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.600515    2556 kubelet.go:953] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
May 26 16:00:38.602808 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.602795    2556 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
May 26 16:00:38.602857 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.602813    2556 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
May 26 16:00:38.602857 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.602820    2556 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
May 26 16:00:38.602857 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.602826    2556 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
May 26 16:00:38.602937 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.602835    2556 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
May 26 16:00:38.602937 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.602872    2556 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/secret"
May 26 16:00:38.602937 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.602879    2556 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
May 26 16:00:38.602937 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.602884    2556 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
May 26 16:00:38.602937 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.602891    2556 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/fc"
May 26 16:00:38.602937 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.602897    2556 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
May 26 16:00:38.602937 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.602906    2556 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/projected"
May 26 16:00:38.602937 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.602911    2556 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
May 26 16:00:38.603777 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.603761    2556 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-f4mds"
May 26 16:00:38.603931 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.603921    2556 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/csi"
May 26 16:00:38.603963 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.603935    2556 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/image"
May 26 16:00:38.605367 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:00:38.605332    2556 reflector.go:200] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
May 26 16:00:38.605734 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:00:38.605715    2556 reflector.go:200] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-134-19.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
May 26 16:00:38.606805 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.606785    2556 csi_plugin.go:988] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-134-19.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
May 26 16:00:38.607507 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.607495    2556 watchdog_linux.go:99] "Systemd watchdog is not enabled"
May 26 16:00:38.607569 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.607533    2556 server.go:1295] "Started kubelet"
May 26 16:00:38.607620 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.607585    2556 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
May 26 16:00:38.607713 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.607650    2556 ratelimit.go:55] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
May 26 16:00:38.607773 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.607755    2556 server_v1.go:47] "podresources" method="list" useActivePods=true
May 26 16:00:38.608445 ip-10-0-134-19 systemd[1]: Started Kubernetes Kubelet.
May 26 16:00:38.609307 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.609293    2556 server.go:255] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
May 26 16:00:38.610720 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.610701    2556 server.go:317] "Adding debug handlers to kubelet server"
May 26 16:00:38.613035 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.613013    2556 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-f4mds"
May 26 16:00:38.617568 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:00:38.617548    2556 kubelet.go:1618] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
May 26 16:00:38.618262 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:00:38.617076    2556 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-19.ec2.internal.18b328e44472cfb1  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-19.ec2.internal,UID:ip-10-0-134-19.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-134-19.ec2.internal,},FirstTimestamp:2026-05-26 16:00:38.607507377 +0000 UTC m=+0.452124363,LastTimestamp:2026-05-26 16:00:38.607507377 +0000 UTC m=+0.452124363,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-19.ec2.internal,}"
May 26 16:00:38.618592 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.618581    2556 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
May 26 16:00:38.618642 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.618623    2556 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
May 26 16:00:38.618700 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.618652    2556 volume_manager.go:295] "The desired_state_of_world populator starts"
May 26 16:00:38.618700 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.618667    2556 volume_manager.go:297] "Starting Kubelet Volume Manager"
May 26 16:00:38.618796 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.618747    2556 desired_state_of_world_populator.go:150] "Desired state populator starts to run"
May 26 16:00:38.620837 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.620817    2556 reconstruct.go:97] "Volume reconstruction finished"
May 26 16:00:38.620837 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.620836    2556 reconciler.go:26] "Reconciler: start to sync state"
May 26 16:00:38.620974 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:00:38.620919    2556 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-134-19.ec2.internal\" not found"
May 26 16:00:38.621399 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.621382    2556 factory.go:55] Registering systemd factory
May 26 16:00:38.621486 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.621413    2556 factory.go:223] Registration of the systemd container factory successfully
May 26 16:00:38.621796 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.621772    2556 factory.go:153] Registering CRI-O factory
May 26 16:00:38.621861 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.621805    2556 factory.go:223] Registration of the crio container factory successfully
May 26 16:00:38.621917 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.621860    2556 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
May 26 16:00:38.621917 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.621883    2556 factory.go:103] Registering Raw factory
May 26 16:00:38.621917 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.621907    2556 manager.go:1196] Started watching for new ooms in manager
May 26 16:00:38.623666 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.623645    2556 manager.go:319] Starting recovery of all containers
May 26 16:00:38.628327 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.628305    2556 reflector.go:430] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
May 26 16:00:38.630814 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:00:38.630602    2556 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-134-19.ec2.internal\" not found" node="ip-10-0-134-19.ec2.internal"
May 26 16:00:38.635538 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.635526    2556 manager.go:324] Recovery completed
May 26 16:00:38.639704 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.639692    2556 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 26 16:00:38.641837 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.641817    2556 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-134-19.ec2.internal" event="NodeHasSufficientMemory"
May 26 16:00:38.641902 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.641851    2556 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-134-19.ec2.internal" event="NodeHasNoDiskPressure"
May 26 16:00:38.641902 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.641862    2556 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-134-19.ec2.internal" event="NodeHasSufficientPID"
May 26 16:00:38.642453 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.642439    2556 cpu_manager.go:222] "Starting CPU manager" policy="none"
May 26 16:00:38.642453 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.642452    2556 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
May 26 16:00:38.642533 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.642470    2556 state_mem.go:36] "Initialized new in-memory state store"
May 26 16:00:38.646370 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.646358    2556 policy_none.go:49] "None policy: Start"
May 26 16:00:38.646421 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.646375    2556 memory_manager.go:186] "Starting memorymanager" policy="None"
May 26 16:00:38.646421 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.646385    2556 state_mem.go:35] "Initializing new in-memory state store"
May 26 16:00:38.688741 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.688722    2556 manager.go:341] "Starting Device Plugin manager"
May 26 16:00:38.706339 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:00:38.688758    2556 manager.go:517] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
May 26 16:00:38.706339 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.688771    2556 server.go:85] "Starting device plugin registration server"
May 26 16:00:38.706339 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.689029    2556 eviction_manager.go:189] "Eviction manager: starting control loop"
May 26 16:00:38.706339 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.689040    2556 container_log_manager.go:189] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
May 26 16:00:38.706339 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.689153    2556 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
May 26 16:00:38.706339 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:00:38.689695    2556 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
May 26 16:00:38.706339 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:00:38.689739    2556 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-134-19.ec2.internal\" not found"
May 26 16:00:38.706339 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.689776    2556 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
May 26 16:00:38.706339 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.689785    2556 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
May 26 16:00:38.765238 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.765159    2556 kubelet_network_linux.go:49] "Initialized iptables rules." protocol="IPv4"
May 26 16:00:38.766327 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.766307    2556 kubelet_network_linux.go:49] "Initialized iptables rules." protocol="IPv6"
May 26 16:00:38.766401 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.766344    2556 status_manager.go:230] "Starting to sync pod status with apiserver"
May 26 16:00:38.766401 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.766371    2556 watchdog_linux.go:127] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
May 26 16:00:38.766401 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.766381    2556 kubelet.go:2451] "Starting kubelet main sync loop"
May 26 16:00:38.766520 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:00:38.766477    2556 kubelet.go:2475] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
May 26 16:00:38.773508 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.773488    2556 reflector.go:430] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
May 26 16:00:38.790140 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.790118    2556 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 26 16:00:38.791071 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.791054    2556 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-134-19.ec2.internal" event="NodeHasSufficientMemory"
May 26 16:00:38.791186 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.791085    2556 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-134-19.ec2.internal" event="NodeHasNoDiskPressure"
May 26 16:00:38.791186 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.791115    2556 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-134-19.ec2.internal" event="NodeHasSufficientPID"
May 26 16:00:38.791186 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.791146    2556 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-134-19.ec2.internal"
May 26 16:00:38.802263 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.802241    2556 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-134-19.ec2.internal"
May 26 16:00:38.802348 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:00:38.802271    2556 kubelet_node_status.go:597] "Error updating node status, will retry" err="error getting node \"ip-10-0-134-19.ec2.internal\": node \"ip-10-0-134-19.ec2.internal\" not found"
May 26 16:00:38.820274 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:00:38.820249    2556 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-134-19.ec2.internal\" not found"
May 26 16:00:38.866667 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.866638    2556 kubelet.go:2537] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-19.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-134-19.ec2.internal"]
May 26 16:00:38.866728 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.866713    2556 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 26 16:00:38.867535 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.867520    2556 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-134-19.ec2.internal" event="NodeHasSufficientMemory"
May 26 16:00:38.867617 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.867553    2556 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-134-19.ec2.internal" event="NodeHasNoDiskPressure"
May 26 16:00:38.867617 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.867567    2556 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-134-19.ec2.internal" event="NodeHasSufficientPID"
May 26 16:00:38.868772 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.868758    2556 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 26 16:00:38.868920 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.868906    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-19.ec2.internal"
May 26 16:00:38.868969 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.868933    2556 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 26 16:00:38.869481 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.869465    2556 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-134-19.ec2.internal" event="NodeHasSufficientMemory"
May 26 16:00:38.869547 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.869495    2556 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-134-19.ec2.internal" event="NodeHasNoDiskPressure"
May 26 16:00:38.869547 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.869471    2556 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-134-19.ec2.internal" event="NodeHasSufficientMemory"
May 26 16:00:38.869547 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.869525    2556 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-134-19.ec2.internal" event="NodeHasNoDiskPressure"
May 26 16:00:38.869547 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.869536    2556 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-134-19.ec2.internal" event="NodeHasSufficientPID"
May 26 16:00:38.869671 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.869506    2556 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-134-19.ec2.internal" event="NodeHasSufficientPID"
May 26 16:00:38.870543 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.870527    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-19.ec2.internal"
May 26 16:00:38.870619 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.870554    2556 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 26 16:00:38.871210 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.871196    2556 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-134-19.ec2.internal" event="NodeHasSufficientMemory"
May 26 16:00:38.871275 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.871223    2556 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-134-19.ec2.internal" event="NodeHasNoDiskPressure"
May 26 16:00:38.871275 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.871233    2556 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-134-19.ec2.internal" event="NodeHasSufficientPID"
May 26 16:00:38.900784 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:00:38.900763    2556 kubelet.go:3336] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-19.ec2.internal\" not found" node="ip-10-0-134-19.ec2.internal"
May 26 16:00:38.905205 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:00:38.905189    2556 kubelet.go:3336] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-19.ec2.internal\" not found" node="ip-10-0-134-19.ec2.internal"
May 26 16:00:38.920314 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:00:38.920294    2556 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-134-19.ec2.internal\" not found"
May 26 16:00:38.923199 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.923184    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f0079648f8b5dce19cfb7d44c2f62663-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-19.ec2.internal\" (UID: \"f0079648f8b5dce19cfb7d44c2f62663\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-19.ec2.internal"
May 26 16:00:38.923251 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.923209    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f0079648f8b5dce19cfb7d44c2f62663-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-19.ec2.internal\" (UID: \"f0079648f8b5dce19cfb7d44c2f62663\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-19.ec2.internal"
May 26 16:00:38.923251 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:38.923225    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c33db3f858f26d571aec8a6b75b8b9e2-config\") pod \"kube-apiserver-proxy-ip-10-0-134-19.ec2.internal\" (UID: \"c33db3f858f26d571aec8a6b75b8b9e2\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-19.ec2.internal"
May 26 16:00:39.020688 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:00:39.020600    2556 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-134-19.ec2.internal\" not found"
May 26 16:00:39.023794 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:39.023775    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f0079648f8b5dce19cfb7d44c2f62663-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-19.ec2.internal\" (UID: \"f0079648f8b5dce19cfb7d44c2f62663\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-19.ec2.internal"
May 26 16:00:39.023879 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:39.023810    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f0079648f8b5dce19cfb7d44c2f62663-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-19.ec2.internal\" (UID: \"f0079648f8b5dce19cfb7d44c2f62663\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-19.ec2.internal"
May 26 16:00:39.023879 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:39.023835    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c33db3f858f26d571aec8a6b75b8b9e2-config\") pod \"kube-apiserver-proxy-ip-10-0-134-19.ec2.internal\" (UID: \"c33db3f858f26d571aec8a6b75b8b9e2\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-19.ec2.internal"
May 26 16:00:39.023967 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:39.023880    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f0079648f8b5dce19cfb7d44c2f62663-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-19.ec2.internal\" (UID: \"f0079648f8b5dce19cfb7d44c2f62663\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-19.ec2.internal"
May 26 16:00:39.023967 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:39.023885    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f0079648f8b5dce19cfb7d44c2f62663-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-19.ec2.internal\" (UID: \"f0079648f8b5dce19cfb7d44c2f62663\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-19.ec2.internal"
May 26 16:00:39.023967 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:39.023887    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c33db3f858f26d571aec8a6b75b8b9e2-config\") pod \"kube-apiserver-proxy-ip-10-0-134-19.ec2.internal\" (UID: \"c33db3f858f26d571aec8a6b75b8b9e2\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-19.ec2.internal"
May 26 16:00:39.121683 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:00:39.121645    2556 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-134-19.ec2.internal\" not found"
May 26 16:00:39.203975 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:39.203953    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-19.ec2.internal"
May 26 16:00:39.209122 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:39.209102    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-19.ec2.internal"
May 26 16:00:39.222302 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:00:39.222285    2556 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-134-19.ec2.internal\" not found"
May 26 16:00:39.322862 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:00:39.322770    2556 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-134-19.ec2.internal\" not found"
May 26 16:00:39.423280 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:00:39.423245    2556 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-134-19.ec2.internal\" not found"
May 26 16:00:39.517609 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:39.517583    2556 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
May 26 16:00:39.518188 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:39.517736    2556 reflector.go:556] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
May 26 16:00:39.518188 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:39.517769    2556 reflector.go:556] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
May 26 16:00:39.523913 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:00:39.523892    2556 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-134-19.ec2.internal\" not found"
May 26 16:00:39.616141 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:39.616085    2556 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-05-25 15:55:38 +0000 UTC" deadline="2027-11-20 07:40:05.181048201 +0000 UTC"
May 26 16:00:39.616141 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:39.616134    2556 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13023h39m25.564917898s"
May 26 16:00:39.618886 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:39.618867    2556 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
May 26 16:00:39.623978 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:00:39.623958    2556 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-134-19.ec2.internal\" not found"
May 26 16:00:39.633801 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:39.633776    2556 reflector.go:430] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
May 26 16:00:39.656121 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:39.656081    2556 reflector.go:430] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
May 26 16:00:39.656786 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:39.656766    2556 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-xsmrq"
May 26 16:00:39.667136 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:39.667117    2556 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-xsmrq"
May 26 16:00:39.712911 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:39.712880    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0079648f8b5dce19cfb7d44c2f62663.slice/crio-e27d80118326c6b23c2ff11615b577218461655cc725faec0debe6b0ed1533a2 WatchSource:0}: Error finding container e27d80118326c6b23c2ff11615b577218461655cc725faec0debe6b0ed1533a2: Status 404 returned error can't find the container with id e27d80118326c6b23c2ff11615b577218461655cc725faec0debe6b0ed1533a2
May 26 16:00:39.713314 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:39.713295    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc33db3f858f26d571aec8a6b75b8b9e2.slice/crio-9f0f601cbfe0f8f70e9af7d07ce0c793e18a957a199f6134348dd5fdbb60cfce WatchSource:0}: Error finding container 9f0f601cbfe0f8f70e9af7d07ce0c793e18a957a199f6134348dd5fdbb60cfce: Status 404 returned error can't find the container with id 9f0f601cbfe0f8f70e9af7d07ce0c793e18a957a199f6134348dd5fdbb60cfce
May 26 16:00:39.718621 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:39.718607    2556 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 26 16:00:39.724203 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:00:39.724187    2556 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-134-19.ec2.internal\" not found"
May 26 16:00:39.769192 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:39.769156    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-19.ec2.internal" event={"ID":"f0079648f8b5dce19cfb7d44c2f62663","Type":"ContainerStarted","Data":"e27d80118326c6b23c2ff11615b577218461655cc725faec0debe6b0ed1533a2"}
May 26 16:00:39.770080 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:39.770060    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-19.ec2.internal" event={"ID":"c33db3f858f26d571aec8a6b75b8b9e2","Type":"ContainerStarted","Data":"9f0f601cbfe0f8f70e9af7d07ce0c793e18a957a199f6134348dd5fdbb60cfce"}
May 26 16:00:39.824370 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:00:39.824348    2556 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-134-19.ec2.internal\" not found"
May 26 16:00:39.913980 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:39.913929    2556 reflector.go:430] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
May 26 16:00:39.919486 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:39.919473    2556 kubelet.go:3340] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-19.ec2.internal"
May 26 16:00:39.932187 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:39.932169    2556 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
May 26 16:00:39.933255 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:39.933245    2556 kubelet.go:3340] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-19.ec2.internal"
May 26 16:00:39.949442 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:39.949425    2556 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
May 26 16:00:40.466917 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.466717    2556 reflector.go:430] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
May 26 16:00:40.594428 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.594390    2556 apiserver.go:52] "Watching apiserver"
May 26 16:00:40.603266 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.603237    2556 reflector.go:430] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
May 26 16:00:40.603653 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.603628    2556 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-134-19.ec2.internal","openshift-image-registry/node-ca-lrkmg","openshift-multus/multus-additional-cni-plugins-9znf9","openshift-multus/multus-zc768","kube-system/konnectivity-agent-b2npd","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pmgqs","openshift-cluster-node-tuning-operator/tuned-gh2df","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-19.ec2.internal","openshift-multus/network-metrics-daemon-g9scb","openshift-network-diagnostics/network-check-target-twfp8","openshift-network-operator/iptables-alerter-s4dxt","openshift-ovn-kubernetes/ovnkube-node-sk7ns","kube-system/global-pull-secret-syncer-wc8kh"]
May 26 16:00:40.609150 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.609124    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-gh2df"
May 26 16:00:40.611468 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.611289    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-lrkmg"
May 26 16:00:40.612272 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.612254    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
May 26 16:00:40.612702 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.612683    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-8rxjf\""
May 26 16:00:40.612853 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.612829    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
May 26 16:00:40.613364 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.613347    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-9znf9"
May 26 16:00:40.613453 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.613431    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-zc768"
May 26 16:00:40.613994 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.613970    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
May 26 16:00:40.614121 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.614085    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-7cdd2\""
May 26 16:00:40.614224 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.613974    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
May 26 16:00:40.614343 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.614299    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
May 26 16:00:40.615668 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.615650    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-b2npd"
May 26 16:00:40.616146 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.616037    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
May 26 16:00:40.616146 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.616052    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-lmwtw\""
May 26 16:00:40.616326 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.616241    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
May 26 16:00:40.616326 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.616314    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
May 26 16:00:40.616559 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.616542    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
May 26 16:00:40.616658 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.616636    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
May 26 16:00:40.617429 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.617166    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-bfhgh\""
May 26 16:00:40.618664 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.617579    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
May 26 16:00:40.618664 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.618229    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pmgqs"
May 26 16:00:40.619268 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.619247    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
May 26 16:00:40.619511 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.619497    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
May 26 16:00:40.621922 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.621878    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g9scb"
May 26 16:00:40.622022 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:00:40.621970    2556 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g9scb" podUID="aed39145-780f-4642-af53-ddccbc03c34c"
May 26 16:00:40.622177 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.622135    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
May 26 16:00:40.622527 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.622509    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
May 26 16:00:40.622709 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.622689    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-5rtx7\""
May 26 16:00:40.622909 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.622894    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-ph77n\""
May 26 16:00:40.622988 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.622971    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
May 26 16:00:40.624213 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.624193    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-twfp8"
May 26 16:00:40.624293 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:00:40.624258    2556 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-twfp8" podUID="115c1521-6997-403b-98d4-38a9198dd28e"
May 26 16:00:40.626504 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.626487    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-s4dxt"
May 26 16:00:40.628884 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.628866    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-sk7ns"
May 26 16:00:40.629275 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.629255    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
May 26 16:00:40.629377 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.629320    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
May 26 16:00:40.629377 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.629258    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-5g6jj\""
May 26 16:00:40.629715 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.629695    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
May 26 16:00:40.631139 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.631112    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6b5a0115-0b0c-4d86-8e58-732e7814f4c9-hostroot\") pod \"multus-zc768\" (UID: \"6b5a0115-0b0c-4d86-8e58-732e7814f4c9\") " pod="openshift-multus/multus-zc768"
May 26 16:00:40.631241 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.631115    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wc8kh"
May 26 16:00:40.631241 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.631150    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
May 26 16:00:40.631241 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.631149    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aed39145-780f-4642-af53-ddccbc03c34c-metrics-certs\") pod \"network-metrics-daemon-g9scb\" (UID: \"aed39145-780f-4642-af53-ddccbc03c34c\") " pod="openshift-multus/network-metrics-daemon-g9scb"
May 26 16:00:40.631241 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.631193    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/bc458685-6108-4373-aa17-f6e49f2c6e1b-serviceca\") pod \"node-ca-lrkmg\" (UID: \"bc458685-6108-4373-aa17-f6e49f2c6e1b\") " pod="openshift-image-registry/node-ca-lrkmg"
May 26 16:00:40.631241 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:00:40.631201    2556 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wc8kh" podUID="c0cc81a8-fc3f-4188-8e90-4640d751dfb5"
May 26 16:00:40.631241 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.631217    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/42e4c880-fbee-4786-bbe5-7004126a87be-etc-sysctl-d\") pod \"tuned-gh2df\" (UID: \"42e4c880-fbee-4786-bbe5-7004126a87be\") " pod="openshift-cluster-node-tuning-operator/tuned-gh2df"
May 26 16:00:40.631241 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.631241    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6b5a0115-0b0c-4d86-8e58-732e7814f4c9-multus-cni-dir\") pod \"multus-zc768\" (UID: \"6b5a0115-0b0c-4d86-8e58-732e7814f4c9\") " pod="openshift-multus/multus-zc768"
May 26 16:00:40.631634 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.631264    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6b5a0115-0b0c-4d86-8e58-732e7814f4c9-cni-binary-copy\") pod \"multus-zc768\" (UID: \"6b5a0115-0b0c-4d86-8e58-732e7814f4c9\") " pod="openshift-multus/multus-zc768"
May 26 16:00:40.631634 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.631287    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6b5a0115-0b0c-4d86-8e58-732e7814f4c9-multus-socket-dir-parent\") pod \"multus-zc768\" (UID: \"6b5a0115-0b0c-4d86-8e58-732e7814f4c9\") " pod="openshift-multus/multus-zc768"
May 26 16:00:40.631634 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.631310    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6b5a0115-0b0c-4d86-8e58-732e7814f4c9-host-var-lib-cni-bin\") pod \"multus-zc768\" (UID: \"6b5a0115-0b0c-4d86-8e58-732e7814f4c9\") " pod="openshift-multus/multus-zc768"
May 26 16:00:40.631634 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.631333    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6b5a0115-0b0c-4d86-8e58-732e7814f4c9-etc-kubernetes\") pod \"multus-zc768\" (UID: \"6b5a0115-0b0c-4d86-8e58-732e7814f4c9\") " pod="openshift-multus/multus-zc768"
May 26 16:00:40.631634 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.631389    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/42e4c880-fbee-4786-bbe5-7004126a87be-etc-modprobe-d\") pod \"tuned-gh2df\" (UID: \"42e4c880-fbee-4786-bbe5-7004126a87be\") " pod="openshift-cluster-node-tuning-operator/tuned-gh2df"
May 26 16:00:40.631634 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.631426    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/42e4c880-fbee-4786-bbe5-7004126a87be-etc-systemd\") pod \"tuned-gh2df\" (UID: \"42e4c880-fbee-4786-bbe5-7004126a87be\") " pod="openshift-cluster-node-tuning-operator/tuned-gh2df"
May 26 16:00:40.631634 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.631453    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6b5a0115-0b0c-4d86-8e58-732e7814f4c9-host-var-lib-cni-multus\") pod \"multus-zc768\" (UID: \"6b5a0115-0b0c-4d86-8e58-732e7814f4c9\") " pod="openshift-multus/multus-zc768"
May 26 16:00:40.631634 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.631489    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/02507d1e-2665-4380-aa87-3605e4411561-agent-certs\") pod \"konnectivity-agent-b2npd\" (UID: \"02507d1e-2665-4380-aa87-3605e4411561\") " pod="kube-system/konnectivity-agent-b2npd"
May 26 16:00:40.631634 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.631531    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/02507d1e-2665-4380-aa87-3605e4411561-konnectivity-ca\") pod \"konnectivity-agent-b2npd\" (UID: \"02507d1e-2665-4380-aa87-3605e4411561\") " pod="kube-system/konnectivity-agent-b2npd"
May 26 16:00:40.631634 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.631579    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbwzt\" (UniqueName: \"kubernetes.io/projected/aed39145-780f-4642-af53-ddccbc03c34c-kube-api-access-fbwzt\") pod \"network-metrics-daemon-g9scb\" (UID: \"aed39145-780f-4642-af53-ddccbc03c34c\") " pod="openshift-multus/network-metrics-daemon-g9scb"
May 26 16:00:40.631634 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.631608    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/42e4c880-fbee-4786-bbe5-7004126a87be-lib-modules\") pod \"tuned-gh2df\" (UID: \"42e4c880-fbee-4786-bbe5-7004126a87be\") " pod="openshift-cluster-node-tuning-operator/tuned-gh2df"
May 26 16:00:40.631634 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.631633    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/42e4c880-fbee-4786-bbe5-7004126a87be-var-lib-kubelet\") pod \"tuned-gh2df\" (UID: \"42e4c880-fbee-4786-bbe5-7004126a87be\") " pod="openshift-cluster-node-tuning-operator/tuned-gh2df"
May 26 16:00:40.632411 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.631656    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p458s\" (UniqueName: \"kubernetes.io/projected/42e4c880-fbee-4786-bbe5-7004126a87be-kube-api-access-p458s\") pod \"tuned-gh2df\" (UID: \"42e4c880-fbee-4786-bbe5-7004126a87be\") " pod="openshift-cluster-node-tuning-operator/tuned-gh2df"
May 26 16:00:40.632411 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.631679    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jlfs9\" (UniqueName: \"kubernetes.io/projected/bc458685-6108-4373-aa17-f6e49f2c6e1b-kube-api-access-jlfs9\") pod \"node-ca-lrkmg\" (UID: \"bc458685-6108-4373-aa17-f6e49f2c6e1b\") " pod="openshift-image-registry/node-ca-lrkmg"
May 26 16:00:40.632411 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.631701    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/391025fc-21eb-4786-8a0b-7f81c9eb90a4-cni-binary-copy\") pod \"multus-additional-cni-plugins-9znf9\" (UID: \"391025fc-21eb-4786-8a0b-7f81c9eb90a4\") " pod="openshift-multus/multus-additional-cni-plugins-9znf9"
May 26 16:00:40.632411 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.631725    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/391025fc-21eb-4786-8a0b-7f81c9eb90a4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9znf9\" (UID: \"391025fc-21eb-4786-8a0b-7f81c9eb90a4\") " pod="openshift-multus/multus-additional-cni-plugins-9znf9"
May 26 16:00:40.632411 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.631766    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tz72f\" (UniqueName: \"kubernetes.io/projected/391025fc-21eb-4786-8a0b-7f81c9eb90a4-kube-api-access-tz72f\") pod \"multus-additional-cni-plugins-9znf9\" (UID: \"391025fc-21eb-4786-8a0b-7f81c9eb90a4\") " pod="openshift-multus/multus-additional-cni-plugins-9znf9"
May 26 16:00:40.632411 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.631783    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6b5a0115-0b0c-4d86-8e58-732e7814f4c9-host-run-multus-certs\") pod \"multus-zc768\" (UID: \"6b5a0115-0b0c-4d86-8e58-732e7814f4c9\") " pod="openshift-multus/multus-zc768"
May 26 16:00:40.632411 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.631794    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
May 26 16:00:40.632411 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.631810    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgm45\" (UniqueName: \"kubernetes.io/projected/6b5a0115-0b0c-4d86-8e58-732e7814f4c9-kube-api-access-kgm45\") pod \"multus-zc768\" (UID: \"6b5a0115-0b0c-4d86-8e58-732e7814f4c9\") " pod="openshift-multus/multus-zc768"
May 26 16:00:40.632411 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.631836    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/42e4c880-fbee-4786-bbe5-7004126a87be-run\") pod \"tuned-gh2df\" (UID: \"42e4c880-fbee-4786-bbe5-7004126a87be\") " pod="openshift-cluster-node-tuning-operator/tuned-gh2df"
May 26 16:00:40.632411 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.631872    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/42e4c880-fbee-4786-bbe5-7004126a87be-etc-tuned\") pod \"tuned-gh2df\" (UID: \"42e4c880-fbee-4786-bbe5-7004126a87be\") " pod="openshift-cluster-node-tuning-operator/tuned-gh2df"
May 26 16:00:40.632411 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.631896    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b75ad1e4-d3a4-4f24-bc06-3ef6661e1b3d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-pmgqs\" (UID: \"b75ad1e4-d3a4-4f24-bc06-3ef6661e1b3d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pmgqs"
May 26 16:00:40.632411 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.631898    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
May 26 16:00:40.632411 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.631917    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b75ad1e4-d3a4-4f24-bc06-3ef6661e1b3d-sys-fs\") pod \"aws-ebs-csi-driver-node-pmgqs\" (UID: \"b75ad1e4-d3a4-4f24-bc06-3ef6661e1b3d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pmgqs"
May 26 16:00:40.632411 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.631963    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/391025fc-21eb-4786-8a0b-7f81c9eb90a4-system-cni-dir\") pod \"multus-additional-cni-plugins-9znf9\" (UID: \"391025fc-21eb-4786-8a0b-7f81c9eb90a4\") " pod="openshift-multus/multus-additional-cni-plugins-9znf9"
May 26 16:00:40.632411 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.631974    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
May 26 16:00:40.632411 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.631990    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/391025fc-21eb-4786-8a0b-7f81c9eb90a4-os-release\") pod \"multus-additional-cni-plugins-9znf9\" (UID: \"391025fc-21eb-4786-8a0b-7f81c9eb90a4\") " pod="openshift-multus/multus-additional-cni-plugins-9znf9"
May 26 16:00:40.632411 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.632000    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
May 26 16:00:40.632411 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.632012    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6b5a0115-0b0c-4d86-8e58-732e7814f4c9-cnibin\") pod \"multus-zc768\" (UID: \"6b5a0115-0b0c-4d86-8e58-732e7814f4c9\") " pod="openshift-multus/multus-zc768"
May 26 16:00:40.633300 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.632037    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6b5a0115-0b0c-4d86-8e58-732e7814f4c9-os-release\") pod \"multus-zc768\" (UID: \"6b5a0115-0b0c-4d86-8e58-732e7814f4c9\") " pod="openshift-multus/multus-zc768"
May 26 16:00:40.633300 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.632059    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/42e4c880-fbee-4786-bbe5-7004126a87be-etc-sysctl-conf\") pod \"tuned-gh2df\" (UID: \"42e4c880-fbee-4786-bbe5-7004126a87be\") " pod="openshift-cluster-node-tuning-operator/tuned-gh2df"
May 26 16:00:40.633300 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.632081    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b75ad1e4-d3a4-4f24-bc06-3ef6661e1b3d-registration-dir\") pod \"aws-ebs-csi-driver-node-pmgqs\" (UID: \"b75ad1e4-d3a4-4f24-bc06-3ef6661e1b3d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pmgqs"
May 26 16:00:40.633300 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.632130    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mw64c\" (UniqueName: \"kubernetes.io/projected/b75ad1e4-d3a4-4f24-bc06-3ef6661e1b3d-kube-api-access-mw64c\") pod \"aws-ebs-csi-driver-node-pmgqs\" (UID: \"b75ad1e4-d3a4-4f24-bc06-3ef6661e1b3d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pmgqs"
May 26 16:00:40.633300 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.632154    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/391025fc-21eb-4786-8a0b-7f81c9eb90a4-cnibin\") pod \"multus-additional-cni-plugins-9znf9\" (UID: \"391025fc-21eb-4786-8a0b-7f81c9eb90a4\") " pod="openshift-multus/multus-additional-cni-plugins-9znf9"
May 26 16:00:40.633300 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.632176    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/391025fc-21eb-4786-8a0b-7f81c9eb90a4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9znf9\" (UID: \"391025fc-21eb-4786-8a0b-7f81c9eb90a4\") " pod="openshift-multus/multus-additional-cni-plugins-9znf9"
May 26 16:00:40.633300 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.632218    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/391025fc-21eb-4786-8a0b-7f81c9eb90a4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9znf9\" (UID: \"391025fc-21eb-4786-8a0b-7f81c9eb90a4\") " pod="openshift-multus/multus-additional-cni-plugins-9znf9"
May 26 16:00:40.633300 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.632242    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6b5a0115-0b0c-4d86-8e58-732e7814f4c9-system-cni-dir\") pod \"multus-zc768\" (UID: \"6b5a0115-0b0c-4d86-8e58-732e7814f4c9\") " pod="openshift-multus/multus-zc768"
May 26 16:00:40.633300 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.632282    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6b5a0115-0b0c-4d86-8e58-732e7814f4c9-multus-conf-dir\") pod \"multus-zc768\" (UID: \"6b5a0115-0b0c-4d86-8e58-732e7814f4c9\") " pod="openshift-multus/multus-zc768"
May 26 16:00:40.633300 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.632309    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/42e4c880-fbee-4786-bbe5-7004126a87be-etc-sysconfig\") pod \"tuned-gh2df\" (UID: \"42e4c880-fbee-4786-bbe5-7004126a87be\") " pod="openshift-cluster-node-tuning-operator/tuned-gh2df"
May 26 16:00:40.633300 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.632331    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/42e4c880-fbee-4786-bbe5-7004126a87be-etc-kubernetes\") pod \"tuned-gh2df\" (UID: \"42e4c880-fbee-4786-bbe5-7004126a87be\") " pod="openshift-cluster-node-tuning-operator/tuned-gh2df"
May 26 16:00:40.633300 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.632357    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/42e4c880-fbee-4786-bbe5-7004126a87be-sys\") pod \"tuned-gh2df\" (UID: \"42e4c880-fbee-4786-bbe5-7004126a87be\") " pod="openshift-cluster-node-tuning-operator/tuned-gh2df"
May 26 16:00:40.633300 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.632412    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/42e4c880-fbee-4786-bbe5-7004126a87be-host\") pod \"tuned-gh2df\" (UID: \"42e4c880-fbee-4786-bbe5-7004126a87be\") " pod="openshift-cluster-node-tuning-operator/tuned-gh2df"
May 26 16:00:40.633300 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.632454    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/42e4c880-fbee-4786-bbe5-7004126a87be-tmp\") pod \"tuned-gh2df\" (UID: \"42e4c880-fbee-4786-bbe5-7004126a87be\") " pod="openshift-cluster-node-tuning-operator/tuned-gh2df"
May 26 16:00:40.633300 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.632676    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b75ad1e4-d3a4-4f24-bc06-3ef6661e1b3d-socket-dir\") pod \"aws-ebs-csi-driver-node-pmgqs\" (UID: \"b75ad1e4-d3a4-4f24-bc06-3ef6661e1b3d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pmgqs"
May 26 16:00:40.633300 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.632705    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6b5a0115-0b0c-4d86-8e58-732e7814f4c9-host-run-k8s-cni-cncf-io\") pod \"multus-zc768\" (UID: \"6b5a0115-0b0c-4d86-8e58-732e7814f4c9\") " pod="openshift-multus/multus-zc768"
May 26 16:00:40.634171 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.632729    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6b5a0115-0b0c-4d86-8e58-732e7814f4c9-host-run-netns\") pod \"multus-zc768\" (UID: \"6b5a0115-0b0c-4d86-8e58-732e7814f4c9\") " pod="openshift-multus/multus-zc768"
May 26 16:00:40.634171 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.632752    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6b5a0115-0b0c-4d86-8e58-732e7814f4c9-host-var-lib-kubelet\") pod \"multus-zc768\" (UID: \"6b5a0115-0b0c-4d86-8e58-732e7814f4c9\") " pod="openshift-multus/multus-zc768"
May 26 16:00:40.634171 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.632786    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6b5a0115-0b0c-4d86-8e58-732e7814f4c9-multus-daemon-config\") pod \"multus-zc768\" (UID: \"6b5a0115-0b0c-4d86-8e58-732e7814f4c9\") " pod="openshift-multus/multus-zc768"
May 26 16:00:40.634171 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.632815    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bc458685-6108-4373-aa17-f6e49f2c6e1b-host\") pod \"node-ca-lrkmg\" (UID: \"bc458685-6108-4373-aa17-f6e49f2c6e1b\") " pod="openshift-image-registry/node-ca-lrkmg"
May 26 16:00:40.634171 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.632854    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b75ad1e4-d3a4-4f24-bc06-3ef6661e1b3d-device-dir\") pod \"aws-ebs-csi-driver-node-pmgqs\" (UID: \"b75ad1e4-d3a4-4f24-bc06-3ef6661e1b3d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pmgqs"
May 26 16:00:40.634171 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.632880    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b75ad1e4-d3a4-4f24-bc06-3ef6661e1b3d-etc-selinux\") pod \"aws-ebs-csi-driver-node-pmgqs\" (UID: \"b75ad1e4-d3a4-4f24-bc06-3ef6661e1b3d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pmgqs"
May 26 16:00:40.634171 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.632936    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-gpv87\""
May 26 16:00:40.635741 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.635723    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
May 26 16:00:40.668246 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.668213    2556 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-05-25 15:55:39 +0000 UTC" deadline="2027-10-30 21:37:54.471595148 +0000 UTC"
May 26 16:00:40.668346 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.668246    2556 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12533h37m13.803353727s"
May 26 16:00:40.720595 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.720516    2556 desired_state_of_world_populator.go:158] "Finished populating initial desired state of world"
May 26 16:00:40.733285 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.733255    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/391025fc-21eb-4786-8a0b-7f81c9eb90a4-cnibin\") pod \"multus-additional-cni-plugins-9znf9\" (UID: \"391025fc-21eb-4786-8a0b-7f81c9eb90a4\") " pod="openshift-multus/multus-additional-cni-plugins-9znf9"
May 26 16:00:40.733513 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.733297    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pvlm\" (UniqueName: \"kubernetes.io/projected/b6c5e632-79b2-4174-b806-a97342c29687-kube-api-access-9pvlm\") pod \"iptables-alerter-s4dxt\" (UID: \"b6c5e632-79b2-4174-b806-a97342c29687\") " pod="openshift-network-operator/iptables-alerter-s4dxt"
May 26 16:00:40.733513 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.733322    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4887c0d3-11fb-4287-bb94-a876a6c38803-run-openvswitch\") pod \"ovnkube-node-sk7ns\" (UID: \"4887c0d3-11fb-4287-bb94-a876a6c38803\") " pod="openshift-ovn-kubernetes/ovnkube-node-sk7ns"
May 26 16:00:40.733513 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.733345    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4887c0d3-11fb-4287-bb94-a876a6c38803-host-run-ovn-kubernetes\") pod \"ovnkube-node-sk7ns\" (UID: \"4887c0d3-11fb-4287-bb94-a876a6c38803\") " pod="openshift-ovn-kubernetes/ovnkube-node-sk7ns"
May 26 16:00:40.733513 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.733366    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c0cc81a8-fc3f-4188-8e90-4640d751dfb5-kubelet-config\") pod \"global-pull-secret-syncer-wc8kh\" (UID: \"c0cc81a8-fc3f-4188-8e90-4640d751dfb5\") " pod="kube-system/global-pull-secret-syncer-wc8kh"
May 26 16:00:40.733513 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.733372    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/391025fc-21eb-4786-8a0b-7f81c9eb90a4-cnibin\") pod \"multus-additional-cni-plugins-9znf9\" (UID: \"391025fc-21eb-4786-8a0b-7f81c9eb90a4\") " pod="openshift-multus/multus-additional-cni-plugins-9znf9"
May 26 16:00:40.733513 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.733393    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b75ad1e4-d3a4-4f24-bc06-3ef6661e1b3d-sys-fs\") pod \"aws-ebs-csi-driver-node-pmgqs\" (UID: \"b75ad1e4-d3a4-4f24-bc06-3ef6661e1b3d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pmgqs"
May 26 16:00:40.733513 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.733446    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/42e4c880-fbee-4786-bbe5-7004126a87be-etc-sysconfig\") pod \"tuned-gh2df\" (UID: \"42e4c880-fbee-4786-bbe5-7004126a87be\") " pod="openshift-cluster-node-tuning-operator/tuned-gh2df"
May 26 16:00:40.733513 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.733497    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/42e4c880-fbee-4786-bbe5-7004126a87be-sys\") pod \"tuned-gh2df\" (UID: \"42e4c880-fbee-4786-bbe5-7004126a87be\") " pod="openshift-cluster-node-tuning-operator/tuned-gh2df"
May 26 16:00:40.733513 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.733515    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/42e4c880-fbee-4786-bbe5-7004126a87be-etc-sysconfig\") pod \"tuned-gh2df\" (UID: \"42e4c880-fbee-4786-bbe5-7004126a87be\") " pod="openshift-cluster-node-tuning-operator/tuned-gh2df"
May 26 16:00:40.733924 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.733520    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/42e4c880-fbee-4786-bbe5-7004126a87be-host\") pod \"tuned-gh2df\" (UID: \"42e4c880-fbee-4786-bbe5-7004126a87be\") " pod="openshift-cluster-node-tuning-operator/tuned-gh2df"
May 26 16:00:40.733924 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.733456    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b75ad1e4-d3a4-4f24-bc06-3ef6661e1b3d-sys-fs\") pod \"aws-ebs-csi-driver-node-pmgqs\" (UID: \"b75ad1e4-d3a4-4f24-bc06-3ef6661e1b3d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pmgqs"
May 26 16:00:40.733924 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.733562    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/42e4c880-fbee-4786-bbe5-7004126a87be-tmp\") pod \"tuned-gh2df\" (UID: \"42e4c880-fbee-4786-bbe5-7004126a87be\") " pod="openshift-cluster-node-tuning-operator/tuned-gh2df"
May 26 16:00:40.733924 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.733571    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/42e4c880-fbee-4786-bbe5-7004126a87be-host\") pod \"tuned-gh2df\" (UID: \"42e4c880-fbee-4786-bbe5-7004126a87be\") " pod="openshift-cluster-node-tuning-operator/tuned-gh2df"
May 26 16:00:40.733924 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.733600    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4887c0d3-11fb-4287-bb94-a876a6c38803-run-ovn\") pod \"ovnkube-node-sk7ns\" (UID: \"4887c0d3-11fb-4287-bb94-a876a6c38803\") " pod="openshift-ovn-kubernetes/ovnkube-node-sk7ns"
May 26 16:00:40.733924 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.733628    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4887c0d3-11fb-4287-bb94-a876a6c38803-env-overrides\") pod \"ovnkube-node-sk7ns\" (UID: \"4887c0d3-11fb-4287-bb94-a876a6c38803\") " pod="openshift-ovn-kubernetes/ovnkube-node-sk7ns"
May 26 16:00:40.733924 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.733637    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/42e4c880-fbee-4786-bbe5-7004126a87be-sys\") pod \"tuned-gh2df\" (UID: \"42e4c880-fbee-4786-bbe5-7004126a87be\") " pod="openshift-cluster-node-tuning-operator/tuned-gh2df"
May 26 16:00:40.733924 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.733655    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6b5a0115-0b0c-4d86-8e58-732e7814f4c9-host-var-lib-kubelet\") pod \"multus-zc768\" (UID: \"6b5a0115-0b0c-4d86-8e58-732e7814f4c9\") " pod="openshift-multus/multus-zc768"
May 26 16:00:40.733924 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.733680    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6b5a0115-0b0c-4d86-8e58-732e7814f4c9-multus-daemon-config\") pod \"multus-zc768\" (UID: \"6b5a0115-0b0c-4d86-8e58-732e7814f4c9\") " pod="openshift-multus/multus-zc768"
May 26 16:00:40.733924 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.733701    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b75ad1e4-d3a4-4f24-bc06-3ef6661e1b3d-device-dir\") pod \"aws-ebs-csi-driver-node-pmgqs\" (UID: \"b75ad1e4-d3a4-4f24-bc06-3ef6661e1b3d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pmgqs"
May 26 16:00:40.733924 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.733726    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hzr5\" (UniqueName: \"kubernetes.io/projected/4887c0d3-11fb-4287-bb94-a876a6c38803-kube-api-access-6hzr5\") pod \"ovnkube-node-sk7ns\" (UID: \"4887c0d3-11fb-4287-bb94-a876a6c38803\") " pod="openshift-ovn-kubernetes/ovnkube-node-sk7ns"
May 26 16:00:40.733924 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.733772    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b6c5e632-79b2-4174-b806-a97342c29687-host-slash\") pod \"iptables-alerter-s4dxt\" (UID: \"b6c5e632-79b2-4174-b806-a97342c29687\") " pod="openshift-network-operator/iptables-alerter-s4dxt"
May 26 16:00:40.733924 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.733796    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/42e4c880-fbee-4786-bbe5-7004126a87be-etc-sysctl-d\") pod \"tuned-gh2df\" (UID: \"42e4c880-fbee-4786-bbe5-7004126a87be\") " pod="openshift-cluster-node-tuning-operator/tuned-gh2df"
May 26 16:00:40.733924 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.733806    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6b5a0115-0b0c-4d86-8e58-732e7814f4c9-host-var-lib-kubelet\") pod \"multus-zc768\" (UID: \"6b5a0115-0b0c-4d86-8e58-732e7814f4c9\") " pod="openshift-multus/multus-zc768"
May 26 16:00:40.733924 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.733818    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6b5a0115-0b0c-4d86-8e58-732e7814f4c9-cni-binary-copy\") pod \"multus-zc768\" (UID: \"6b5a0115-0b0c-4d86-8e58-732e7814f4c9\") " pod="openshift-multus/multus-zc768"
May 26 16:00:40.733924 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.733770    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b75ad1e4-d3a4-4f24-bc06-3ef6661e1b3d-device-dir\") pod \"aws-ebs-csi-driver-node-pmgqs\" (UID: \"b75ad1e4-d3a4-4f24-bc06-3ef6661e1b3d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pmgqs"
May 26 16:00:40.733924 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.733843    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6b5a0115-0b0c-4d86-8e58-732e7814f4c9-multus-socket-dir-parent\") pod \"multus-zc768\" (UID: \"6b5a0115-0b0c-4d86-8e58-732e7814f4c9\") " pod="openshift-multus/multus-zc768"
May 26 16:00:40.734707 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.733871    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6b5a0115-0b0c-4d86-8e58-732e7814f4c9-host-var-lib-cni-bin\") pod \"multus-zc768\" (UID: \"6b5a0115-0b0c-4d86-8e58-732e7814f4c9\") " pod="openshift-multus/multus-zc768"
May 26 16:00:40.734707 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.733894    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/42e4c880-fbee-4786-bbe5-7004126a87be-etc-modprobe-d\") pod \"tuned-gh2df\" (UID: \"42e4c880-fbee-4786-bbe5-7004126a87be\") " pod="openshift-cluster-node-tuning-operator/tuned-gh2df"
May 26 16:00:40.734707 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.733912    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6b5a0115-0b0c-4d86-8e58-732e7814f4c9-multus-socket-dir-parent\") pod \"multus-zc768\" (UID: \"6b5a0115-0b0c-4d86-8e58-732e7814f4c9\") " pod="openshift-multus/multus-zc768"
May 26 16:00:40.734707 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.733919    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/42e4c880-fbee-4786-bbe5-7004126a87be-etc-systemd\") pod \"tuned-gh2df\" (UID: \"42e4c880-fbee-4786-bbe5-7004126a87be\") " pod="openshift-cluster-node-tuning-operator/tuned-gh2df"
May 26 16:00:40.734707 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.733965    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/42e4c880-fbee-4786-bbe5-7004126a87be-etc-sysctl-d\") pod \"tuned-gh2df\" (UID: \"42e4c880-fbee-4786-bbe5-7004126a87be\") " pod="openshift-cluster-node-tuning-operator/tuned-gh2df"
May 26 16:00:40.734707 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.733968    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6b5a0115-0b0c-4d86-8e58-732e7814f4c9-host-var-lib-cni-bin\") pod \"multus-zc768\" (UID: \"6b5a0115-0b0c-4d86-8e58-732e7814f4c9\") " pod="openshift-multus/multus-zc768"
May 26 16:00:40.734707 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.733976    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-jlfs9\" (UniqueName: \"kubernetes.io/projected/bc458685-6108-4373-aa17-f6e49f2c6e1b-kube-api-access-jlfs9\") pod \"node-ca-lrkmg\" (UID: \"bc458685-6108-4373-aa17-f6e49f2c6e1b\") " pod="openshift-image-registry/node-ca-lrkmg"
May 26 16:00:40.734707 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.734010    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/42e4c880-fbee-4786-bbe5-7004126a87be-etc-modprobe-d\") pod \"tuned-gh2df\" (UID: \"42e4c880-fbee-4786-bbe5-7004126a87be\") " pod="openshift-cluster-node-tuning-operator/tuned-gh2df"
May 26 16:00:40.734707 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.734012    2556 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
May 26 16:00:40.734707 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.734027    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4887c0d3-11fb-4287-bb94-a876a6c38803-run-systemd\") pod \"ovnkube-node-sk7ns\" (UID: \"4887c0d3-11fb-4287-bb94-a876a6c38803\") " pod="openshift-ovn-kubernetes/ovnkube-node-sk7ns"
May 26 16:00:40.734707 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.734055    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6b5a0115-0b0c-4d86-8e58-732e7814f4c9-host-var-lib-cni-multus\") pod \"multus-zc768\" (UID: \"6b5a0115-0b0c-4d86-8e58-732e7814f4c9\") " pod="openshift-multus/multus-zc768"
May 26 16:00:40.734707 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.734065    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/42e4c880-fbee-4786-bbe5-7004126a87be-etc-systemd\") pod \"tuned-gh2df\" (UID: \"42e4c880-fbee-4786-bbe5-7004126a87be\") " pod="openshift-cluster-node-tuning-operator/tuned-gh2df"
May 26 16:00:40.734707 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.734081    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/02507d1e-2665-4380-aa87-3605e4411561-agent-certs\") pod \"konnectivity-agent-b2npd\" (UID: \"02507d1e-2665-4380-aa87-3605e4411561\") " pod="kube-system/konnectivity-agent-b2npd"
May 26 16:00:40.734707 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.734122    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/02507d1e-2665-4380-aa87-3605e4411561-konnectivity-ca\") pod \"konnectivity-agent-b2npd\" (UID: \"02507d1e-2665-4380-aa87-3605e4411561\") " pod="kube-system/konnectivity-agent-b2npd"
May 26 16:00:40.734707 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.734136    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6b5a0115-0b0c-4d86-8e58-732e7814f4c9-host-var-lib-cni-multus\") pod \"multus-zc768\" (UID: \"6b5a0115-0b0c-4d86-8e58-732e7814f4c9\") " pod="openshift-multus/multus-zc768"
May 26 16:00:40.734707 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.734144    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/42e4c880-fbee-4786-bbe5-7004126a87be-var-lib-kubelet\") pod \"tuned-gh2df\" (UID: \"42e4c880-fbee-4786-bbe5-7004126a87be\") " pod="openshift-cluster-node-tuning-operator/tuned-gh2df"
May 26 16:00:40.734707 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.734167    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-tz72f\" (UniqueName: \"kubernetes.io/projected/391025fc-21eb-4786-8a0b-7f81c9eb90a4-kube-api-access-tz72f\") pod \"multus-additional-cni-plugins-9znf9\" (UID: \"391025fc-21eb-4786-8a0b-7f81c9eb90a4\") " pod="openshift-multus/multus-additional-cni-plugins-9znf9"
May 26 16:00:40.734707 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.734196    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdxbg\" (UniqueName: \"kubernetes.io/projected/115c1521-6997-403b-98d4-38a9198dd28e-kube-api-access-xdxbg\") pod \"network-check-target-twfp8\" (UID: \"115c1521-6997-403b-98d4-38a9198dd28e\") " pod="openshift-network-diagnostics/network-check-target-twfp8"
May 26 16:00:40.735552 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.734223    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4887c0d3-11fb-4287-bb94-a876a6c38803-host-kubelet\") pod \"ovnkube-node-sk7ns\" (UID: \"4887c0d3-11fb-4287-bb94-a876a6c38803\") " pod="openshift-ovn-kubernetes/ovnkube-node-sk7ns"
May 26 16:00:40.735552 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.734232    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/42e4c880-fbee-4786-bbe5-7004126a87be-var-lib-kubelet\") pod \"tuned-gh2df\" (UID: \"42e4c880-fbee-4786-bbe5-7004126a87be\") " pod="openshift-cluster-node-tuning-operator/tuned-gh2df"
May 26 16:00:40.735552 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.734248    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4887c0d3-11fb-4287-bb94-a876a6c38803-var-lib-openvswitch\") pod \"ovnkube-node-sk7ns\" (UID: \"4887c0d3-11fb-4287-bb94-a876a6c38803\") " pod="openshift-ovn-kubernetes/ovnkube-node-sk7ns"
May 26 16:00:40.735552 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.734275    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/391025fc-21eb-4786-8a0b-7f81c9eb90a4-cni-binary-copy\") pod \"multus-additional-cni-plugins-9znf9\" (UID: \"391025fc-21eb-4786-8a0b-7f81c9eb90a4\") " pod="openshift-multus/multus-additional-cni-plugins-9znf9"
May 26 16:00:40.735552 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.734298    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/391025fc-21eb-4786-8a0b-7f81c9eb90a4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9znf9\" (UID: \"391025fc-21eb-4786-8a0b-7f81c9eb90a4\") " pod="openshift-multus/multus-additional-cni-plugins-9znf9"
May 26 16:00:40.735552 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.734325    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4887c0d3-11fb-4287-bb94-a876a6c38803-host-cni-bin\") pod \"ovnkube-node-sk7ns\" (UID: \"4887c0d3-11fb-4287-bb94-a876a6c38803\") " pod="openshift-ovn-kubernetes/ovnkube-node-sk7ns"
May 26 16:00:40.735552 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.734357    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4887c0d3-11fb-4287-bb94-a876a6c38803-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-sk7ns\" (UID: \"4887c0d3-11fb-4287-bb94-a876a6c38803\") " pod="openshift-ovn-kubernetes/ovnkube-node-sk7ns"
May 26 16:00:40.735552 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.734367    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6b5a0115-0b0c-4d86-8e58-732e7814f4c9-multus-daemon-config\") pod \"multus-zc768\" (UID: \"6b5a0115-0b0c-4d86-8e58-732e7814f4c9\") " pod="openshift-multus/multus-zc768"
May 26 16:00:40.735552 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.734383    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-kgm45\" (UniqueName: \"kubernetes.io/projected/6b5a0115-0b0c-4d86-8e58-732e7814f4c9-kube-api-access-kgm45\") pod \"multus-zc768\" (UID: \"6b5a0115-0b0c-4d86-8e58-732e7814f4c9\") " pod="openshift-multus/multus-zc768"
May 26 16:00:40.735552 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.734409    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/42e4c880-fbee-4786-bbe5-7004126a87be-etc-tuned\") pod \"tuned-gh2df\" (UID: \"42e4c880-fbee-4786-bbe5-7004126a87be\") " pod="openshift-cluster-node-tuning-operator/tuned-gh2df"
May 26 16:00:40.735552 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.734432    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4887c0d3-11fb-4287-bb94-a876a6c38803-log-socket\") pod \"ovnkube-node-sk7ns\" (UID: \"4887c0d3-11fb-4287-bb94-a876a6c38803\") " pod="openshift-ovn-kubernetes/ovnkube-node-sk7ns"
May 26 16:00:40.735552 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.734454    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c0cc81a8-fc3f-4188-8e90-4640d751dfb5-original-pull-secret\") pod \"global-pull-secret-syncer-wc8kh\" (UID: \"c0cc81a8-fc3f-4188-8e90-4640d751dfb5\") " pod="kube-system/global-pull-secret-syncer-wc8kh"
May 26 16:00:40.735552 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.734479    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6b5a0115-0b0c-4d86-8e58-732e7814f4c9-cnibin\") pod \"multus-zc768\" (UID: \"6b5a0115-0b0c-4d86-8e58-732e7814f4c9\") " pod="openshift-multus/multus-zc768"
May 26 16:00:40.735552 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.734499    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6b5a0115-0b0c-4d86-8e58-732e7814f4c9-cni-binary-copy\") pod \"multus-zc768\" (UID: \"6b5a0115-0b0c-4d86-8e58-732e7814f4c9\") " pod="openshift-multus/multus-zc768"
May 26 16:00:40.735552 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.734623    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/42e4c880-fbee-4786-bbe5-7004126a87be-etc-sysctl-conf\") pod \"tuned-gh2df\" (UID: \"42e4c880-fbee-4786-bbe5-7004126a87be\") " pod="openshift-cluster-node-tuning-operator/tuned-gh2df"
May 26 16:00:40.735552 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.734773    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/02507d1e-2665-4380-aa87-3605e4411561-konnectivity-ca\") pod \"konnectivity-agent-b2npd\" (UID: \"02507d1e-2665-4380-aa87-3605e4411561\") " pod="kube-system/konnectivity-agent-b2npd"
May 26 16:00:40.735552 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.734503    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/42e4c880-fbee-4786-bbe5-7004126a87be-etc-sysctl-conf\") pod \"tuned-gh2df\" (UID: \"42e4c880-fbee-4786-bbe5-7004126a87be\") " pod="openshift-cluster-node-tuning-operator/tuned-gh2df"
May 26 16:00:40.736316 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.734837    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-mw64c\" (UniqueName: \"kubernetes.io/projected/b75ad1e4-d3a4-4f24-bc06-3ef6661e1b3d-kube-api-access-mw64c\") pod \"aws-ebs-csi-driver-node-pmgqs\" (UID: \"b75ad1e4-d3a4-4f24-bc06-3ef6661e1b3d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pmgqs"
May 26 16:00:40.736316 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.734864    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6b5a0115-0b0c-4d86-8e58-732e7814f4c9-system-cni-dir\") pod \"multus-zc768\" (UID: \"6b5a0115-0b0c-4d86-8e58-732e7814f4c9\") " pod="openshift-multus/multus-zc768"
May 26 16:00:40.736316 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.734891    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/391025fc-21eb-4786-8a0b-7f81c9eb90a4-cni-binary-copy\") pod \"multus-additional-cni-plugins-9znf9\" (UID: \"391025fc-21eb-4786-8a0b-7f81c9eb90a4\") " pod="openshift-multus/multus-additional-cni-plugins-9znf9"
May 26 16:00:40.736316 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.734908    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6b5a0115-0b0c-4d86-8e58-732e7814f4c9-multus-conf-dir\") pod \"multus-zc768\" (UID: \"6b5a0115-0b0c-4d86-8e58-732e7814f4c9\") " pod="openshift-multus/multus-zc768"
May 26 16:00:40.736316 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.734955    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6b5a0115-0b0c-4d86-8e58-732e7814f4c9-multus-conf-dir\") pod \"multus-zc768\" (UID: \"6b5a0115-0b0c-4d86-8e58-732e7814f4c9\") " pod="openshift-multus/multus-zc768"
May 26 16:00:40.736316 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.735055    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/391025fc-21eb-4786-8a0b-7f81c9eb90a4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9znf9\" (UID: \"391025fc-21eb-4786-8a0b-7f81c9eb90a4\") " pod="openshift-multus/multus-additional-cni-plugins-9znf9"
May 26 16:00:40.736316 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.735145    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6b5a0115-0b0c-4d86-8e58-732e7814f4c9-system-cni-dir\") pod \"multus-zc768\" (UID: \"6b5a0115-0b0c-4d86-8e58-732e7814f4c9\") " pod="openshift-multus/multus-zc768"
May 26 16:00:40.736316 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.735170    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/42e4c880-fbee-4786-bbe5-7004126a87be-etc-kubernetes\") pod \"tuned-gh2df\" (UID: \"42e4c880-fbee-4786-bbe5-7004126a87be\") " pod="openshift-cluster-node-tuning-operator/tuned-gh2df"
May 26 16:00:40.736316 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.735202    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b75ad1e4-d3a4-4f24-bc06-3ef6661e1b3d-socket-dir\") pod \"aws-ebs-csi-driver-node-pmgqs\" (UID: \"b75ad1e4-d3a4-4f24-bc06-3ef6661e1b3d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pmgqs"
May 26 16:00:40.736316 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.735230    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6b5a0115-0b0c-4d86-8e58-732e7814f4c9-host-run-k8s-cni-cncf-io\") pod \"multus-zc768\" (UID: \"6b5a0115-0b0c-4d86-8e58-732e7814f4c9\") " pod="openshift-multus/multus-zc768"
May 26 16:00:40.736316 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.735258    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/42e4c880-fbee-4786-bbe5-7004126a87be-etc-kubernetes\") pod \"tuned-gh2df\" (UID: \"42e4c880-fbee-4786-bbe5-7004126a87be\") " pod="openshift-cluster-node-tuning-operator/tuned-gh2df"
May 26 16:00:40.736316 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.735204    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6b5a0115-0b0c-4d86-8e58-732e7814f4c9-cnibin\") pod \"multus-zc768\" (UID: \"6b5a0115-0b0c-4d86-8e58-732e7814f4c9\") " pod="openshift-multus/multus-zc768"
May 26 16:00:40.736316 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.735278    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6b5a0115-0b0c-4d86-8e58-732e7814f4c9-host-run-k8s-cni-cncf-io\") pod \"multus-zc768\" (UID: \"6b5a0115-0b0c-4d86-8e58-732e7814f4c9\") " pod="openshift-multus/multus-zc768"
May 26 16:00:40.736316 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.735301    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6b5a0115-0b0c-4d86-8e58-732e7814f4c9-host-run-netns\") pod \"multus-zc768\" (UID: \"6b5a0115-0b0c-4d86-8e58-732e7814f4c9\") " pod="openshift-multus/multus-zc768"
May 26 16:00:40.736316 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.735328    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bc458685-6108-4373-aa17-f6e49f2c6e1b-host\") pod \"node-ca-lrkmg\" (UID: \"bc458685-6108-4373-aa17-f6e49f2c6e1b\") " pod="openshift-image-registry/node-ca-lrkmg"
May 26 16:00:40.736316 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.735337    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b75ad1e4-d3a4-4f24-bc06-3ef6661e1b3d-socket-dir\") pod \"aws-ebs-csi-driver-node-pmgqs\" (UID: \"b75ad1e4-d3a4-4f24-bc06-3ef6661e1b3d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pmgqs"
May 26 16:00:40.736316 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.735352    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b75ad1e4-d3a4-4f24-bc06-3ef6661e1b3d-etc-selinux\") pod \"aws-ebs-csi-driver-node-pmgqs\" (UID: \"b75ad1e4-d3a4-4f24-bc06-3ef6661e1b3d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pmgqs"
May 26 16:00:40.736316 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.735372    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6b5a0115-0b0c-4d86-8e58-732e7814f4c9-host-run-netns\") pod \"multus-zc768\" (UID: \"6b5a0115-0b0c-4d86-8e58-732e7814f4c9\") " pod="openshift-multus/multus-zc768"
May 26 16:00:40.737052 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.735382    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4887c0d3-11fb-4287-bb94-a876a6c38803-ovnkube-script-lib\") pod \"ovnkube-node-sk7ns\" (UID: \"4887c0d3-11fb-4287-bb94-a876a6c38803\") " pod="openshift-ovn-kubernetes/ovnkube-node-sk7ns"
May 26 16:00:40.737052 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.735386    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bc458685-6108-4373-aa17-f6e49f2c6e1b-host\") pod \"node-ca-lrkmg\" (UID: \"bc458685-6108-4373-aa17-f6e49f2c6e1b\") " pod="openshift-image-registry/node-ca-lrkmg"
May 26 16:00:40.737052 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.735412    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6b5a0115-0b0c-4d86-8e58-732e7814f4c9-hostroot\") pod \"multus-zc768\" (UID: \"6b5a0115-0b0c-4d86-8e58-732e7814f4c9\") " pod="openshift-multus/multus-zc768"
May 26 16:00:40.737052 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.735431    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b75ad1e4-d3a4-4f24-bc06-3ef6661e1b3d-etc-selinux\") pod \"aws-ebs-csi-driver-node-pmgqs\" (UID: \"b75ad1e4-d3a4-4f24-bc06-3ef6661e1b3d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pmgqs"
May 26 16:00:40.737052 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.735438    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aed39145-780f-4642-af53-ddccbc03c34c-metrics-certs\") pod \"network-metrics-daemon-g9scb\" (UID: \"aed39145-780f-4642-af53-ddccbc03c34c\") " pod="openshift-multus/network-metrics-daemon-g9scb"
May 26 16:00:40.737052 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.735444    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6b5a0115-0b0c-4d86-8e58-732e7814f4c9-hostroot\") pod \"multus-zc768\" (UID: \"6b5a0115-0b0c-4d86-8e58-732e7814f4c9\") " pod="openshift-multus/multus-zc768"
May 26 16:00:40.737052 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.735472    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/bc458685-6108-4373-aa17-f6e49f2c6e1b-serviceca\") pod \"node-ca-lrkmg\" (UID: \"bc458685-6108-4373-aa17-f6e49f2c6e1b\") " pod="openshift-image-registry/node-ca-lrkmg"
May 26 16:00:40.737052 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.735501    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4887c0d3-11fb-4287-bb94-a876a6c38803-etc-openvswitch\") pod \"ovnkube-node-sk7ns\" (UID: \"4887c0d3-11fb-4287-bb94-a876a6c38803\") " pod="openshift-ovn-kubernetes/ovnkube-node-sk7ns"
May 26 16:00:40.737052 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:00:40.735516    2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 26 16:00:40.737052 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.735535    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4887c0d3-11fb-4287-bb94-a876a6c38803-host-cni-netd\") pod \"ovnkube-node-sk7ns\" (UID: \"4887c0d3-11fb-4287-bb94-a876a6c38803\") " pod="openshift-ovn-kubernetes/ovnkube-node-sk7ns"
May 26 16:00:40.737052 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.735559    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4887c0d3-11fb-4287-bb94-a876a6c38803-ovnkube-config\") pod \"ovnkube-node-sk7ns\" (UID: \"4887c0d3-11fb-4287-bb94-a876a6c38803\") " pod="openshift-ovn-kubernetes/ovnkube-node-sk7ns"
May 26 16:00:40.737052 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:00:40.735609    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aed39145-780f-4642-af53-ddccbc03c34c-metrics-certs podName:aed39145-780f-4642-af53-ddccbc03c34c nodeName:}" failed. No retries permitted until 2026-05-26 16:00:41.235564924 +0000 UTC m=+3.080181915 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/aed39145-780f-4642-af53-ddccbc03c34c-metrics-certs") pod "network-metrics-daemon-g9scb" (UID: "aed39145-780f-4642-af53-ddccbc03c34c") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 26 16:00:40.737052 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.735650    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c0cc81a8-fc3f-4188-8e90-4640d751dfb5-dbus\") pod \"global-pull-secret-syncer-wc8kh\" (UID: \"c0cc81a8-fc3f-4188-8e90-4640d751dfb5\") " pod="kube-system/global-pull-secret-syncer-wc8kh"
May 26 16:00:40.737052 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.735686    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6b5a0115-0b0c-4d86-8e58-732e7814f4c9-multus-cni-dir\") pod \"multus-zc768\" (UID: \"6b5a0115-0b0c-4d86-8e58-732e7814f4c9\") " pod="openshift-multus/multus-zc768"
May 26 16:00:40.737052 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.735713    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6b5a0115-0b0c-4d86-8e58-732e7814f4c9-etc-kubernetes\") pod \"multus-zc768\" (UID: \"6b5a0115-0b0c-4d86-8e58-732e7814f4c9\") " pod="openshift-multus/multus-zc768"
May 26 16:00:40.737052 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.735752    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-p458s\" (UniqueName: \"kubernetes.io/projected/42e4c880-fbee-4786-bbe5-7004126a87be-kube-api-access-p458s\") pod \"tuned-gh2df\" (UID: \"42e4c880-fbee-4786-bbe5-7004126a87be\") " pod="openshift-cluster-node-tuning-operator/tuned-gh2df"
May 26 16:00:40.737052 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.735788    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b75ad1e4-d3a4-4f24-bc06-3ef6661e1b3d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-pmgqs\" (UID: \"b75ad1e4-d3a4-4f24-bc06-3ef6661e1b3d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pmgqs"
May 26 16:00:40.737709 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.735813    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/391025fc-21eb-4786-8a0b-7f81c9eb90a4-os-release\") pod \"multus-additional-cni-plugins-9znf9\" (UID: \"391025fc-21eb-4786-8a0b-7f81c9eb90a4\") " pod="openshift-multus/multus-additional-cni-plugins-9znf9"
May 26 16:00:40.737709 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.735850    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b6c5e632-79b2-4174-b806-a97342c29687-iptables-alerter-script\") pod \"iptables-alerter-s4dxt\" (UID: \"b6c5e632-79b2-4174-b806-a97342c29687\") " pod="openshift-network-operator/iptables-alerter-s4dxt"
May 26 16:00:40.737709 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.735848    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6b5a0115-0b0c-4d86-8e58-732e7814f4c9-multus-cni-dir\") pod \"multus-zc768\" (UID: \"6b5a0115-0b0c-4d86-8e58-732e7814f4c9\") " pod="openshift-multus/multus-zc768"
May 26 16:00:40.737709 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.735898    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6b5a0115-0b0c-4d86-8e58-732e7814f4c9-etc-kubernetes\") pod \"multus-zc768\" (UID: \"6b5a0115-0b0c-4d86-8e58-732e7814f4c9\") " pod="openshift-multus/multus-zc768"
May 26 16:00:40.737709 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.735927    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4887c0d3-11fb-4287-bb94-a876a6c38803-node-log\") pod \"ovnkube-node-sk7ns\" (UID: \"4887c0d3-11fb-4287-bb94-a876a6c38803\") " pod="openshift-ovn-kubernetes/ovnkube-node-sk7ns"
May 26 16:00:40.737709 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.735959    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/bc458685-6108-4373-aa17-f6e49f2c6e1b-serviceca\") pod \"node-ca-lrkmg\" (UID: \"bc458685-6108-4373-aa17-f6e49f2c6e1b\") " pod="openshift-image-registry/node-ca-lrkmg"
May 26 16:00:40.737709 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.735970    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-fbwzt\" (UniqueName: \"kubernetes.io/projected/aed39145-780f-4642-af53-ddccbc03c34c-kube-api-access-fbwzt\") pod \"network-metrics-daemon-g9scb\" (UID: \"aed39145-780f-4642-af53-ddccbc03c34c\") " pod="openshift-multus/network-metrics-daemon-g9scb"
May 26 16:00:40.737709 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.736025    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/42e4c880-fbee-4786-bbe5-7004126a87be-lib-modules\") pod \"tuned-gh2df\" (UID: \"42e4c880-fbee-4786-bbe5-7004126a87be\") " pod="openshift-cluster-node-tuning-operator/tuned-gh2df"
May 26 16:00:40.737709 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.736047    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/391025fc-21eb-4786-8a0b-7f81c9eb90a4-os-release\") pod \"multus-additional-cni-plugins-9znf9\" (UID: \"391025fc-21eb-4786-8a0b-7f81c9eb90a4\") " pod="openshift-multus/multus-additional-cni-plugins-9znf9"
May 26 16:00:40.737709 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.736069    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4887c0d3-11fb-4287-bb94-a876a6c38803-host-slash\") pod \"ovnkube-node-sk7ns\" (UID: \"4887c0d3-11fb-4287-bb94-a876a6c38803\") " pod="openshift-ovn-kubernetes/ovnkube-node-sk7ns"
May 26 16:00:40.737709 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.736116    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4887c0d3-11fb-4287-bb94-a876a6c38803-host-run-netns\") pod \"ovnkube-node-sk7ns\" (UID: \"4887c0d3-11fb-4287-bb94-a876a6c38803\") " pod="openshift-ovn-kubernetes/ovnkube-node-sk7ns"
May 26 16:00:40.737709 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.736143    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4887c0d3-11fb-4287-bb94-a876a6c38803-ovn-node-metrics-cert\") pod \"ovnkube-node-sk7ns\" (UID: \"4887c0d3-11fb-4287-bb94-a876a6c38803\") " pod="openshift-ovn-kubernetes/ovnkube-node-sk7ns"
May 26 16:00:40.737709 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.736172    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6b5a0115-0b0c-4d86-8e58-732e7814f4c9-host-run-multus-certs\") pod \"multus-zc768\" (UID: \"6b5a0115-0b0c-4d86-8e58-732e7814f4c9\") " pod="openshift-multus/multus-zc768"
May 26 16:00:40.737709 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.736221    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6b5a0115-0b0c-4d86-8e58-732e7814f4c9-host-run-multus-certs\") pod \"multus-zc768\" (UID: \"6b5a0115-0b0c-4d86-8e58-732e7814f4c9\") " pod="openshift-multus/multus-zc768"
May 26 16:00:40.737709 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.736306    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b75ad1e4-d3a4-4f24-bc06-3ef6661e1b3d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-pmgqs\" (UID: \"b75ad1e4-d3a4-4f24-bc06-3ef6661e1b3d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pmgqs"
May 26 16:00:40.737709 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.736311    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/42e4c880-fbee-4786-bbe5-7004126a87be-lib-modules\") pod \"tuned-gh2df\" (UID: \"42e4c880-fbee-4786-bbe5-7004126a87be\") " pod="openshift-cluster-node-tuning-operator/tuned-gh2df"
May 26 16:00:40.737709 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.736394    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/42e4c880-fbee-4786-bbe5-7004126a87be-run\") pod \"tuned-gh2df\" (UID: \"42e4c880-fbee-4786-bbe5-7004126a87be\") " pod="openshift-cluster-node-tuning-operator/tuned-gh2df"
May 26 16:00:40.738274 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.736427    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/391025fc-21eb-4786-8a0b-7f81c9eb90a4-system-cni-dir\") pod \"multus-additional-cni-plugins-9znf9\" (UID: \"391025fc-21eb-4786-8a0b-7f81c9eb90a4\") " pod="openshift-multus/multus-additional-cni-plugins-9znf9"
May 26 16:00:40.738274 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.736430    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/42e4c880-fbee-4786-bbe5-7004126a87be-run\") pod \"tuned-gh2df\" (UID: \"42e4c880-fbee-4786-bbe5-7004126a87be\") " pod="openshift-cluster-node-tuning-operator/tuned-gh2df"
May 26 16:00:40.738274 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.736452    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/391025fc-21eb-4786-8a0b-7f81c9eb90a4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9znf9\" (UID: \"391025fc-21eb-4786-8a0b-7f81c9eb90a4\") " pod="openshift-multus/multus-additional-cni-plugins-9znf9"
May 26 16:00:40.738274 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.736478    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/391025fc-21eb-4786-8a0b-7f81c9eb90a4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9znf9\" (UID: \"391025fc-21eb-4786-8a0b-7f81c9eb90a4\") " pod="openshift-multus/multus-additional-cni-plugins-9znf9"
May 26 16:00:40.738274 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.736500    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/391025fc-21eb-4786-8a0b-7f81c9eb90a4-system-cni-dir\") pod \"multus-additional-cni-plugins-9znf9\" (UID: \"391025fc-21eb-4786-8a0b-7f81c9eb90a4\") " pod="openshift-multus/multus-additional-cni-plugins-9znf9"
May 26 16:00:40.738274 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.736509    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4887c0d3-11fb-4287-bb94-a876a6c38803-systemd-units\") pod \"ovnkube-node-sk7ns\" (UID: \"4887c0d3-11fb-4287-bb94-a876a6c38803\") " pod="openshift-ovn-kubernetes/ovnkube-node-sk7ns"
May 26 16:00:40.738274 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.736534    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6b5a0115-0b0c-4d86-8e58-732e7814f4c9-os-release\") pod \"multus-zc768\" (UID: \"6b5a0115-0b0c-4d86-8e58-732e7814f4c9\") " pod="openshift-multus/multus-zc768"
May 26 16:00:40.738274 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.736563    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b75ad1e4-d3a4-4f24-bc06-3ef6661e1b3d-registration-dir\") pod \"aws-ebs-csi-driver-node-pmgqs\" (UID: \"b75ad1e4-d3a4-4f24-bc06-3ef6661e1b3d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pmgqs"
May 26 16:00:40.738274 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.736619    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/391025fc-21eb-4786-8a0b-7f81c9eb90a4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9znf9\" (UID: \"391025fc-21eb-4786-8a0b-7f81c9eb90a4\") " pod="openshift-multus/multus-additional-cni-plugins-9znf9"
May 26 16:00:40.738274 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.736668    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b75ad1e4-d3a4-4f24-bc06-3ef6661e1b3d-registration-dir\") pod \"aws-ebs-csi-driver-node-pmgqs\" (UID: \"b75ad1e4-d3a4-4f24-bc06-3ef6661e1b3d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pmgqs"
May 26 16:00:40.738274 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.736685    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6b5a0115-0b0c-4d86-8e58-732e7814f4c9-os-release\") pod \"multus-zc768\" (UID: \"6b5a0115-0b0c-4d86-8e58-732e7814f4c9\") " pod="openshift-multus/multus-zc768"
May 26 16:00:40.738274 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.737988    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/02507d1e-2665-4380-aa87-3605e4411561-agent-certs\") pod \"konnectivity-agent-b2npd\" (UID: \"02507d1e-2665-4380-aa87-3605e4411561\") " pod="kube-system/konnectivity-agent-b2npd"
May 26 16:00:40.740818 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.740796    2556 reflector.go:430] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
May 26 16:00:40.744030 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.744004    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/42e4c880-fbee-4786-bbe5-7004126a87be-etc-tuned\") pod \"tuned-gh2df\" (UID: \"42e4c880-fbee-4786-bbe5-7004126a87be\") " pod="openshift-cluster-node-tuning-operator/tuned-gh2df"
May 26 16:00:40.744164 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.744052    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/42e4c880-fbee-4786-bbe5-7004126a87be-tmp\") pod \"tuned-gh2df\" (UID: \"42e4c880-fbee-4786-bbe5-7004126a87be\") " pod="openshift-cluster-node-tuning-operator/tuned-gh2df"
May 26 16:00:40.744227 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.744212    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/391025fc-21eb-4786-8a0b-7f81c9eb90a4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9znf9\" (UID: \"391025fc-21eb-4786-8a0b-7f81c9eb90a4\") " pod="openshift-multus/multus-additional-cni-plugins-9znf9"
May 26 16:00:40.745267 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.745243    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgm45\" (UniqueName: \"kubernetes.io/projected/6b5a0115-0b0c-4d86-8e58-732e7814f4c9-kube-api-access-kgm45\") pod \"multus-zc768\" (UID: \"6b5a0115-0b0c-4d86-8e58-732e7814f4c9\") " pod="openshift-multus/multus-zc768"
May 26 16:00:40.746251 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.746232    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlfs9\" (UniqueName: \"kubernetes.io/projected/bc458685-6108-4373-aa17-f6e49f2c6e1b-kube-api-access-jlfs9\") pod \"node-ca-lrkmg\" (UID: \"bc458685-6108-4373-aa17-f6e49f2c6e1b\") " pod="openshift-image-registry/node-ca-lrkmg"
May 26 16:00:40.750211 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.750186    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tz72f\" (UniqueName: \"kubernetes.io/projected/391025fc-21eb-4786-8a0b-7f81c9eb90a4-kube-api-access-tz72f\") pod \"multus-additional-cni-plugins-9znf9\" (UID: \"391025fc-21eb-4786-8a0b-7f81c9eb90a4\") " pod="openshift-multus/multus-additional-cni-plugins-9znf9"
May 26 16:00:40.760400 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.760375    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mw64c\" (UniqueName: \"kubernetes.io/projected/b75ad1e4-d3a4-4f24-bc06-3ef6661e1b3d-kube-api-access-mw64c\") pod \"aws-ebs-csi-driver-node-pmgqs\" (UID: \"b75ad1e4-d3a4-4f24-bc06-3ef6661e1b3d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pmgqs"
May 26 16:00:40.760506 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.760400    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p458s\" (UniqueName: \"kubernetes.io/projected/42e4c880-fbee-4786-bbe5-7004126a87be-kube-api-access-p458s\") pod \"tuned-gh2df\" (UID: \"42e4c880-fbee-4786-bbe5-7004126a87be\") " pod="openshift-cluster-node-tuning-operator/tuned-gh2df"
May 26 16:00:40.761575 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.761555    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbwzt\" (UniqueName: \"kubernetes.io/projected/aed39145-780f-4642-af53-ddccbc03c34c-kube-api-access-fbwzt\") pod \"network-metrics-daemon-g9scb\" (UID: \"aed39145-780f-4642-af53-ddccbc03c34c\") " pod="openshift-multus/network-metrics-daemon-g9scb"
May 26 16:00:40.837749 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.837715    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4887c0d3-11fb-4287-bb94-a876a6c38803-systemd-units\") pod \"ovnkube-node-sk7ns\" (UID: \"4887c0d3-11fb-4287-bb94-a876a6c38803\") " pod="openshift-ovn-kubernetes/ovnkube-node-sk7ns"
May 26 16:00:40.837927 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.837761    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-9pvlm\" (UniqueName: \"kubernetes.io/projected/b6c5e632-79b2-4174-b806-a97342c29687-kube-api-access-9pvlm\") pod \"iptables-alerter-s4dxt\" (UID: \"b6c5e632-79b2-4174-b806-a97342c29687\") " pod="openshift-network-operator/iptables-alerter-s4dxt"
May 26 16:00:40.837927 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.837807    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4887c0d3-11fb-4287-bb94-a876a6c38803-run-openvswitch\") pod \"ovnkube-node-sk7ns\" (UID: \"4887c0d3-11fb-4287-bb94-a876a6c38803\") " pod="openshift-ovn-kubernetes/ovnkube-node-sk7ns"
May 26 16:00:40.837927 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.837825    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4887c0d3-11fb-4287-bb94-a876a6c38803-systemd-units\") pod \"ovnkube-node-sk7ns\" (UID: \"4887c0d3-11fb-4287-bb94-a876a6c38803\") " pod="openshift-ovn-kubernetes/ovnkube-node-sk7ns"
May 26 16:00:40.837927 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.837833    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4887c0d3-11fb-4287-bb94-a876a6c38803-host-run-ovn-kubernetes\") pod \"ovnkube-node-sk7ns\" (UID: \"4887c0d3-11fb-4287-bb94-a876a6c38803\") " pod="openshift-ovn-kubernetes/ovnkube-node-sk7ns"
May 26 16:00:40.837927 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.837875    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4887c0d3-11fb-4287-bb94-a876a6c38803-run-openvswitch\") pod \"ovnkube-node-sk7ns\" (UID: \"4887c0d3-11fb-4287-bb94-a876a6c38803\") " pod="openshift-ovn-kubernetes/ovnkube-node-sk7ns"
May 26 16:00:40.837927 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.837877    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4887c0d3-11fb-4287-bb94-a876a6c38803-host-run-ovn-kubernetes\") pod \"ovnkube-node-sk7ns\" (UID: \"4887c0d3-11fb-4287-bb94-a876a6c38803\") " pod="openshift-ovn-kubernetes/ovnkube-node-sk7ns"
May 26 16:00:40.837927 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.837903    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c0cc81a8-fc3f-4188-8e90-4640d751dfb5-kubelet-config\") pod \"global-pull-secret-syncer-wc8kh\" (UID: \"c0cc81a8-fc3f-4188-8e90-4640d751dfb5\") " pod="kube-system/global-pull-secret-syncer-wc8kh"
May 26 16:00:40.838269 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.837960    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4887c0d3-11fb-4287-bb94-a876a6c38803-run-ovn\") pod \"ovnkube-node-sk7ns\" (UID: \"4887c0d3-11fb-4287-bb94-a876a6c38803\") " pod="openshift-ovn-kubernetes/ovnkube-node-sk7ns"
May 26 16:00:40.838269 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.837989    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4887c0d3-11fb-4287-bb94-a876a6c38803-env-overrides\") pod \"ovnkube-node-sk7ns\" (UID: \"4887c0d3-11fb-4287-bb94-a876a6c38803\") " pod="openshift-ovn-kubernetes/ovnkube-node-sk7ns"
May 26 16:00:40.838269 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.837988    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c0cc81a8-fc3f-4188-8e90-4640d751dfb5-kubelet-config\") pod \"global-pull-secret-syncer-wc8kh\" (UID: \"c0cc81a8-fc3f-4188-8e90-4640d751dfb5\") " pod="kube-system/global-pull-secret-syncer-wc8kh"
May 26 16:00:40.838269 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.838018    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-6hzr5\" (UniqueName: \"kubernetes.io/projected/4887c0d3-11fb-4287-bb94-a876a6c38803-kube-api-access-6hzr5\") pod \"ovnkube-node-sk7ns\" (UID: \"4887c0d3-11fb-4287-bb94-a876a6c38803\") " pod="openshift-ovn-kubernetes/ovnkube-node-sk7ns"
May 26 16:00:40.838269 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.838030    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4887c0d3-11fb-4287-bb94-a876a6c38803-run-ovn\") pod \"ovnkube-node-sk7ns\" (UID: \"4887c0d3-11fb-4287-bb94-a876a6c38803\") " pod="openshift-ovn-kubernetes/ovnkube-node-sk7ns"
May 26 16:00:40.838269 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.838070    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b6c5e632-79b2-4174-b806-a97342c29687-host-slash\") pod \"iptables-alerter-s4dxt\" (UID: \"b6c5e632-79b2-4174-b806-a97342c29687\") " pod="openshift-network-operator/iptables-alerter-s4dxt"
May 26 16:00:40.838269 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.838121    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4887c0d3-11fb-4287-bb94-a876a6c38803-run-systemd\") pod \"ovnkube-node-sk7ns\" (UID: \"4887c0d3-11fb-4287-bb94-a876a6c38803\") " pod="openshift-ovn-kubernetes/ovnkube-node-sk7ns"
May 26 16:00:40.838269 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.838154    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-xdxbg\" (UniqueName: \"kubernetes.io/projected/115c1521-6997-403b-98d4-38a9198dd28e-kube-api-access-xdxbg\") pod \"network-check-target-twfp8\" (UID: \"115c1521-6997-403b-98d4-38a9198dd28e\") " pod="openshift-network-diagnostics/network-check-target-twfp8"
May 26 16:00:40.838269 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.838179    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4887c0d3-11fb-4287-bb94-a876a6c38803-host-kubelet\") pod \"ovnkube-node-sk7ns\" (UID: \"4887c0d3-11fb-4287-bb94-a876a6c38803\") " pod="openshift-ovn-kubernetes/ovnkube-node-sk7ns"
May 26 16:00:40.838269 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.838204    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4887c0d3-11fb-4287-bb94-a876a6c38803-var-lib-openvswitch\") pod \"ovnkube-node-sk7ns\" (UID: \"4887c0d3-11fb-4287-bb94-a876a6c38803\") " pod="openshift-ovn-kubernetes/ovnkube-node-sk7ns"
May 26 16:00:40.838269 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.838176    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b6c5e632-79b2-4174-b806-a97342c29687-host-slash\") pod \"iptables-alerter-s4dxt\" (UID: \"b6c5e632-79b2-4174-b806-a97342c29687\") " pod="openshift-network-operator/iptables-alerter-s4dxt"
May 26 16:00:40.838269 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.838233    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4887c0d3-11fb-4287-bb94-a876a6c38803-host-cni-bin\") pod \"ovnkube-node-sk7ns\" (UID: \"4887c0d3-11fb-4287-bb94-a876a6c38803\") " pod="openshift-ovn-kubernetes/ovnkube-node-sk7ns"
May 26 16:00:40.838269 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.838259    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4887c0d3-11fb-4287-bb94-a876a6c38803-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-sk7ns\" (UID: \"4887c0d3-11fb-4287-bb94-a876a6c38803\") " pod="openshift-ovn-kubernetes/ovnkube-node-sk7ns"
May 26 16:00:40.838269 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.838263    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4887c0d3-11fb-4287-bb94-a876a6c38803-run-systemd\") pod \"ovnkube-node-sk7ns\" (UID: \"4887c0d3-11fb-4287-bb94-a876a6c38803\") " pod="openshift-ovn-kubernetes/ovnkube-node-sk7ns"
May 26 16:00:40.838269 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.838272    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4887c0d3-11fb-4287-bb94-a876a6c38803-host-kubelet\") pod \"ovnkube-node-sk7ns\" (UID: \"4887c0d3-11fb-4287-bb94-a876a6c38803\") " pod="openshift-ovn-kubernetes/ovnkube-node-sk7ns"
May 26 16:00:40.838935 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.838289    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4887c0d3-11fb-4287-bb94-a876a6c38803-log-socket\") pod \"ovnkube-node-sk7ns\" (UID: \"4887c0d3-11fb-4287-bb94-a876a6c38803\") " pod="openshift-ovn-kubernetes/ovnkube-node-sk7ns"
May 26 16:00:40.838935 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.838312    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4887c0d3-11fb-4287-bb94-a876a6c38803-var-lib-openvswitch\") pod \"ovnkube-node-sk7ns\" (UID: \"4887c0d3-11fb-4287-bb94-a876a6c38803\") " pod="openshift-ovn-kubernetes/ovnkube-node-sk7ns"
May 26 16:00:40.838935 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.838355    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4887c0d3-11fb-4287-bb94-a876a6c38803-host-cni-bin\") pod \"ovnkube-node-sk7ns\" (UID: \"4887c0d3-11fb-4287-bb94-a876a6c38803\") " pod="openshift-ovn-kubernetes/ovnkube-node-sk7ns"
May 26 16:00:40.838935 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.838380    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c0cc81a8-fc3f-4188-8e90-4640d751dfb5-original-pull-secret\") pod \"global-pull-secret-syncer-wc8kh\" (UID: \"c0cc81a8-fc3f-4188-8e90-4640d751dfb5\") " pod="kube-system/global-pull-secret-syncer-wc8kh"
May 26 16:00:40.838935 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.838393    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4887c0d3-11fb-4287-bb94-a876a6c38803-log-socket\") pod \"ovnkube-node-sk7ns\" (UID: \"4887c0d3-11fb-4287-bb94-a876a6c38803\") " pod="openshift-ovn-kubernetes/ovnkube-node-sk7ns"
May 26 16:00:40.838935 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.838412    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4887c0d3-11fb-4287-bb94-a876a6c38803-ovnkube-script-lib\") pod \"ovnkube-node-sk7ns\" (UID: \"4887c0d3-11fb-4287-bb94-a876a6c38803\") " pod="openshift-ovn-kubernetes/ovnkube-node-sk7ns"
May 26 16:00:40.838935 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.838439    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4887c0d3-11fb-4287-bb94-a876a6c38803-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-sk7ns\" (UID: \"4887c0d3-11fb-4287-bb94-a876a6c38803\") " pod="openshift-ovn-kubernetes/ovnkube-node-sk7ns"
May 26 16:00:40.838935 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.838444    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4887c0d3-11fb-4287-bb94-a876a6c38803-etc-openvswitch\") pod \"ovnkube-node-sk7ns\" (UID: \"4887c0d3-11fb-4287-bb94-a876a6c38803\") " pod="openshift-ovn-kubernetes/ovnkube-node-sk7ns"
May 26 16:00:40.838935 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.838480    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4887c0d3-11fb-4287-bb94-a876a6c38803-etc-openvswitch\") pod \"ovnkube-node-sk7ns\" (UID: \"4887c0d3-11fb-4287-bb94-a876a6c38803\") " pod="openshift-ovn-kubernetes/ovnkube-node-sk7ns"
May 26 16:00:40.838935 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.838485    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4887c0d3-11fb-4287-bb94-a876a6c38803-host-cni-netd\") pod \"ovnkube-node-sk7ns\" (UID: \"4887c0d3-11fb-4287-bb94-a876a6c38803\") " pod="openshift-ovn-kubernetes/ovnkube-node-sk7ns"
May 26 16:00:40.838935 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.838485    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4887c0d3-11fb-4287-bb94-a876a6c38803-env-overrides\") pod \"ovnkube-node-sk7ns\" (UID: \"4887c0d3-11fb-4287-bb94-a876a6c38803\") " pod="openshift-ovn-kubernetes/ovnkube-node-sk7ns"
May 26 16:00:40.838935 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.838514    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4887c0d3-11fb-4287-bb94-a876a6c38803-ovnkube-config\") pod \"ovnkube-node-sk7ns\" (UID: \"4887c0d3-11fb-4287-bb94-a876a6c38803\") " pod="openshift-ovn-kubernetes/ovnkube-node-sk7ns"
May 26 16:00:40.838935 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.838524    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4887c0d3-11fb-4287-bb94-a876a6c38803-host-cni-netd\") pod \"ovnkube-node-sk7ns\" (UID: \"4887c0d3-11fb-4287-bb94-a876a6c38803\") " pod="openshift-ovn-kubernetes/ovnkube-node-sk7ns"
May 26 16:00:40.838935 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:00:40.838535    2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
May 26 16:00:40.838935 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.838542    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c0cc81a8-fc3f-4188-8e90-4640d751dfb5-dbus\") pod \"global-pull-secret-syncer-wc8kh\" (UID: \"c0cc81a8-fc3f-4188-8e90-4640d751dfb5\") " pod="kube-system/global-pull-secret-syncer-wc8kh"
May 26 16:00:40.838935 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:00:40.838578    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0cc81a8-fc3f-4188-8e90-4640d751dfb5-original-pull-secret podName:c0cc81a8-fc3f-4188-8e90-4640d751dfb5 nodeName:}" failed. No retries permitted until 2026-05-26 16:00:41.338563782 +0000 UTC m=+3.183180752 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c0cc81a8-fc3f-4188-8e90-4640d751dfb5-original-pull-secret") pod "global-pull-secret-syncer-wc8kh" (UID: "c0cc81a8-fc3f-4188-8e90-4640d751dfb5") : object "kube-system"/"original-pull-secret" not registered
May 26 16:00:40.838935 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.838597    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b6c5e632-79b2-4174-b806-a97342c29687-iptables-alerter-script\") pod \"iptables-alerter-s4dxt\" (UID: \"b6c5e632-79b2-4174-b806-a97342c29687\") " pod="openshift-network-operator/iptables-alerter-s4dxt"
May 26 16:00:40.839793 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.838613    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c0cc81a8-fc3f-4188-8e90-4640d751dfb5-dbus\") pod \"global-pull-secret-syncer-wc8kh\" (UID: \"c0cc81a8-fc3f-4188-8e90-4640d751dfb5\") " pod="kube-system/global-pull-secret-syncer-wc8kh"
May 26 16:00:40.839793 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.838621    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4887c0d3-11fb-4287-bb94-a876a6c38803-node-log\") pod \"ovnkube-node-sk7ns\" (UID: \"4887c0d3-11fb-4287-bb94-a876a6c38803\") " pod="openshift-ovn-kubernetes/ovnkube-node-sk7ns"
May 26 16:00:40.839793 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.838650    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4887c0d3-11fb-4287-bb94-a876a6c38803-host-slash\") pod \"ovnkube-node-sk7ns\" (UID: \"4887c0d3-11fb-4287-bb94-a876a6c38803\") " pod="openshift-ovn-kubernetes/ovnkube-node-sk7ns"
May 26 16:00:40.839793 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.838684    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4887c0d3-11fb-4287-bb94-a876a6c38803-host-slash\") pod \"ovnkube-node-sk7ns\" (UID: \"4887c0d3-11fb-4287-bb94-a876a6c38803\") " pod="openshift-ovn-kubernetes/ovnkube-node-sk7ns"
May 26 16:00:40.839793 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.839122    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4887c0d3-11fb-4287-bb94-a876a6c38803-host-run-netns\") pod \"ovnkube-node-sk7ns\" (UID: \"4887c0d3-11fb-4287-bb94-a876a6c38803\") " pod="openshift-ovn-kubernetes/ovnkube-node-sk7ns"
May 26 16:00:40.839793 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.839166    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4887c0d3-11fb-4287-bb94-a876a6c38803-ovn-node-metrics-cert\") pod \"ovnkube-node-sk7ns\" (UID: \"4887c0d3-11fb-4287-bb94-a876a6c38803\") " pod="openshift-ovn-kubernetes/ovnkube-node-sk7ns"
May 26 16:00:40.839793 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.839256    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4887c0d3-11fb-4287-bb94-a876a6c38803-host-run-netns\") pod \"ovnkube-node-sk7ns\" (UID: \"4887c0d3-11fb-4287-bb94-a876a6c38803\") " pod="openshift-ovn-kubernetes/ovnkube-node-sk7ns"
May 26 16:00:40.839793 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.839325    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4887c0d3-11fb-4287-bb94-a876a6c38803-node-log\") pod \"ovnkube-node-sk7ns\" (UID: \"4887c0d3-11fb-4287-bb94-a876a6c38803\") " pod="openshift-ovn-kubernetes/ovnkube-node-sk7ns"
May 26 16:00:40.839793 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.839413    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4887c0d3-11fb-4287-bb94-a876a6c38803-ovnkube-config\") pod \"ovnkube-node-sk7ns\" (UID: \"4887c0d3-11fb-4287-bb94-a876a6c38803\") " pod="openshift-ovn-kubernetes/ovnkube-node-sk7ns"
May 26 16:00:40.839793 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.839607    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4887c0d3-11fb-4287-bb94-a876a6c38803-ovnkube-script-lib\") pod \"ovnkube-node-sk7ns\" (UID: \"4887c0d3-11fb-4287-bb94-a876a6c38803\") " pod="openshift-ovn-kubernetes/ovnkube-node-sk7ns"
May 26 16:00:40.839793 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.839729    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b6c5e632-79b2-4174-b806-a97342c29687-iptables-alerter-script\") pod \"iptables-alerter-s4dxt\" (UID: \"b6c5e632-79b2-4174-b806-a97342c29687\") " pod="openshift-network-operator/iptables-alerter-s4dxt"
May 26 16:00:40.841964 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.841905    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4887c0d3-11fb-4287-bb94-a876a6c38803-ovn-node-metrics-cert\") pod \"ovnkube-node-sk7ns\" (UID: \"4887c0d3-11fb-4287-bb94-a876a6c38803\") " pod="openshift-ovn-kubernetes/ovnkube-node-sk7ns"
May 26 16:00:40.870663 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:00:40.870639    2556 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 26 16:00:40.870663 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:00:40.870665    2556 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 26 16:00:40.870841 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:00:40.870679    2556 projected.go:194] Error preparing data for projected volume kube-api-access-xdxbg for pod openshift-network-diagnostics/network-check-target-twfp8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 26 16:00:40.870841 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:00:40.870741    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/115c1521-6997-403b-98d4-38a9198dd28e-kube-api-access-xdxbg podName:115c1521-6997-403b-98d4-38a9198dd28e nodeName:}" failed. No retries permitted until 2026-05-26 16:00:41.370727689 +0000 UTC m=+3.215344659 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-xdxbg" (UniqueName: "kubernetes.io/projected/115c1521-6997-403b-98d4-38a9198dd28e-kube-api-access-xdxbg") pod "network-check-target-twfp8" (UID: "115c1521-6997-403b-98d4-38a9198dd28e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 26 16:00:40.875351 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.875327    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hzr5\" (UniqueName: \"kubernetes.io/projected/4887c0d3-11fb-4287-bb94-a876a6c38803-kube-api-access-6hzr5\") pod \"ovnkube-node-sk7ns\" (UID: \"4887c0d3-11fb-4287-bb94-a876a6c38803\") " pod="openshift-ovn-kubernetes/ovnkube-node-sk7ns"
May 26 16:00:40.876186 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.876163    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pvlm\" (UniqueName: \"kubernetes.io/projected/b6c5e632-79b2-4174-b806-a97342c29687-kube-api-access-9pvlm\") pod \"iptables-alerter-s4dxt\" (UID: \"b6c5e632-79b2-4174-b806-a97342c29687\") " pod="openshift-network-operator/iptables-alerter-s4dxt"
May 26 16:00:40.921127 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.921086    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-gh2df"
May 26 16:00:40.928869 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.928847    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-lrkmg"
May 26 16:00:40.937511 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.937482    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-9znf9"
May 26 16:00:40.941986 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.941959    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-zc768"
May 26 16:00:40.947538 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.947515    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-b2npd"
May 26 16:00:40.955143 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.955123    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pmgqs"
May 26 16:00:40.961671 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.961647    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-s4dxt"
May 26 16:00:40.967321 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:40.967301    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-sk7ns"
May 26 16:00:41.242080 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:41.242047    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aed39145-780f-4642-af53-ddccbc03c34c-metrics-certs\") pod \"network-metrics-daemon-g9scb\" (UID: \"aed39145-780f-4642-af53-ddccbc03c34c\") " pod="openshift-multus/network-metrics-daemon-g9scb"
May 26 16:00:41.242276 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:00:41.242190    2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 26 16:00:41.242276 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:00:41.242255    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aed39145-780f-4642-af53-ddccbc03c34c-metrics-certs podName:aed39145-780f-4642-af53-ddccbc03c34c nodeName:}" failed. No retries permitted until 2026-05-26 16:00:42.242236678 +0000 UTC m=+4.086853653 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/aed39145-780f-4642-af53-ddccbc03c34c-metrics-certs") pod "network-metrics-daemon-g9scb" (UID: "aed39145-780f-4642-af53-ddccbc03c34c") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 26 16:00:41.307758 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:41.307726    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02507d1e_2665_4380_aa87_3605e4411561.slice/crio-55f1a005a5d525d6a975f07f9c65d801204f7451cc4b80b07affc1f69d557588 WatchSource:0}: Error finding container 55f1a005a5d525d6a975f07f9c65d801204f7451cc4b80b07affc1f69d557588: Status 404 returned error can't find the container with id 55f1a005a5d525d6a975f07f9c65d801204f7451cc4b80b07affc1f69d557588
May 26 16:00:41.308705 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:41.308685    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod391025fc_21eb_4786_8a0b_7f81c9eb90a4.slice/crio-cd28dd068c286db08340f151a498d0eed1e5c32d77e947e0661d28898f304496 WatchSource:0}: Error finding container cd28dd068c286db08340f151a498d0eed1e5c32d77e947e0661d28898f304496: Status 404 returned error can't find the container with id cd28dd068c286db08340f151a498d0eed1e5c32d77e947e0661d28898f304496
May 26 16:00:41.310254 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:41.310228    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42e4c880_fbee_4786_bbe5_7004126a87be.slice/crio-4bb26d712648e288e3def26ca4936465f5fbed090a65299fbc3b370e4218d4a7 WatchSource:0}: Error finding container 4bb26d712648e288e3def26ca4936465f5fbed090a65299fbc3b370e4218d4a7: Status 404 returned error can't find the container with id 4bb26d712648e288e3def26ca4936465f5fbed090a65299fbc3b370e4218d4a7
May 26 16:00:41.313290 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:41.313263    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc458685_6108_4373_aa17_f6e49f2c6e1b.slice/crio-fd3bf5697257525c2b48da21312e8e393f9bb70ea4ac1db2d491dac8adacb371 WatchSource:0}: Error finding container fd3bf5697257525c2b48da21312e8e393f9bb70ea4ac1db2d491dac8adacb371: Status 404 returned error can't find the container with id fd3bf5697257525c2b48da21312e8e393f9bb70ea4ac1db2d491dac8adacb371
May 26 16:00:41.314807 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:41.314733    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb75ad1e4_d3a4_4f24_bc06_3ef6661e1b3d.slice/crio-519531736bc908da330170259063323db8f0edc662a5a8c50665b08bebed7fde WatchSource:0}: Error finding container 519531736bc908da330170259063323db8f0edc662a5a8c50665b08bebed7fde: Status 404 returned error can't find the container with id 519531736bc908da330170259063323db8f0edc662a5a8c50665b08bebed7fde
May 26 16:00:41.315018 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:41.314996    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b5a0115_0b0c_4d86_8e58_732e7814f4c9.slice/crio-9c0c7f5e3fbfd1ede7b33378bf1d13c539d41616a76675f897476de4a778e8fd WatchSource:0}: Error finding container 9c0c7f5e3fbfd1ede7b33378bf1d13c539d41616a76675f897476de4a778e8fd: Status 404 returned error can't find the container with id 9c0c7f5e3fbfd1ede7b33378bf1d13c539d41616a76675f897476de4a778e8fd
May 26 16:00:41.315857 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:41.315834    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6c5e632_79b2_4174_b806_a97342c29687.slice/crio-7b260e816da8b1a2a4437013f4c8520857b1a8b69ef81c76e8f7fb406be0f123 WatchSource:0}: Error finding container 7b260e816da8b1a2a4437013f4c8520857b1a8b69ef81c76e8f7fb406be0f123: Status 404 returned error can't find the container with id 7b260e816da8b1a2a4437013f4c8520857b1a8b69ef81c76e8f7fb406be0f123
May 26 16:00:41.336465 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:41.336434    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4887c0d3_11fb_4287_bb94_a876a6c38803.slice/crio-6d3a181b32de3d38a7976ccfecdd13befccf96978e2426835ba7d78bd2744f2e WatchSource:0}: Error finding container 6d3a181b32de3d38a7976ccfecdd13befccf96978e2426835ba7d78bd2744f2e: Status 404 returned error can't find the container with id 6d3a181b32de3d38a7976ccfecdd13befccf96978e2426835ba7d78bd2744f2e
May 26 16:00:41.342402 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:41.342379    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c0cc81a8-fc3f-4188-8e90-4640d751dfb5-original-pull-secret\") pod \"global-pull-secret-syncer-wc8kh\" (UID: \"c0cc81a8-fc3f-4188-8e90-4640d751dfb5\") " pod="kube-system/global-pull-secret-syncer-wc8kh"
May 26 16:00:41.342559 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:00:41.342536    2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
May 26 16:00:41.342632 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:00:41.342618    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0cc81a8-fc3f-4188-8e90-4640d751dfb5-original-pull-secret podName:c0cc81a8-fc3f-4188-8e90-4640d751dfb5 nodeName:}" failed. No retries permitted until 2026-05-26 16:00:42.342594392 +0000 UTC m=+4.187211386 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c0cc81a8-fc3f-4188-8e90-4640d751dfb5-original-pull-secret") pod "global-pull-secret-syncer-wc8kh" (UID: "c0cc81a8-fc3f-4188-8e90-4640d751dfb5") : object "kube-system"/"original-pull-secret" not registered
May 26 16:00:41.443110 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:41.443068    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-xdxbg\" (UniqueName: \"kubernetes.io/projected/115c1521-6997-403b-98d4-38a9198dd28e-kube-api-access-xdxbg\") pod \"network-check-target-twfp8\" (UID: \"115c1521-6997-403b-98d4-38a9198dd28e\") " pod="openshift-network-diagnostics/network-check-target-twfp8"
May 26 16:00:41.443244 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:00:41.443219    2556 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 26 16:00:41.443244 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:00:41.443239    2556 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 26 16:00:41.443313 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:00:41.443248    2556 projected.go:194] Error preparing data for projected volume kube-api-access-xdxbg for pod openshift-network-diagnostics/network-check-target-twfp8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 26 16:00:41.443313 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:00:41.443300    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/115c1521-6997-403b-98d4-38a9198dd28e-kube-api-access-xdxbg podName:115c1521-6997-403b-98d4-38a9198dd28e nodeName:}" failed. No retries permitted until 2026-05-26 16:00:42.44328433 +0000 UTC m=+4.287901300 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-xdxbg" (UniqueName: "kubernetes.io/projected/115c1521-6997-403b-98d4-38a9198dd28e-kube-api-access-xdxbg") pod "network-check-target-twfp8" (UID: "115c1521-6997-403b-98d4-38a9198dd28e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 26 16:00:41.668978 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:41.668944    2556 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-05-25 15:55:39 +0000 UTC" deadline="2028-02-11 07:46:39.91071114 +0000 UTC"
May 26 16:00:41.668978 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:41.668973    2556 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15015h45m58.241739751s"
May 26 16:00:41.774386 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:41.774329    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-b2npd" event={"ID":"02507d1e-2665-4380-aa87-3605e4411561","Type":"ContainerStarted","Data":"55f1a005a5d525d6a975f07f9c65d801204f7451cc4b80b07affc1f69d557588"}
May 26 16:00:41.776575 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:41.776539    2556 generic.go:358] "Generic (PLEG): container finished" podID="f0079648f8b5dce19cfb7d44c2f62663" containerID="b839226d67604c556a1fcc4f26d6b623e385548c3429c65e884dbb6308a183df" exitCode=0
May 26 16:00:41.776832 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:41.776807    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-19.ec2.internal" event={"ID":"f0079648f8b5dce19cfb7d44c2f62663","Type":"ContainerDied","Data":"b839226d67604c556a1fcc4f26d6b623e385548c3429c65e884dbb6308a183df"}
May 26 16:00:41.778061 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:41.778036    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-lrkmg" event={"ID":"bc458685-6108-4373-aa17-f6e49f2c6e1b","Type":"ContainerStarted","Data":"fd3bf5697257525c2b48da21312e8e393f9bb70ea4ac1db2d491dac8adacb371"}
May 26 16:00:41.779995 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:41.779878    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-19.ec2.internal" event={"ID":"c33db3f858f26d571aec8a6b75b8b9e2","Type":"ContainerStarted","Data":"75d781a83815673a0da511ad2ff9b52297a8277315b3f13b4d7811972dcac4b7"}
May 26 16:00:41.781209 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:41.781186    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sk7ns" event={"ID":"4887c0d3-11fb-4287-bb94-a876a6c38803","Type":"ContainerStarted","Data":"6d3a181b32de3d38a7976ccfecdd13befccf96978e2426835ba7d78bd2744f2e"}
May 26 16:00:41.782705 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:41.782684    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-s4dxt" event={"ID":"b6c5e632-79b2-4174-b806-a97342c29687","Type":"ContainerStarted","Data":"7b260e816da8b1a2a4437013f4c8520857b1a8b69ef81c76e8f7fb406be0f123"}
May 26 16:00:41.784516 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:41.784490    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-zc768" event={"ID":"6b5a0115-0b0c-4d86-8e58-732e7814f4c9","Type":"ContainerStarted","Data":"9c0c7f5e3fbfd1ede7b33378bf1d13c539d41616a76675f897476de4a778e8fd"}
May 26 16:00:41.786011 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:41.785986    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pmgqs" event={"ID":"b75ad1e4-d3a4-4f24-bc06-3ef6661e1b3d","Type":"ContainerStarted","Data":"519531736bc908da330170259063323db8f0edc662a5a8c50665b08bebed7fde"}
May 26 16:00:41.787188 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:41.787165    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-gh2df" event={"ID":"42e4c880-fbee-4786-bbe5-7004126a87be","Type":"ContainerStarted","Data":"4bb26d712648e288e3def26ca4936465f5fbed090a65299fbc3b370e4218d4a7"}
May 26 16:00:41.788512 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:41.788488    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9znf9" event={"ID":"391025fc-21eb-4786-8a0b-7f81c9eb90a4","Type":"ContainerStarted","Data":"cd28dd068c286db08340f151a498d0eed1e5c32d77e947e0661d28898f304496"}
May 26 16:00:41.820128 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:41.819959    2556 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-19.ec2.internal" podStartSLOduration=2.819941521 podStartE2EDuration="2.819941521s" podCreationTimestamp="2026-05-26 16:00:39 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 16:00:41.819599893 +0000 UTC m=+3.664216886" watchObservedRunningTime="2026-05-26 16:00:41.819941521 +0000 UTC m=+3.664558517"
May 26 16:00:42.250372 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:42.250073    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aed39145-780f-4642-af53-ddccbc03c34c-metrics-certs\") pod \"network-metrics-daemon-g9scb\" (UID: \"aed39145-780f-4642-af53-ddccbc03c34c\") " pod="openshift-multus/network-metrics-daemon-g9scb"
May 26 16:00:42.250372 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:00:42.250285    2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 26 16:00:42.250603 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:00:42.250436    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aed39145-780f-4642-af53-ddccbc03c34c-metrics-certs podName:aed39145-780f-4642-af53-ddccbc03c34c nodeName:}" failed. No retries permitted until 2026-05-26 16:00:44.250415625 +0000 UTC m=+6.095032602 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/aed39145-780f-4642-af53-ddccbc03c34c-metrics-certs") pod "network-metrics-daemon-g9scb" (UID: "aed39145-780f-4642-af53-ddccbc03c34c") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 26 16:00:42.351555 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:42.351512    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c0cc81a8-fc3f-4188-8e90-4640d751dfb5-original-pull-secret\") pod \"global-pull-secret-syncer-wc8kh\" (UID: \"c0cc81a8-fc3f-4188-8e90-4640d751dfb5\") " pod="kube-system/global-pull-secret-syncer-wc8kh"
May 26 16:00:42.351745 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:00:42.351731    2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
May 26 16:00:42.352263 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:00:42.351871    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0cc81a8-fc3f-4188-8e90-4640d751dfb5-original-pull-secret podName:c0cc81a8-fc3f-4188-8e90-4640d751dfb5 nodeName:}" failed. No retries permitted until 2026-05-26 16:00:44.351828338 +0000 UTC m=+6.196445324 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c0cc81a8-fc3f-4188-8e90-4640d751dfb5-original-pull-secret") pod "global-pull-secret-syncer-wc8kh" (UID: "c0cc81a8-fc3f-4188-8e90-4640d751dfb5") : object "kube-system"/"original-pull-secret" not registered
May 26 16:00:42.452663 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:42.452623    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-xdxbg\" (UniqueName: \"kubernetes.io/projected/115c1521-6997-403b-98d4-38a9198dd28e-kube-api-access-xdxbg\") pod \"network-check-target-twfp8\" (UID: \"115c1521-6997-403b-98d4-38a9198dd28e\") " pod="openshift-network-diagnostics/network-check-target-twfp8"
May 26 16:00:42.452829 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:00:42.452815    2556 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 26 16:00:42.452887 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:00:42.452835    2556 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 26 16:00:42.452887 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:00:42.452848    2556 projected.go:194] Error preparing data for projected volume kube-api-access-xdxbg for pod openshift-network-diagnostics/network-check-target-twfp8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 26 16:00:42.453147 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:00:42.452909    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/115c1521-6997-403b-98d4-38a9198dd28e-kube-api-access-xdxbg podName:115c1521-6997-403b-98d4-38a9198dd28e nodeName:}" failed. No retries permitted until 2026-05-26 16:00:44.452890415 +0000 UTC m=+6.297507406 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-xdxbg" (UniqueName: "kubernetes.io/projected/115c1521-6997-403b-98d4-38a9198dd28e-kube-api-access-xdxbg") pod "network-check-target-twfp8" (UID: "115c1521-6997-403b-98d4-38a9198dd28e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 26 16:00:42.770388 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:42.769473    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-twfp8"
May 26 16:00:42.770388 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:00:42.769593    2556 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-twfp8" podUID="115c1521-6997-403b-98d4-38a9198dd28e"
May 26 16:00:42.770388 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:42.770025    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wc8kh"
May 26 16:00:42.770388 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:00:42.770135    2556 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wc8kh" podUID="c0cc81a8-fc3f-4188-8e90-4640d751dfb5"
May 26 16:00:42.770388 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:42.770219    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g9scb"
May 26 16:00:42.770388 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:00:42.770299    2556 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g9scb" podUID="aed39145-780f-4642-af53-ddccbc03c34c"
May 26 16:00:42.802194 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:42.802157    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-19.ec2.internal" event={"ID":"f0079648f8b5dce19cfb7d44c2f62663","Type":"ContainerStarted","Data":"75555f79975408abfddf827a8b0d0db05a5144938660c49f1b123c0a2c4d393d"}
May 26 16:00:42.824143 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:42.823972    2556 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-19.ec2.internal" podStartSLOduration=3.823952775 podStartE2EDuration="3.823952775s" podCreationTimestamp="2026-05-26 16:00:39 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 16:00:42.823170085 +0000 UTC m=+4.667787082" watchObservedRunningTime="2026-05-26 16:00:42.823952775 +0000 UTC m=+4.668569768"
May 26 16:00:44.271796 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:44.271339    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aed39145-780f-4642-af53-ddccbc03c34c-metrics-certs\") pod \"network-metrics-daemon-g9scb\" (UID: \"aed39145-780f-4642-af53-ddccbc03c34c\") " pod="openshift-multus/network-metrics-daemon-g9scb"
May 26 16:00:44.271796 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:00:44.271474    2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 26 16:00:44.271796 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:00:44.271551    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aed39145-780f-4642-af53-ddccbc03c34c-metrics-certs podName:aed39145-780f-4642-af53-ddccbc03c34c nodeName:}" failed. No retries permitted until 2026-05-26 16:00:48.271528246 +0000 UTC m=+10.116145222 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/aed39145-780f-4642-af53-ddccbc03c34c-metrics-certs") pod "network-metrics-daemon-g9scb" (UID: "aed39145-780f-4642-af53-ddccbc03c34c") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 26 16:00:44.372790 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:44.372746    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c0cc81a8-fc3f-4188-8e90-4640d751dfb5-original-pull-secret\") pod \"global-pull-secret-syncer-wc8kh\" (UID: \"c0cc81a8-fc3f-4188-8e90-4640d751dfb5\") " pod="kube-system/global-pull-secret-syncer-wc8kh"
May 26 16:00:44.372970 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:00:44.372922    2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
May 26 16:00:44.373024 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:00:44.372983    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0cc81a8-fc3f-4188-8e90-4640d751dfb5-original-pull-secret podName:c0cc81a8-fc3f-4188-8e90-4640d751dfb5 nodeName:}" failed. No retries permitted until 2026-05-26 16:00:48.37296332 +0000 UTC m=+10.217580304 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c0cc81a8-fc3f-4188-8e90-4640d751dfb5-original-pull-secret") pod "global-pull-secret-syncer-wc8kh" (UID: "c0cc81a8-fc3f-4188-8e90-4640d751dfb5") : object "kube-system"/"original-pull-secret" not registered
May 26 16:00:44.473536 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:44.473495    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-xdxbg\" (UniqueName: \"kubernetes.io/projected/115c1521-6997-403b-98d4-38a9198dd28e-kube-api-access-xdxbg\") pod \"network-check-target-twfp8\" (UID: \"115c1521-6997-403b-98d4-38a9198dd28e\") " pod="openshift-network-diagnostics/network-check-target-twfp8"
May 26 16:00:44.474146 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:00:44.473730    2556 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 26 16:00:44.474146 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:00:44.473752    2556 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 26 16:00:44.474146 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:00:44.473765    2556 projected.go:194] Error preparing data for projected volume kube-api-access-xdxbg for pod openshift-network-diagnostics/network-check-target-twfp8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 26 16:00:44.474146 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:00:44.473825    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/115c1521-6997-403b-98d4-38a9198dd28e-kube-api-access-xdxbg podName:115c1521-6997-403b-98d4-38a9198dd28e nodeName:}" failed. No retries permitted until 2026-05-26 16:00:48.473805582 +0000 UTC m=+10.318422555 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-xdxbg" (UniqueName: "kubernetes.io/projected/115c1521-6997-403b-98d4-38a9198dd28e-kube-api-access-xdxbg") pod "network-check-target-twfp8" (UID: "115c1521-6997-403b-98d4-38a9198dd28e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 26 16:00:44.770566 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:44.770054    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wc8kh"
May 26 16:00:44.770566 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:44.770205    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g9scb"
May 26 16:00:44.770566 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:00:44.770289    2556 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g9scb" podUID="aed39145-780f-4642-af53-ddccbc03c34c"
May 26 16:00:44.770566 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:00:44.770198    2556 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wc8kh" podUID="c0cc81a8-fc3f-4188-8e90-4640d751dfb5"
May 26 16:00:44.770566 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:44.770346    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-twfp8"
May 26 16:00:44.770566 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:00:44.770405    2556 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-twfp8" podUID="115c1521-6997-403b-98d4-38a9198dd28e"
May 26 16:00:46.767106 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:46.767050    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wc8kh"
May 26 16:00:46.767554 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:46.767064    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-twfp8"
May 26 16:00:46.767554 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:00:46.767210    2556 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wc8kh" podUID="c0cc81a8-fc3f-4188-8e90-4640d751dfb5"
May 26 16:00:46.767554 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:00:46.767320    2556 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-twfp8" podUID="115c1521-6997-403b-98d4-38a9198dd28e"
May 26 16:00:46.767554 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:46.767066    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g9scb"
May 26 16:00:46.767554 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:00:46.767401    2556 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g9scb" podUID="aed39145-780f-4642-af53-ddccbc03c34c"
May 26 16:00:48.308934 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:48.308891    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aed39145-780f-4642-af53-ddccbc03c34c-metrics-certs\") pod \"network-metrics-daemon-g9scb\" (UID: \"aed39145-780f-4642-af53-ddccbc03c34c\") " pod="openshift-multus/network-metrics-daemon-g9scb"
May 26 16:00:48.309438 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:00:48.309063    2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 26 16:00:48.309438 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:00:48.309165    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aed39145-780f-4642-af53-ddccbc03c34c-metrics-certs podName:aed39145-780f-4642-af53-ddccbc03c34c nodeName:}" failed. No retries permitted until 2026-05-26 16:00:56.309141926 +0000 UTC m=+18.153758910 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/aed39145-780f-4642-af53-ddccbc03c34c-metrics-certs") pod "network-metrics-daemon-g9scb" (UID: "aed39145-780f-4642-af53-ddccbc03c34c") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 26 16:00:48.409778 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:48.409737    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c0cc81a8-fc3f-4188-8e90-4640d751dfb5-original-pull-secret\") pod \"global-pull-secret-syncer-wc8kh\" (UID: \"c0cc81a8-fc3f-4188-8e90-4640d751dfb5\") " pod="kube-system/global-pull-secret-syncer-wc8kh"
May 26 16:00:48.409943 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:00:48.409895    2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
May 26 16:00:48.410012 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:00:48.409957    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0cc81a8-fc3f-4188-8e90-4640d751dfb5-original-pull-secret podName:c0cc81a8-fc3f-4188-8e90-4640d751dfb5 nodeName:}" failed. No retries permitted until 2026-05-26 16:00:56.409943167 +0000 UTC m=+18.254560145 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c0cc81a8-fc3f-4188-8e90-4640d751dfb5-original-pull-secret") pod "global-pull-secret-syncer-wc8kh" (UID: "c0cc81a8-fc3f-4188-8e90-4640d751dfb5") : object "kube-system"/"original-pull-secret" not registered
May 26 16:00:48.511222 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:48.511010    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-xdxbg\" (UniqueName: \"kubernetes.io/projected/115c1521-6997-403b-98d4-38a9198dd28e-kube-api-access-xdxbg\") pod \"network-check-target-twfp8\" (UID: \"115c1521-6997-403b-98d4-38a9198dd28e\") " pod="openshift-network-diagnostics/network-check-target-twfp8"
May 26 16:00:48.511222 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:00:48.511205    2556 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 26 16:00:48.511222 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:00:48.511224    2556 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 26 16:00:48.511547 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:00:48.511237    2556 projected.go:194] Error preparing data for projected volume kube-api-access-xdxbg for pod openshift-network-diagnostics/network-check-target-twfp8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 26 16:00:48.511547 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:00:48.511304    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/115c1521-6997-403b-98d4-38a9198dd28e-kube-api-access-xdxbg podName:115c1521-6997-403b-98d4-38a9198dd28e nodeName:}" failed. No retries permitted until 2026-05-26 16:00:56.511283928 +0000 UTC m=+18.355900909 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-xdxbg" (UniqueName: "kubernetes.io/projected/115c1521-6997-403b-98d4-38a9198dd28e-kube-api-access-xdxbg") pod "network-check-target-twfp8" (UID: "115c1521-6997-403b-98d4-38a9198dd28e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 26 16:00:48.767935 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:48.767852    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-twfp8"
May 26 16:00:48.768085 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:00:48.767962    2556 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-twfp8" podUID="115c1521-6997-403b-98d4-38a9198dd28e"
May 26 16:00:48.768085 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:48.768013    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g9scb"
May 26 16:00:48.768240 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:00:48.768166    2556 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g9scb" podUID="aed39145-780f-4642-af53-ddccbc03c34c"
May 26 16:00:48.768240 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:48.768199    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wc8kh"
May 26 16:00:48.768350 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:00:48.768267    2556 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wc8kh" podUID="c0cc81a8-fc3f-4188-8e90-4640d751dfb5"
May 26 16:00:50.770250 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:50.770210    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g9scb"
May 26 16:00:50.770250 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:50.770210    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wc8kh"
May 26 16:00:50.770250 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:50.770234    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-twfp8"
May 26 16:00:50.770822 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:00:50.770336    2556 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g9scb" podUID="aed39145-780f-4642-af53-ddccbc03c34c"
May 26 16:00:50.770822 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:00:50.770367    2556 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-twfp8" podUID="115c1521-6997-403b-98d4-38a9198dd28e"
May 26 16:00:50.770822 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:00:50.770430    2556 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wc8kh" podUID="c0cc81a8-fc3f-4188-8e90-4640d751dfb5"
May 26 16:00:52.766629 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:52.766593    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-twfp8"
May 26 16:00:52.767110 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:52.766720    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g9scb"
May 26 16:00:52.767110 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:00:52.766725    2556 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-twfp8" podUID="115c1521-6997-403b-98d4-38a9198dd28e"
May 26 16:00:52.767110 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:00:52.766842    2556 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g9scb" podUID="aed39145-780f-4642-af53-ddccbc03c34c"
May 26 16:00:52.767110 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:52.766729    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wc8kh"
May 26 16:00:52.767110 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:00:52.766963    2556 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wc8kh" podUID="c0cc81a8-fc3f-4188-8e90-4640d751dfb5"
May 26 16:00:53.351684 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:53.351627    2556 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-ghdqr"]
May 26 16:00:53.355886 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:53.355857    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-ghdqr"
May 26 16:00:53.358814 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:53.358701    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-tp96z\""
May 26 16:00:53.358814 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:53.358714    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
May 26 16:00:53.359825 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:53.359803    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
May 26 16:00:53.445947 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:53.445905    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f8c28cc-c746-4f30-a75d-4d7c1a6a5475-tmp-dir\") pod \"node-resolver-ghdqr\" (UID: \"5f8c28cc-c746-4f30-a75d-4d7c1a6a5475\") " pod="openshift-dns/node-resolver-ghdqr"
May 26 16:00:53.446154 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:53.445962    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5f8c28cc-c746-4f30-a75d-4d7c1a6a5475-hosts-file\") pod \"node-resolver-ghdqr\" (UID: \"5f8c28cc-c746-4f30-a75d-4d7c1a6a5475\") " pod="openshift-dns/node-resolver-ghdqr"
May 26 16:00:53.446154 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:53.446047    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4dxw\" (UniqueName: \"kubernetes.io/projected/5f8c28cc-c746-4f30-a75d-4d7c1a6a5475-kube-api-access-t4dxw\") pod \"node-resolver-ghdqr\" (UID: \"5f8c28cc-c746-4f30-a75d-4d7c1a6a5475\") " pod="openshift-dns/node-resolver-ghdqr"
May 26 16:00:53.546672 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:53.546635    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f8c28cc-c746-4f30-a75d-4d7c1a6a5475-tmp-dir\") pod \"node-resolver-ghdqr\" (UID: \"5f8c28cc-c746-4f30-a75d-4d7c1a6a5475\") " pod="openshift-dns/node-resolver-ghdqr"
May 26 16:00:53.546672 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:53.546680    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5f8c28cc-c746-4f30-a75d-4d7c1a6a5475-hosts-file\") pod \"node-resolver-ghdqr\" (UID: \"5f8c28cc-c746-4f30-a75d-4d7c1a6a5475\") " pod="openshift-dns/node-resolver-ghdqr"
May 26 16:00:53.546908 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:53.546708    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-t4dxw\" (UniqueName: \"kubernetes.io/projected/5f8c28cc-c746-4f30-a75d-4d7c1a6a5475-kube-api-access-t4dxw\") pod \"node-resolver-ghdqr\" (UID: \"5f8c28cc-c746-4f30-a75d-4d7c1a6a5475\") " pod="openshift-dns/node-resolver-ghdqr"
May 26 16:00:53.546908 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:53.546778    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5f8c28cc-c746-4f30-a75d-4d7c1a6a5475-hosts-file\") pod \"node-resolver-ghdqr\" (UID: \"5f8c28cc-c746-4f30-a75d-4d7c1a6a5475\") " pod="openshift-dns/node-resolver-ghdqr"
May 26 16:00:53.547057 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:53.547033    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f8c28cc-c746-4f30-a75d-4d7c1a6a5475-tmp-dir\") pod \"node-resolver-ghdqr\" (UID: \"5f8c28cc-c746-4f30-a75d-4d7c1a6a5475\") " pod="openshift-dns/node-resolver-ghdqr"
May 26 16:00:53.573552 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:53.573516    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4dxw\" (UniqueName: \"kubernetes.io/projected/5f8c28cc-c746-4f30-a75d-4d7c1a6a5475-kube-api-access-t4dxw\") pod \"node-resolver-ghdqr\" (UID: \"5f8c28cc-c746-4f30-a75d-4d7c1a6a5475\") " pod="openshift-dns/node-resolver-ghdqr"
May 26 16:00:53.666985 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:53.666872    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-ghdqr"
May 26 16:00:54.770642 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:54.770605    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wc8kh"
May 26 16:00:54.771138 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:54.770652    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g9scb"
May 26 16:00:54.771138 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:54.770655    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-twfp8"
May 26 16:00:54.771138 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:00:54.770747    2556 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wc8kh" podUID="c0cc81a8-fc3f-4188-8e90-4640d751dfb5"
May 26 16:00:54.771138 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:00:54.770887    2556 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g9scb" podUID="aed39145-780f-4642-af53-ddccbc03c34c"
May 26 16:00:54.771138 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:00:54.770971    2556 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-twfp8" podUID="115c1521-6997-403b-98d4-38a9198dd28e"
May 26 16:00:56.368543 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:56.368503    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aed39145-780f-4642-af53-ddccbc03c34c-metrics-certs\") pod \"network-metrics-daemon-g9scb\" (UID: \"aed39145-780f-4642-af53-ddccbc03c34c\") " pod="openshift-multus/network-metrics-daemon-g9scb"
May 26 16:00:56.369021 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:00:56.368672    2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 26 16:00:56.369021 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:00:56.368810    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aed39145-780f-4642-af53-ddccbc03c34c-metrics-certs podName:aed39145-780f-4642-af53-ddccbc03c34c nodeName:}" failed. No retries permitted until 2026-05-26 16:01:12.368786516 +0000 UTC m=+34.213403502 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/aed39145-780f-4642-af53-ddccbc03c34c-metrics-certs") pod "network-metrics-daemon-g9scb" (UID: "aed39145-780f-4642-af53-ddccbc03c34c") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 26 16:00:56.469800 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:56.469762    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c0cc81a8-fc3f-4188-8e90-4640d751dfb5-original-pull-secret\") pod \"global-pull-secret-syncer-wc8kh\" (UID: \"c0cc81a8-fc3f-4188-8e90-4640d751dfb5\") " pod="kube-system/global-pull-secret-syncer-wc8kh"
May 26 16:00:56.469977 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:00:56.469901    2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
May 26 16:00:56.469977 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:00:56.469963    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0cc81a8-fc3f-4188-8e90-4640d751dfb5-original-pull-secret podName:c0cc81a8-fc3f-4188-8e90-4640d751dfb5 nodeName:}" failed. No retries permitted until 2026-05-26 16:01:12.469945842 +0000 UTC m=+34.314562821 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c0cc81a8-fc3f-4188-8e90-4640d751dfb5-original-pull-secret") pod "global-pull-secret-syncer-wc8kh" (UID: "c0cc81a8-fc3f-4188-8e90-4640d751dfb5") : object "kube-system"/"original-pull-secret" not registered
May 26 16:00:56.570982 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:56.570946    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-xdxbg\" (UniqueName: \"kubernetes.io/projected/115c1521-6997-403b-98d4-38a9198dd28e-kube-api-access-xdxbg\") pod \"network-check-target-twfp8\" (UID: \"115c1521-6997-403b-98d4-38a9198dd28e\") " pod="openshift-network-diagnostics/network-check-target-twfp8"
May 26 16:00:56.571167 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:00:56.571121    2556 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 26 16:00:56.571167 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:00:56.571144    2556 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 26 16:00:56.571167 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:00:56.571156    2556 projected.go:194] Error preparing data for projected volume kube-api-access-xdxbg for pod openshift-network-diagnostics/network-check-target-twfp8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 26 16:00:56.571295 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:00:56.571219    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/115c1521-6997-403b-98d4-38a9198dd28e-kube-api-access-xdxbg podName:115c1521-6997-403b-98d4-38a9198dd28e nodeName:}" failed. No retries permitted until 2026-05-26 16:01:12.571201409 +0000 UTC m=+34.415818385 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-xdxbg" (UniqueName: "kubernetes.io/projected/115c1521-6997-403b-98d4-38a9198dd28e-kube-api-access-xdxbg") pod "network-check-target-twfp8" (UID: "115c1521-6997-403b-98d4-38a9198dd28e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 26 16:00:56.770154 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:56.770064    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g9scb"
May 26 16:00:56.770154 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:56.770114    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wc8kh"
May 26 16:00:56.770154 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:56.770154    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-twfp8"
May 26 16:00:56.770422 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:00:56.770238    2556 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-twfp8" podUID="115c1521-6997-403b-98d4-38a9198dd28e"
May 26 16:00:56.770422 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:00:56.770387    2556 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wc8kh" podUID="c0cc81a8-fc3f-4188-8e90-4640d751dfb5"
May 26 16:00:56.770527 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:00:56.770492    2556 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g9scb" podUID="aed39145-780f-4642-af53-ddccbc03c34c"
May 26 16:00:58.767271 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:58.767232    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g9scb"
May 26 16:00:58.767808 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:00:58.767334    2556 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g9scb" podUID="aed39145-780f-4642-af53-ddccbc03c34c"
May 26 16:00:58.767808 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:58.767350    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-twfp8"
May 26 16:00:58.767808 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:58.767368    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wc8kh"
May 26 16:00:58.767808 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:00:58.767465    2556 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-twfp8" podUID="115c1521-6997-403b-98d4-38a9198dd28e"
May 26 16:00:58.767808 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:00:58.767546    2556 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wc8kh" podUID="c0cc81a8-fc3f-4188-8e90-4640d751dfb5"
May 26 16:00:58.915468 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:00:58.915436    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f8c28cc_c746_4f30_a75d_4d7c1a6a5475.slice/crio-1ebcd067f03e2959e88e2d306b9a04bfa7a542f7af9ef4210f71ecd8f6e0a256 WatchSource:0}: Error finding container 1ebcd067f03e2959e88e2d306b9a04bfa7a542f7af9ef4210f71ecd8f6e0a256: Status 404 returned error can't find the container with id 1ebcd067f03e2959e88e2d306b9a04bfa7a542f7af9ef4210f71ecd8f6e0a256
May 26 16:00:59.833649 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:59.833270    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sk7ns_4887c0d3-11fb-4287-bb94-a876a6c38803/ovn-acl-logging/0.log"
May 26 16:00:59.834254 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:59.833975    2556 generic.go:358] "Generic (PLEG): container finished" podID="4887c0d3-11fb-4287-bb94-a876a6c38803" containerID="8a792fca5f72ee4341f7ead3ad7f9555fa2319fbc63492f5d9debe04217031c1" exitCode=1
May 26 16:00:59.834254 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:59.834009    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sk7ns" event={"ID":"4887c0d3-11fb-4287-bb94-a876a6c38803","Type":"ContainerStarted","Data":"b19b5124aa5d306a4b9241164e82e920937ccc30576ce4c3ac4f63f922891a2a"}
May 26 16:00:59.834254 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:59.834078    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sk7ns" event={"ID":"4887c0d3-11fb-4287-bb94-a876a6c38803","Type":"ContainerStarted","Data":"8fb4ca353b06b3e997404ca689d6d4becc61f0be6f3beedcbba1315be6aff1dc"}
May 26 16:00:59.834254 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:59.834135    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sk7ns" event={"ID":"4887c0d3-11fb-4287-bb94-a876a6c38803","Type":"ContainerStarted","Data":"7fd6a98ff61c3c015895a918f6d2c792369c118fe58bd012036d9adab7727198"}
May 26 16:00:59.834254 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:59.834150    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sk7ns" event={"ID":"4887c0d3-11fb-4287-bb94-a876a6c38803","Type":"ContainerStarted","Data":"c38c95434ae71104c158e8c584d06ed04c5c05df0bd57d1aed1b9950f5df25a8"}
May 26 16:00:59.834254 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:59.834162    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sk7ns" event={"ID":"4887c0d3-11fb-4287-bb94-a876a6c38803","Type":"ContainerDied","Data":"8a792fca5f72ee4341f7ead3ad7f9555fa2319fbc63492f5d9debe04217031c1"}
May 26 16:00:59.834254 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:59.834178    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sk7ns" event={"ID":"4887c0d3-11fb-4287-bb94-a876a6c38803","Type":"ContainerStarted","Data":"9698206afb8e464aecce60431bff2aa582e3d70dfeb8f931eb9aab61b4fa7c7d"}
May 26 16:00:59.835462 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:59.835432    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-zc768" event={"ID":"6b5a0115-0b0c-4d86-8e58-732e7814f4c9","Type":"ContainerStarted","Data":"6a9588afa86d51ea698f643f9b6b322fcc98ec85d6cf3b5623ac64aa17106363"}
May 26 16:00:59.836810 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:59.836784    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pmgqs" event={"ID":"b75ad1e4-d3a4-4f24-bc06-3ef6661e1b3d","Type":"ContainerStarted","Data":"c91dc3e4489e5fdd553fbaf2e5f5c9581249d5f683b17a26d73442c047f8fc72"}
May 26 16:00:59.838180 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:59.838157    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-gh2df" event={"ID":"42e4c880-fbee-4786-bbe5-7004126a87be","Type":"ContainerStarted","Data":"6cc47aa94424d2619912445d0d6f660d112f69521fc6374ccedcf1aa4cf00687"}
May 26 16:00:59.839653 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:59.839634    2556 generic.go:358] "Generic (PLEG): container finished" podID="391025fc-21eb-4786-8a0b-7f81c9eb90a4" containerID="0cdf02918dc91cce756ef10a3e5aa182cc8d3cf3120327c8785f100cc0e7257a" exitCode=0
May 26 16:00:59.839745 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:59.839698    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9znf9" event={"ID":"391025fc-21eb-4786-8a0b-7f81c9eb90a4","Type":"ContainerDied","Data":"0cdf02918dc91cce756ef10a3e5aa182cc8d3cf3120327c8785f100cc0e7257a"}
May 26 16:00:59.841320 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:59.841302    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-b2npd" event={"ID":"02507d1e-2665-4380-aa87-3605e4411561","Type":"ContainerStarted","Data":"72e76c1edd36b46cd2caf7fcf4019587d9c786489ebc74b5926d1d03435a1c91"}
May 26 16:00:59.842682 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:59.842658    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-lrkmg" event={"ID":"bc458685-6108-4373-aa17-f6e49f2c6e1b","Type":"ContainerStarted","Data":"86794b8760f35f6db4c346f5fba154080fee449e654969469b66116c622990a4"}
May 26 16:00:59.844380 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:59.844358    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-ghdqr" event={"ID":"5f8c28cc-c746-4f30-a75d-4d7c1a6a5475","Type":"ContainerStarted","Data":"6159cf5ffed630602d6651cfbbe832dab8c8435308187178bdc24d5939296a3e"}
May 26 16:00:59.844469 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:59.844388    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-ghdqr" event={"ID":"5f8c28cc-c746-4f30-a75d-4d7c1a6a5475","Type":"ContainerStarted","Data":"1ebcd067f03e2959e88e2d306b9a04bfa7a542f7af9ef4210f71ecd8f6e0a256"}
May 26 16:00:59.862019 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:59.861954    2556 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/multus-zc768" podStartSLOduration=4.038592618 podStartE2EDuration="21.861943137s" podCreationTimestamp="2026-05-26 16:00:38 +0000 UTC" firstStartedPulling="2026-05-26 16:00:41.334858023 +0000 UTC m=+3.179474996" lastFinishedPulling="2026-05-26 16:00:59.158208539 +0000 UTC m=+21.002825515" observedRunningTime="2026-05-26 16:00:59.861464315 +0000 UTC m=+21.706081308" watchObservedRunningTime="2026-05-26 16:00:59.861943137 +0000 UTC m=+21.706560364"
May 26 16:00:59.908404 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:59.908363    2556 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/konnectivity-agent-b2npd" podStartSLOduration=4.307688264 podStartE2EDuration="21.908352573s" podCreationTimestamp="2026-05-26 16:00:38 +0000 UTC" firstStartedPulling="2026-05-26 16:00:41.30939368 +0000 UTC m=+3.154010653" lastFinishedPulling="2026-05-26 16:00:58.910057991 +0000 UTC m=+20.754674962" observedRunningTime="2026-05-26 16:00:59.907493732 +0000 UTC m=+21.752110724" watchObservedRunningTime="2026-05-26 16:00:59.908352573 +0000 UTC m=+21.752969562"
May 26 16:00:59.968435 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:59.968398    2556 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-dns/node-resolver-ghdqr" podStartSLOduration=6.968385634 podStartE2EDuration="6.968385634s" podCreationTimestamp="2026-05-26 16:00:53 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 16:00:59.96829837 +0000 UTC m=+21.812915362" watchObservedRunningTime="2026-05-26 16:00:59.968385634 +0000 UTC m=+21.813002626"
May 26 16:00:59.968617 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:00:59.968591    2556 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-gh2df" podStartSLOduration=4.36874162 podStartE2EDuration="21.968583626s" podCreationTimestamp="2026-05-26 16:00:38 +0000 UTC" firstStartedPulling="2026-05-26 16:00:41.312481473 +0000 UTC m=+3.157098446" lastFinishedPulling="2026-05-26 16:00:58.912323467 +0000 UTC m=+20.756940452" observedRunningTime="2026-05-26 16:00:59.941071425 +0000 UTC m=+21.785688562" watchObservedRunningTime="2026-05-26 16:00:59.968583626 +0000 UTC m=+21.813200620"
May 26 16:01:00.000428 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:00.000393    2556 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-image-registry/node-ca-lrkmg" podStartSLOduration=4.44531386 podStartE2EDuration="22.000382111s" podCreationTimestamp="2026-05-26 16:00:38 +0000 UTC" firstStartedPulling="2026-05-26 16:00:41.315275549 +0000 UTC m=+3.159892534" lastFinishedPulling="2026-05-26 16:00:58.870343806 +0000 UTC m=+20.714960785" observedRunningTime="2026-05-26 16:01:00.000241342 +0000 UTC m=+21.844858359" watchObservedRunningTime="2026-05-26 16:01:00.000382111 +0000 UTC m=+21.844999103"
May 26 16:01:00.118153 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:00.118127    2556 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-b2npd"
May 26 16:01:00.118637 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:00.118622    2556 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-b2npd"
May 26 16:01:00.528323 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:00.528124    2556 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
May 26 16:01:00.700182 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:00.700044    2556 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-05-26T16:01:00.528138412Z","UUID":"28128673-d0d6-4efc-bf90-067bddd01e76","Handler":null,"Name":"","Endpoint":""}
May 26 16:01:00.702801 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:00.702778    2556 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
May 26 16:01:00.702801 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:00.702804    2556 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
May 26 16:01:00.766730 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:00.766707    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g9scb"
May 26 16:01:00.766844 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:00.766738    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-twfp8"
May 26 16:01:00.766844 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:01:00.766805    2556 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g9scb" podUID="aed39145-780f-4642-af53-ddccbc03c34c"
May 26 16:01:00.766844 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:00.766832    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wc8kh"
May 26 16:01:00.766947 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:01:00.766926    2556 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wc8kh" podUID="c0cc81a8-fc3f-4188-8e90-4640d751dfb5"
May 26 16:01:00.767024 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:01:00.767006    2556 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-twfp8" podUID="115c1521-6997-403b-98d4-38a9198dd28e"
May 26 16:01:00.847981 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:00.847950    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-s4dxt" event={"ID":"b6c5e632-79b2-4174-b806-a97342c29687","Type":"ContainerStarted","Data":"a060b3cf7563496e262ef156a88369c0c3f38517eaeed66ae38c83a2eaa57eb7"}
May 26 16:01:00.849842 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:00.849814    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pmgqs" event={"ID":"b75ad1e4-d3a4-4f24-bc06-3ef6661e1b3d","Type":"ContainerStarted","Data":"0e882e89ebeb8b18b8b4fe03ecfd45abb6f2d8b98e829b85be53e02509d61788"}
May 26 16:01:00.850453 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:00.850430    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-b2npd"
May 26 16:01:00.850900 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:00.850885    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-b2npd"
May 26 16:01:00.871008 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:00.870973    2556 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-s4dxt" podStartSLOduration=5.295188541 podStartE2EDuration="22.870962504s" podCreationTimestamp="2026-05-26 16:00:38 +0000 UTC" firstStartedPulling="2026-05-26 16:00:41.334712142 +0000 UTC m=+3.179329117" lastFinishedPulling="2026-05-26 16:00:58.910486108 +0000 UTC m=+20.755103080" observedRunningTime="2026-05-26 16:01:00.869963956 +0000 UTC m=+22.714580949" watchObservedRunningTime="2026-05-26 16:01:00.870962504 +0000 UTC m=+22.715579510"
May 26 16:01:02.767473 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:02.767437    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-twfp8"
May 26 16:01:02.768128 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:02.767437    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g9scb"
May 26 16:01:02.768128 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:01:02.767550    2556 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-twfp8" podUID="115c1521-6997-403b-98d4-38a9198dd28e"
May 26 16:01:02.768128 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:01:02.767624    2556 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g9scb" podUID="aed39145-780f-4642-af53-ddccbc03c34c"
May 26 16:01:02.768128 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:02.767449    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wc8kh"
May 26 16:01:02.768128 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:01:02.767760    2556 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wc8kh" podUID="c0cc81a8-fc3f-4188-8e90-4640d751dfb5"
May 26 16:01:02.856427 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:02.856401    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sk7ns_4887c0d3-11fb-4287-bb94-a876a6c38803/ovn-acl-logging/0.log"
May 26 16:01:02.856884 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:02.856857    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sk7ns" event={"ID":"4887c0d3-11fb-4287-bb94-a876a6c38803","Type":"ContainerStarted","Data":"013bf54a79398e1e5d9911860c53c4b1bb243c76c431c9c08e3df8230a1307b0"}
May 26 16:01:02.858857 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:02.858820    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pmgqs" event={"ID":"b75ad1e4-d3a4-4f24-bc06-3ef6661e1b3d","Type":"ContainerStarted","Data":"b89dd436abf21b2383bfe0c7339fe1af6a5f011391339a9ff64ca626c498c253"}
May 26 16:01:04.766723 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:04.766599    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-twfp8"
May 26 16:01:04.767047 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:04.766599    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g9scb"
May 26 16:01:04.767047 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:01:04.766824    2556 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-twfp8" podUID="115c1521-6997-403b-98d4-38a9198dd28e"
May 26 16:01:04.767047 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:04.766601    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wc8kh"
May 26 16:01:04.767047 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:01:04.766879    2556 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g9scb" podUID="aed39145-780f-4642-af53-ddccbc03c34c"
May 26 16:01:04.767047 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:01:04.766971    2556 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wc8kh" podUID="c0cc81a8-fc3f-4188-8e90-4640d751dfb5"
May 26 16:01:04.864915 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:04.864895    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sk7ns_4887c0d3-11fb-4287-bb94-a876a6c38803/ovn-acl-logging/0.log"
May 26 16:01:04.865264 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:04.865230    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sk7ns" event={"ID":"4887c0d3-11fb-4287-bb94-a876a6c38803","Type":"ContainerStarted","Data":"e9ea8dd0287763f02bda20e5057fbda89f4a305f67c2b726f37e6274328da150"}
May 26 16:01:04.865597 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:04.865576    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-sk7ns"
May 26 16:01:04.865672 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:04.865607    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-sk7ns"
May 26 16:01:04.865772 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:04.865756    2556 scope.go:117] "RemoveContainer" containerID="8a792fca5f72ee4341f7ead3ad7f9555fa2319fbc63492f5d9debe04217031c1"
May 26 16:01:04.883112 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:04.883055    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-sk7ns"
May 26 16:01:04.902630 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:04.902570    2556 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pmgqs" podStartSLOduration=6.394431478 podStartE2EDuration="26.902556065s" podCreationTimestamp="2026-05-26 16:00:38 +0000 UTC" firstStartedPulling="2026-05-26 16:00:41.334815046 +0000 UTC m=+3.179432016" lastFinishedPulling="2026-05-26 16:01:01.842939633 +0000 UTC m=+23.687556603" observedRunningTime="2026-05-26 16:01:02.882806568 +0000 UTC m=+24.727423561" watchObservedRunningTime="2026-05-26 16:01:04.902556065 +0000 UTC m=+26.747173057"
May 26 16:01:05.870008 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:05.869861    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sk7ns_4887c0d3-11fb-4287-bb94-a876a6c38803/ovn-acl-logging/0.log"
May 26 16:01:05.870442 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:05.870329    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sk7ns" event={"ID":"4887c0d3-11fb-4287-bb94-a876a6c38803","Type":"ContainerStarted","Data":"cc6b1ffc81722868f97dcfe0c3c1b24adb00ed32a124b43ee3c08939ef071153"}
May 26 16:01:05.870518 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:05.870505    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-sk7ns"
May 26 16:01:05.871921 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:05.871897    2556 generic.go:358] "Generic (PLEG): container finished" podID="391025fc-21eb-4786-8a0b-7f81c9eb90a4" containerID="4291c8ebde0cc2fa00e98ba1823add74bcb54d6a2159403c6c8b7468c5728071" exitCode=0
May 26 16:01:05.872059 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:05.871933    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9znf9" event={"ID":"391025fc-21eb-4786-8a0b-7f81c9eb90a4","Type":"ContainerDied","Data":"4291c8ebde0cc2fa00e98ba1823add74bcb54d6a2159403c6c8b7468c5728071"}
May 26 16:01:05.885710 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:05.885690    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-sk7ns"
May 26 16:01:05.974712 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:05.974585    2556 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-sk7ns" podStartSLOduration=10.357933151 podStartE2EDuration="27.974566508s" podCreationTimestamp="2026-05-26 16:00:38 +0000 UTC" firstStartedPulling="2026-05-26 16:00:41.33963768 +0000 UTC m=+3.184254651" lastFinishedPulling="2026-05-26 16:00:58.95627102 +0000 UTC m=+20.800888008" observedRunningTime="2026-05-26 16:01:05.925005941 +0000 UTC m=+27.769622959" watchObservedRunningTime="2026-05-26 16:01:05.974566508 +0000 UTC m=+27.819183513"
May 26 16:01:06.767108 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:06.767024    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-twfp8"
May 26 16:01:06.767231 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:06.767025    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g9scb"
May 26 16:01:06.767231 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:01:06.767139    2556 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-twfp8" podUID="115c1521-6997-403b-98d4-38a9198dd28e"
May 26 16:01:06.767231 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:06.767155    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wc8kh"
May 26 16:01:06.767390 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:01:06.767233    2556 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g9scb" podUID="aed39145-780f-4642-af53-ddccbc03c34c"
May 26 16:01:06.767390 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:01:06.767293    2556 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wc8kh" podUID="c0cc81a8-fc3f-4188-8e90-4640d751dfb5"
May 26 16:01:06.798048 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:06.798022    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-g9scb"]
May 26 16:01:06.799969 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:06.799942    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-wc8kh"]
May 26 16:01:06.800544 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:06.800513    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-twfp8"]
May 26 16:01:06.875232 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:06.875207    2556 generic.go:358] "Generic (PLEG): container finished" podID="391025fc-21eb-4786-8a0b-7f81c9eb90a4" containerID="cd02e837d94b8d151d7c47cc70cf6e8ae9fa96963f4db9f7ef208e6732c54ee1" exitCode=0
May 26 16:01:06.875878 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:06.875284    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-twfp8"
May 26 16:01:06.875878 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:06.875320    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9znf9" event={"ID":"391025fc-21eb-4786-8a0b-7f81c9eb90a4","Type":"ContainerDied","Data":"cd02e837d94b8d151d7c47cc70cf6e8ae9fa96963f4db9f7ef208e6732c54ee1"}
May 26 16:01:06.875878 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:06.875457    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g9scb"
May 26 16:01:06.875878 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:01:06.875551    2556 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g9scb" podUID="aed39145-780f-4642-af53-ddccbc03c34c"
May 26 16:01:06.875878 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:06.875642    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wc8kh"
May 26 16:01:06.875878 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:01:06.875755    2556 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wc8kh" podUID="c0cc81a8-fc3f-4188-8e90-4640d751dfb5"
May 26 16:01:06.876354 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:01:06.875903    2556 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-twfp8" podUID="115c1521-6997-403b-98d4-38a9198dd28e"
May 26 16:01:07.879238 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:07.879208    2556 generic.go:358] "Generic (PLEG): container finished" podID="391025fc-21eb-4786-8a0b-7f81c9eb90a4" containerID="30030c3004519f1c814ff88ffdc79eb04e9d6144d49f03a60bfde4e58f28f70e" exitCode=0
May 26 16:01:07.879671 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:07.879298    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9znf9" event={"ID":"391025fc-21eb-4786-8a0b-7f81c9eb90a4","Type":"ContainerDied","Data":"30030c3004519f1c814ff88ffdc79eb04e9d6144d49f03a60bfde4e58f28f70e"}
May 26 16:01:08.767591 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:08.767519    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wc8kh"
May 26 16:01:08.767739 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:08.767606    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g9scb"
May 26 16:01:08.767739 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:01:08.767644    2556 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wc8kh" podUID="c0cc81a8-fc3f-4188-8e90-4640d751dfb5"
May 26 16:01:08.767739 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:01:08.767676    2556 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g9scb" podUID="aed39145-780f-4642-af53-ddccbc03c34c"
May 26 16:01:08.767739 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:08.767701    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-twfp8"
May 26 16:01:08.767897 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:01:08.767759    2556 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-twfp8" podUID="115c1521-6997-403b-98d4-38a9198dd28e"
May 26 16:01:10.766897 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:10.766626    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g9scb"
May 26 16:01:10.766897 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:10.766625    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-twfp8"
May 26 16:01:10.766897 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:10.766640    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wc8kh"
May 26 16:01:10.767349 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:01:10.766967    2556 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g9scb" podUID="aed39145-780f-4642-af53-ddccbc03c34c"
May 26 16:01:10.767349 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:01:10.767066    2556 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-twfp8" podUID="115c1521-6997-403b-98d4-38a9198dd28e"
May 26 16:01:10.767349 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:01:10.767177    2556 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wc8kh" podUID="c0cc81a8-fc3f-4188-8e90-4640d751dfb5"
May 26 16:01:11.984850 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:11.984813    2556 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-134-19.ec2.internal" event="NodeReady"
May 26 16:01:11.985372 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:11.984963    2556 kubelet_node_status.go:550] "Fast updating node status as it just became ready"
May 26 16:01:12.195955 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:12.195922    2556 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-blmvt"]
May 26 16:01:12.201127 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:12.201085    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-blmvt"
May 26 16:01:12.207951 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:12.207859    2556 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-tkpb6"]
May 26 16:01:12.210678 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:12.210655    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
May 26 16:01:12.211363 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:12.211342    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-tkpb6"
May 26 16:01:12.239877 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:12.239780    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
May 26 16:01:12.240392 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:12.240235    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
May 26 16:01:12.240392 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:12.240272    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
May 26 16:01:12.240392 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:12.240291    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
May 26 16:01:12.240392 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:12.240343    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-6bgll\""
May 26 16:01:12.240656 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:12.240420    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-ghgvl\""
May 26 16:01:12.240874 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:12.240851    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-blmvt"]
May 26 16:01:12.288764 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:12.288736    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a131cc32-eb9c-4881-9af5-0985a6ad3b40-config-volume\") pod \"dns-default-blmvt\" (UID: \"a131cc32-eb9c-4881-9af5-0985a6ad3b40\") " pod="openshift-dns/dns-default-blmvt"
May 26 16:01:12.288902 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:12.288813    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwrhp\" (UniqueName: \"kubernetes.io/projected/a131cc32-eb9c-4881-9af5-0985a6ad3b40-kube-api-access-wwrhp\") pod \"dns-default-blmvt\" (UID: \"a131cc32-eb9c-4881-9af5-0985a6ad3b40\") " pod="openshift-dns/dns-default-blmvt"
May 26 16:01:12.288902 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:12.288844    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a131cc32-eb9c-4881-9af5-0985a6ad3b40-metrics-tls\") pod \"dns-default-blmvt\" (UID: \"a131cc32-eb9c-4881-9af5-0985a6ad3b40\") " pod="openshift-dns/dns-default-blmvt"
May 26 16:01:12.289008 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:12.288927    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a131cc32-eb9c-4881-9af5-0985a6ad3b40-tmp-dir\") pod \"dns-default-blmvt\" (UID: \"a131cc32-eb9c-4881-9af5-0985a6ad3b40\") " pod="openshift-dns/dns-default-blmvt"
May 26 16:01:12.301219 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:12.301197    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-tkpb6"]
May 26 16:01:12.389752 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:12.389725    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a131cc32-eb9c-4881-9af5-0985a6ad3b40-config-volume\") pod \"dns-default-blmvt\" (UID: \"a131cc32-eb9c-4881-9af5-0985a6ad3b40\") " pod="openshift-dns/dns-default-blmvt"
May 26 16:01:12.389902 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:12.389771    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aed39145-780f-4642-af53-ddccbc03c34c-metrics-certs\") pod \"network-metrics-daemon-g9scb\" (UID: \"aed39145-780f-4642-af53-ddccbc03c34c\") " pod="openshift-multus/network-metrics-daemon-g9scb"
May 26 16:01:12.389902 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:12.389807    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-wwrhp\" (UniqueName: \"kubernetes.io/projected/a131cc32-eb9c-4881-9af5-0985a6ad3b40-kube-api-access-wwrhp\") pod \"dns-default-blmvt\" (UID: \"a131cc32-eb9c-4881-9af5-0985a6ad3b40\") " pod="openshift-dns/dns-default-blmvt"
May 26 16:01:12.389902 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:12.389840    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a131cc32-eb9c-4881-9af5-0985a6ad3b40-metrics-tls\") pod \"dns-default-blmvt\" (UID: \"a131cc32-eb9c-4881-9af5-0985a6ad3b40\") " pod="openshift-dns/dns-default-blmvt"
May 26 16:01:12.389902 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:12.389887    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a131cc32-eb9c-4881-9af5-0985a6ad3b40-tmp-dir\") pod \"dns-default-blmvt\" (UID: \"a131cc32-eb9c-4881-9af5-0985a6ad3b40\") " pod="openshift-dns/dns-default-blmvt"
May 26 16:01:12.390140 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:01:12.389903    2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 26 16:01:12.390140 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:12.389920    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzwxk\" (UniqueName: \"kubernetes.io/projected/001b5d49-d123-4cec-85c4-e7f8927bf13a-kube-api-access-rzwxk\") pod \"ingress-canary-tkpb6\" (UID: \"001b5d49-d123-4cec-85c4-e7f8927bf13a\") " pod="openshift-ingress-canary/ingress-canary-tkpb6"
May 26 16:01:12.390140 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:12.389947    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/001b5d49-d123-4cec-85c4-e7f8927bf13a-cert\") pod \"ingress-canary-tkpb6\" (UID: \"001b5d49-d123-4cec-85c4-e7f8927bf13a\") " pod="openshift-ingress-canary/ingress-canary-tkpb6"
May 26 16:01:12.390140 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:01:12.389966    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aed39145-780f-4642-af53-ddccbc03c34c-metrics-certs podName:aed39145-780f-4642-af53-ddccbc03c34c nodeName:}" failed. No retries permitted until 2026-05-26 16:01:44.389950194 +0000 UTC m=+66.234567176 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/aed39145-780f-4642-af53-ddccbc03c34c-metrics-certs") pod "network-metrics-daemon-g9scb" (UID: "aed39145-780f-4642-af53-ddccbc03c34c") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 26 16:01:12.390140 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:01:12.390027    2556 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 26 16:01:12.390140 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:01:12.390078    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a131cc32-eb9c-4881-9af5-0985a6ad3b40-metrics-tls podName:a131cc32-eb9c-4881-9af5-0985a6ad3b40 nodeName:}" failed. No retries permitted until 2026-05-26 16:01:12.890066301 +0000 UTC m=+34.734683272 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a131cc32-eb9c-4881-9af5-0985a6ad3b40-metrics-tls") pod "dns-default-blmvt" (UID: "a131cc32-eb9c-4881-9af5-0985a6ad3b40") : secret "dns-default-metrics-tls" not found
May 26 16:01:12.390423 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:12.390247    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a131cc32-eb9c-4881-9af5-0985a6ad3b40-tmp-dir\") pod \"dns-default-blmvt\" (UID: \"a131cc32-eb9c-4881-9af5-0985a6ad3b40\") " pod="openshift-dns/dns-default-blmvt"
May 26 16:01:12.390423 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:12.390372    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a131cc32-eb9c-4881-9af5-0985a6ad3b40-config-volume\") pod \"dns-default-blmvt\" (UID: \"a131cc32-eb9c-4881-9af5-0985a6ad3b40\") " pod="openshift-dns/dns-default-blmvt"
May 26 16:01:12.409617 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:12.409567    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwrhp\" (UniqueName: \"kubernetes.io/projected/a131cc32-eb9c-4881-9af5-0985a6ad3b40-kube-api-access-wwrhp\") pod \"dns-default-blmvt\" (UID: \"a131cc32-eb9c-4881-9af5-0985a6ad3b40\") " pod="openshift-dns/dns-default-blmvt"
May 26 16:01:12.491270 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:12.491190    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-rzwxk\" (UniqueName: \"kubernetes.io/projected/001b5d49-d123-4cec-85c4-e7f8927bf13a-kube-api-access-rzwxk\") pod \"ingress-canary-tkpb6\" (UID: \"001b5d49-d123-4cec-85c4-e7f8927bf13a\") " pod="openshift-ingress-canary/ingress-canary-tkpb6"
May 26 16:01:12.491270 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:12.491240    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/001b5d49-d123-4cec-85c4-e7f8927bf13a-cert\") pod \"ingress-canary-tkpb6\" (UID: \"001b5d49-d123-4cec-85c4-e7f8927bf13a\") " pod="openshift-ingress-canary/ingress-canary-tkpb6"
May 26 16:01:12.491467 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:12.491295    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c0cc81a8-fc3f-4188-8e90-4640d751dfb5-original-pull-secret\") pod \"global-pull-secret-syncer-wc8kh\" (UID: \"c0cc81a8-fc3f-4188-8e90-4640d751dfb5\") " pod="kube-system/global-pull-secret-syncer-wc8kh"
May 26 16:01:12.491467 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:01:12.491403    2556 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 26 16:01:12.491467 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:01:12.491463    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/001b5d49-d123-4cec-85c4-e7f8927bf13a-cert podName:001b5d49-d123-4cec-85c4-e7f8927bf13a nodeName:}" failed. No retries permitted until 2026-05-26 16:01:12.991445353 +0000 UTC m=+34.836062334 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/001b5d49-d123-4cec-85c4-e7f8927bf13a-cert") pod "ingress-canary-tkpb6" (UID: "001b5d49-d123-4cec-85c4-e7f8927bf13a") : secret "canary-serving-cert" not found
May 26 16:01:12.491633 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:01:12.491403    2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
May 26 16:01:12.491633 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:01:12.491576    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0cc81a8-fc3f-4188-8e90-4640d751dfb5-original-pull-secret podName:c0cc81a8-fc3f-4188-8e90-4640d751dfb5 nodeName:}" failed. No retries permitted until 2026-05-26 16:01:44.491555191 +0000 UTC m=+66.336172174 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c0cc81a8-fc3f-4188-8e90-4640d751dfb5-original-pull-secret") pod "global-pull-secret-syncer-wc8kh" (UID: "c0cc81a8-fc3f-4188-8e90-4640d751dfb5") : object "kube-system"/"original-pull-secret" not registered
May 26 16:01:12.502064 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:12.502036    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzwxk\" (UniqueName: \"kubernetes.io/projected/001b5d49-d123-4cec-85c4-e7f8927bf13a-kube-api-access-rzwxk\") pod \"ingress-canary-tkpb6\" (UID: \"001b5d49-d123-4cec-85c4-e7f8927bf13a\") " pod="openshift-ingress-canary/ingress-canary-tkpb6"
May 26 16:01:12.592423 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:12.592393    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-xdxbg\" (UniqueName: \"kubernetes.io/projected/115c1521-6997-403b-98d4-38a9198dd28e-kube-api-access-xdxbg\") pod \"network-check-target-twfp8\" (UID: \"115c1521-6997-403b-98d4-38a9198dd28e\") " pod="openshift-network-diagnostics/network-check-target-twfp8"
May 26 16:01:12.592598 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:01:12.592549    2556 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 26 16:01:12.592598 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:01:12.592571    2556 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 26 16:01:12.592598 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:01:12.592582    2556 projected.go:194] Error preparing data for projected volume kube-api-access-xdxbg for pod openshift-network-diagnostics/network-check-target-twfp8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 26 16:01:12.592754 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:01:12.592640    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/115c1521-6997-403b-98d4-38a9198dd28e-kube-api-access-xdxbg podName:115c1521-6997-403b-98d4-38a9198dd28e nodeName:}" failed. No retries permitted until 2026-05-26 16:01:44.5926235 +0000 UTC m=+66.437240487 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-xdxbg" (UniqueName: "kubernetes.io/projected/115c1521-6997-403b-98d4-38a9198dd28e-kube-api-access-xdxbg") pod "network-check-target-twfp8" (UID: "115c1521-6997-403b-98d4-38a9198dd28e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 26 16:01:12.767538 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:12.767429    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wc8kh"
May 26 16:01:12.767538 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:12.767507    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g9scb"
May 26 16:01:12.767722 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:12.767626    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-twfp8"
May 26 16:01:12.770913 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:12.770891    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-f8kf8\""
May 26 16:01:12.771226 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:12.771203    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
May 26 16:01:12.771226 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:12.771225    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
May 26 16:01:12.771396 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:12.771207    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
May 26 16:01:12.771396 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:12.771328    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-8h4d8\""
May 26 16:01:12.771506 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:12.771494    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
May 26 16:01:12.894668 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:12.894639    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a131cc32-eb9c-4881-9af5-0985a6ad3b40-metrics-tls\") pod \"dns-default-blmvt\" (UID: \"a131cc32-eb9c-4881-9af5-0985a6ad3b40\") " pod="openshift-dns/dns-default-blmvt"
May 26 16:01:12.894841 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:01:12.894786    2556 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 26 16:01:12.894906 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:01:12.894845    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a131cc32-eb9c-4881-9af5-0985a6ad3b40-metrics-tls podName:a131cc32-eb9c-4881-9af5-0985a6ad3b40 nodeName:}" failed. No retries permitted until 2026-05-26 16:01:13.894828516 +0000 UTC m=+35.739445506 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a131cc32-eb9c-4881-9af5-0985a6ad3b40-metrics-tls") pod "dns-default-blmvt" (UID: "a131cc32-eb9c-4881-9af5-0985a6ad3b40") : secret "dns-default-metrics-tls" not found
May 26 16:01:12.995602 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:12.995560    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/001b5d49-d123-4cec-85c4-e7f8927bf13a-cert\") pod \"ingress-canary-tkpb6\" (UID: \"001b5d49-d123-4cec-85c4-e7f8927bf13a\") " pod="openshift-ingress-canary/ingress-canary-tkpb6"
May 26 16:01:12.996044 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:01:12.995727    2556 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 26 16:01:12.996044 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:01:12.995804    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/001b5d49-d123-4cec-85c4-e7f8927bf13a-cert podName:001b5d49-d123-4cec-85c4-e7f8927bf13a nodeName:}" failed. No retries permitted until 2026-05-26 16:01:13.995786276 +0000 UTC m=+35.840403248 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/001b5d49-d123-4cec-85c4-e7f8927bf13a-cert") pod "ingress-canary-tkpb6" (UID: "001b5d49-d123-4cec-85c4-e7f8927bf13a") : secret "canary-serving-cert" not found
May 26 16:01:13.624294 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:13.624261    2556 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-b985f5d89-62f95"]
May 26 16:01:13.640959 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:13.640932    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-b985f5d89-62f95"]
May 26 16:01:13.641068 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:13.641033    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-b985f5d89-62f95"
May 26 16:01:13.652941 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:13.652923    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-6zxcr\""
May 26 16:01:13.653065 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:13.653046    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
May 26 16:01:13.654199 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:13.654183    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
May 26 16:01:13.801161 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:13.801132    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krctg\" (UniqueName: \"kubernetes.io/projected/39981191-a506-435c-8237-74dffe29862a-kube-api-access-krctg\") pod \"migrator-b985f5d89-62f95\" (UID: \"39981191-a506-435c-8237-74dffe29862a\") " pod="openshift-kube-storage-version-migrator/migrator-b985f5d89-62f95"
May 26 16:01:13.901548 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:13.901522    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-krctg\" (UniqueName: \"kubernetes.io/projected/39981191-a506-435c-8237-74dffe29862a-kube-api-access-krctg\") pod \"migrator-b985f5d89-62f95\" (UID: \"39981191-a506-435c-8237-74dffe29862a\") " pod="openshift-kube-storage-version-migrator/migrator-b985f5d89-62f95"
May 26 16:01:13.901647 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:13.901572    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a131cc32-eb9c-4881-9af5-0985a6ad3b40-metrics-tls\") pod \"dns-default-blmvt\" (UID: \"a131cc32-eb9c-4881-9af5-0985a6ad3b40\") " pod="openshift-dns/dns-default-blmvt"
May 26 16:01:13.901685 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:01:13.901666    2556 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 26 16:01:13.901735 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:01:13.901723    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a131cc32-eb9c-4881-9af5-0985a6ad3b40-metrics-tls podName:a131cc32-eb9c-4881-9af5-0985a6ad3b40 nodeName:}" failed. No retries permitted until 2026-05-26 16:01:15.901706135 +0000 UTC m=+37.746323120 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a131cc32-eb9c-4881-9af5-0985a6ad3b40-metrics-tls") pod "dns-default-blmvt" (UID: "a131cc32-eb9c-4881-9af5-0985a6ad3b40") : secret "dns-default-metrics-tls" not found
May 26 16:01:13.911048 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:13.911032    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-krctg\" (UniqueName: \"kubernetes.io/projected/39981191-a506-435c-8237-74dffe29862a-kube-api-access-krctg\") pod \"migrator-b985f5d89-62f95\" (UID: \"39981191-a506-435c-8237-74dffe29862a\") " pod="openshift-kube-storage-version-migrator/migrator-b985f5d89-62f95"
May 26 16:01:13.949740 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:13.949707    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-b985f5d89-62f95"
May 26 16:01:14.002793 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:14.002758    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/001b5d49-d123-4cec-85c4-e7f8927bf13a-cert\") pod \"ingress-canary-tkpb6\" (UID: \"001b5d49-d123-4cec-85c4-e7f8927bf13a\") " pod="openshift-ingress-canary/ingress-canary-tkpb6"
May 26 16:01:14.003172 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:01:14.002864    2556 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 26 16:01:14.003172 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:01:14.002911    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/001b5d49-d123-4cec-85c4-e7f8927bf13a-cert podName:001b5d49-d123-4cec-85c4-e7f8927bf13a nodeName:}" failed. No retries permitted until 2026-05-26 16:01:16.002897482 +0000 UTC m=+37.847514456 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/001b5d49-d123-4cec-85c4-e7f8927bf13a-cert") pod "ingress-canary-tkpb6" (UID: "001b5d49-d123-4cec-85c4-e7f8927bf13a") : secret "canary-serving-cert" not found
May 26 16:01:14.127412 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:14.127387    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-b985f5d89-62f95"]
May 26 16:01:14.152040 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:01:14.151886    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39981191_a506_435c_8237_74dffe29862a.slice/crio-4378481755e02f6eaa1880e801b852d44e0b9778044ca578180f3ede449a1f3f WatchSource:0}: Error finding container 4378481755e02f6eaa1880e801b852d44e0b9778044ca578180f3ede449a1f3f: Status 404 returned error can't find the container with id 4378481755e02f6eaa1880e801b852d44e0b9778044ca578180f3ede449a1f3f
May 26 16:01:14.893454 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:14.893422    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-b985f5d89-62f95" event={"ID":"39981191-a506-435c-8237-74dffe29862a","Type":"ContainerStarted","Data":"4378481755e02f6eaa1880e801b852d44e0b9778044ca578180f3ede449a1f3f"}
May 26 16:01:14.896221 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:14.896191    2556 generic.go:358] "Generic (PLEG): container finished" podID="391025fc-21eb-4786-8a0b-7f81c9eb90a4" containerID="0afc1752cb0b7c68e2b2491e71846d33801aff6f817136a47ad235dbf0fdee20" exitCode=0
May 26 16:01:14.896335 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:14.896243    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9znf9" event={"ID":"391025fc-21eb-4786-8a0b-7f81c9eb90a4","Type":"ContainerDied","Data":"0afc1752cb0b7c68e2b2491e71846d33801aff6f817136a47ad235dbf0fdee20"}
May 26 16:01:15.011988 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:15.011969    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-ghdqr_5f8c28cc-c746-4f30-a75d-4d7c1a6a5475/dns-node-resolver/0.log"
May 26 16:01:15.389672 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:15.389641    2556 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-67d667c7d-x5dn4"]
May 26 16:01:15.397238 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:15.397214    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-67d667c7d-x5dn4"
May 26 16:01:15.400084 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:15.400063    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
May 26 16:01:15.400394 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:15.400378    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
May 26 16:01:15.400459 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:15.400428    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
May 26 16:01:15.400459 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:15.400448    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-vvv28\""
May 26 16:01:15.401411 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:15.401395    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
May 26 16:01:15.402816 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:15.402787    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-67d667c7d-x5dn4"]
May 26 16:01:15.516018 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:15.515988    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wlxq\" (UniqueName: \"kubernetes.io/projected/5aa4eb59-73ca-43fd-ad74-7c24509ac253-kube-api-access-9wlxq\") pod \"service-ca-67d667c7d-x5dn4\" (UID: \"5aa4eb59-73ca-43fd-ad74-7c24509ac253\") " pod="openshift-service-ca/service-ca-67d667c7d-x5dn4"
May 26 16:01:15.516147 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:15.516034    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/5aa4eb59-73ca-43fd-ad74-7c24509ac253-signing-key\") pod \"service-ca-67d667c7d-x5dn4\" (UID: \"5aa4eb59-73ca-43fd-ad74-7c24509ac253\") " pod="openshift-service-ca/service-ca-67d667c7d-x5dn4"
May 26 16:01:15.516210 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:15.516170    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/5aa4eb59-73ca-43fd-ad74-7c24509ac253-signing-cabundle\") pod \"service-ca-67d667c7d-x5dn4\" (UID: \"5aa4eb59-73ca-43fd-ad74-7c24509ac253\") " pod="openshift-service-ca/service-ca-67d667c7d-x5dn4"
May 26 16:01:15.617181 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:15.617148    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/5aa4eb59-73ca-43fd-ad74-7c24509ac253-signing-cabundle\") pod \"service-ca-67d667c7d-x5dn4\" (UID: \"5aa4eb59-73ca-43fd-ad74-7c24509ac253\") " pod="openshift-service-ca/service-ca-67d667c7d-x5dn4"
May 26 16:01:15.617279 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:15.617232    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-9wlxq\" (UniqueName: \"kubernetes.io/projected/5aa4eb59-73ca-43fd-ad74-7c24509ac253-kube-api-access-9wlxq\") pod \"service-ca-67d667c7d-x5dn4\" (UID: \"5aa4eb59-73ca-43fd-ad74-7c24509ac253\") " pod="openshift-service-ca/service-ca-67d667c7d-x5dn4"
May 26 16:01:15.617279 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:15.617257    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/5aa4eb59-73ca-43fd-ad74-7c24509ac253-signing-key\") pod \"service-ca-67d667c7d-x5dn4\" (UID: \"5aa4eb59-73ca-43fd-ad74-7c24509ac253\") " pod="openshift-service-ca/service-ca-67d667c7d-x5dn4"
May 26 16:01:15.620448 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:15.620425    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/5aa4eb59-73ca-43fd-ad74-7c24509ac253-signing-key\") pod \"service-ca-67d667c7d-x5dn4\" (UID: \"5aa4eb59-73ca-43fd-ad74-7c24509ac253\") " pod="openshift-service-ca/service-ca-67d667c7d-x5dn4"
May 26 16:01:15.622697 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:15.622674    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/5aa4eb59-73ca-43fd-ad74-7c24509ac253-signing-cabundle\") pod \"service-ca-67d667c7d-x5dn4\" (UID: \"5aa4eb59-73ca-43fd-ad74-7c24509ac253\") " pod="openshift-service-ca/service-ca-67d667c7d-x5dn4"
May 26 16:01:15.627689 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:15.627673    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wlxq\" (UniqueName: \"kubernetes.io/projected/5aa4eb59-73ca-43fd-ad74-7c24509ac253-kube-api-access-9wlxq\") pod \"service-ca-67d667c7d-x5dn4\" (UID: \"5aa4eb59-73ca-43fd-ad74-7c24509ac253\") " pod="openshift-service-ca/service-ca-67d667c7d-x5dn4"
May 26 16:01:15.706298 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:15.706277    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-67d667c7d-x5dn4"
May 26 16:01:15.823038 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:15.822838    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-67d667c7d-x5dn4"]
May 26 16:01:15.827060 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:01:15.827029    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5aa4eb59_73ca_43fd_ad74_7c24509ac253.slice/crio-579d0aefb91eac77e69ce1855107165c9ed6154825024049706b11e0a5c3e9ec WatchSource:0}: Error finding container 579d0aefb91eac77e69ce1855107165c9ed6154825024049706b11e0a5c3e9ec: Status 404 returned error can't find the container with id 579d0aefb91eac77e69ce1855107165c9ed6154825024049706b11e0a5c3e9ec
May 26 16:01:15.900014 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:15.899985    2556 generic.go:358] "Generic (PLEG): container finished" podID="391025fc-21eb-4786-8a0b-7f81c9eb90a4" containerID="728eeb546a04799572a3893dbe9b7f80555460cc70f98a174bfea6cc0c4351f9" exitCode=0
May 26 16:01:15.900117 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:15.900061    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9znf9" event={"ID":"391025fc-21eb-4786-8a0b-7f81c9eb90a4","Type":"ContainerDied","Data":"728eeb546a04799572a3893dbe9b7f80555460cc70f98a174bfea6cc0c4351f9"}
May 26 16:01:15.901571 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:15.901544    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-b985f5d89-62f95" event={"ID":"39981191-a506-435c-8237-74dffe29862a","Type":"ContainerStarted","Data":"6b213ab0ce5de74fd15af95a135b4785bab6196f387af286e4f7b8ba40a5845e"}
May 26 16:01:15.901571 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:15.901571    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-b985f5d89-62f95" event={"ID":"39981191-a506-435c-8237-74dffe29862a","Type":"ContainerStarted","Data":"fb0e5c7cecaa15896def7aa5e3051e41dda8ef4683b53f2fd78f2f58fcb115b0"}
May 26 16:01:15.902522 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:15.902502    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-67d667c7d-x5dn4" event={"ID":"5aa4eb59-73ca-43fd-ad74-7c24509ac253","Type":"ContainerStarted","Data":"579d0aefb91eac77e69ce1855107165c9ed6154825024049706b11e0a5c3e9ec"}
May 26 16:01:15.919569 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:15.919548    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a131cc32-eb9c-4881-9af5-0985a6ad3b40-metrics-tls\") pod \"dns-default-blmvt\" (UID: \"a131cc32-eb9c-4881-9af5-0985a6ad3b40\") " pod="openshift-dns/dns-default-blmvt"
May 26 16:01:15.919647 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:01:15.919636    2556 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 26 16:01:15.919689 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:01:15.919679    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a131cc32-eb9c-4881-9af5-0985a6ad3b40-metrics-tls podName:a131cc32-eb9c-4881-9af5-0985a6ad3b40 nodeName:}" failed. No retries permitted until 2026-05-26 16:01:19.919666766 +0000 UTC m=+41.764283735 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a131cc32-eb9c-4881-9af5-0985a6ad3b40-metrics-tls") pod "dns-default-blmvt" (UID: "a131cc32-eb9c-4881-9af5-0985a6ad3b40") : secret "dns-default-metrics-tls" not found
May 26 16:01:15.942963 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:15.942630    2556 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-b985f5d89-62f95" podStartSLOduration=1.630383189 podStartE2EDuration="2.942615725s" podCreationTimestamp="2026-05-26 16:01:13 +0000 UTC" firstStartedPulling="2026-05-26 16:01:14.15398234 +0000 UTC m=+35.998599310" lastFinishedPulling="2026-05-26 16:01:15.466214877 +0000 UTC m=+37.310831846" observedRunningTime="2026-05-26 16:01:15.94157928 +0000 UTC m=+37.786196276" watchObservedRunningTime="2026-05-26 16:01:15.942615725 +0000 UTC m=+37.787232718"
May 26 16:01:16.020552 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:16.020525    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/001b5d49-d123-4cec-85c4-e7f8927bf13a-cert\") pod \"ingress-canary-tkpb6\" (UID: \"001b5d49-d123-4cec-85c4-e7f8927bf13a\") " pod="openshift-ingress-canary/ingress-canary-tkpb6"
May 26 16:01:16.021152 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:01:16.020676    2556 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 26 16:01:16.021152 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:01:16.020732    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/001b5d49-d123-4cec-85c4-e7f8927bf13a-cert podName:001b5d49-d123-4cec-85c4-e7f8927bf13a nodeName:}" failed. No retries permitted until 2026-05-26 16:01:20.020715469 +0000 UTC m=+41.865332463 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/001b5d49-d123-4cec-85c4-e7f8927bf13a-cert") pod "ingress-canary-tkpb6" (UID: "001b5d49-d123-4cec-85c4-e7f8927bf13a") : secret "canary-serving-cert" not found
May 26 16:01:16.210720 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:16.210698    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-lrkmg_bc458685-6108-4373-aa17-f6e49f2c6e1b/node-ca/0.log"
May 26 16:01:16.907649 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:16.907617    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9znf9" event={"ID":"391025fc-21eb-4786-8a0b-7f81c9eb90a4","Type":"ContainerStarted","Data":"f73194e49fa9c7d7532700a056bc98784f9104b79bceed00be83e9478bc71d98"}
May 26 16:01:16.935596 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:16.935543    2556 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-9znf9" podStartSLOduration=6.416798795 podStartE2EDuration="38.935528968s" podCreationTimestamp="2026-05-26 16:00:38 +0000 UTC" firstStartedPulling="2026-05-26 16:00:41.310659568 +0000 UTC m=+3.155276540" lastFinishedPulling="2026-05-26 16:01:13.829389743 +0000 UTC m=+35.674006713" observedRunningTime="2026-05-26 16:01:16.93389625 +0000 UTC m=+38.778513243" watchObservedRunningTime="2026-05-26 16:01:16.935528968 +0000 UTC m=+38.780145960"
May 26 16:01:18.913364 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:18.913320    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-67d667c7d-x5dn4" event={"ID":"5aa4eb59-73ca-43fd-ad74-7c24509ac253","Type":"ContainerStarted","Data":"4ebbbc00f94ac69ad3c357b61d450032c301cd0f568fce87aa7fc16d121488ce"}
May 26 16:01:18.936947 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:18.936898    2556 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-service-ca/service-ca-67d667c7d-x5dn4" podStartSLOduration=1.491762571 podStartE2EDuration="3.936881125s" podCreationTimestamp="2026-05-26 16:01:15 +0000 UTC" firstStartedPulling="2026-05-26 16:01:15.828751562 +0000 UTC m=+37.673368532" lastFinishedPulling="2026-05-26 16:01:18.273870112 +0000 UTC m=+40.118487086" observedRunningTime="2026-05-26 16:01:18.936416401 +0000 UTC m=+40.781033394" watchObservedRunningTime="2026-05-26 16:01:18.936881125 +0000 UTC m=+40.781498111"
May 26 16:01:19.948748 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:19.948720    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a131cc32-eb9c-4881-9af5-0985a6ad3b40-metrics-tls\") pod \"dns-default-blmvt\" (UID: \"a131cc32-eb9c-4881-9af5-0985a6ad3b40\") " pod="openshift-dns/dns-default-blmvt"
May 26 16:01:19.949113 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:01:19.948826    2556 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 26 16:01:19.949113 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:01:19.948871    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a131cc32-eb9c-4881-9af5-0985a6ad3b40-metrics-tls podName:a131cc32-eb9c-4881-9af5-0985a6ad3b40 nodeName:}" failed. No retries permitted until 2026-05-26 16:01:27.948858516 +0000 UTC m=+49.793475486 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a131cc32-eb9c-4881-9af5-0985a6ad3b40-metrics-tls") pod "dns-default-blmvt" (UID: "a131cc32-eb9c-4881-9af5-0985a6ad3b40") : secret "dns-default-metrics-tls" not found
May 26 16:01:20.049966 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:20.049929    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/001b5d49-d123-4cec-85c4-e7f8927bf13a-cert\") pod \"ingress-canary-tkpb6\" (UID: \"001b5d49-d123-4cec-85c4-e7f8927bf13a\") " pod="openshift-ingress-canary/ingress-canary-tkpb6"
May 26 16:01:20.050171 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:01:20.050110    2556 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 26 16:01:20.050252 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:01:20.050179    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/001b5d49-d123-4cec-85c4-e7f8927bf13a-cert podName:001b5d49-d123-4cec-85c4-e7f8927bf13a nodeName:}" failed. No retries permitted until 2026-05-26 16:01:28.050159275 +0000 UTC m=+49.894776245 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/001b5d49-d123-4cec-85c4-e7f8927bf13a-cert") pod "ingress-canary-tkpb6" (UID: "001b5d49-d123-4cec-85c4-e7f8927bf13a") : secret "canary-serving-cert" not found
May 26 16:01:28.007923 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:28.007887    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a131cc32-eb9c-4881-9af5-0985a6ad3b40-metrics-tls\") pod \"dns-default-blmvt\" (UID: \"a131cc32-eb9c-4881-9af5-0985a6ad3b40\") " pod="openshift-dns/dns-default-blmvt"
May 26 16:01:28.010945 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:28.010919    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a131cc32-eb9c-4881-9af5-0985a6ad3b40-metrics-tls\") pod \"dns-default-blmvt\" (UID: \"a131cc32-eb9c-4881-9af5-0985a6ad3b40\") " pod="openshift-dns/dns-default-blmvt"
May 26 16:01:28.109179 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:28.109137    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/001b5d49-d123-4cec-85c4-e7f8927bf13a-cert\") pod \"ingress-canary-tkpb6\" (UID: \"001b5d49-d123-4cec-85c4-e7f8927bf13a\") " pod="openshift-ingress-canary/ingress-canary-tkpb6"
May 26 16:01:28.111101 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:28.111064    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/001b5d49-d123-4cec-85c4-e7f8927bf13a-cert\") pod \"ingress-canary-tkpb6\" (UID: \"001b5d49-d123-4cec-85c4-e7f8927bf13a\") " pod="openshift-ingress-canary/ingress-canary-tkpb6"
May 26 16:01:28.111959 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:28.111937    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-blmvt"
May 26 16:01:28.120654 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:28.120630    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-tkpb6"
May 26 16:01:28.241724 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:28.241696    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-blmvt"]
May 26 16:01:28.244810 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:01:28.244779    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda131cc32_eb9c_4881_9af5_0985a6ad3b40.slice/crio-f6f044c84309ca79c123a3a23c137b4aa9c5b05dced8de93e6b38e777a73262d WatchSource:0}: Error finding container f6f044c84309ca79c123a3a23c137b4aa9c5b05dced8de93e6b38e777a73262d: Status 404 returned error can't find the container with id f6f044c84309ca79c123a3a23c137b4aa9c5b05dced8de93e6b38e777a73262d
May 26 16:01:28.256560 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:28.256535    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-tkpb6"]
May 26 16:01:28.259248 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:01:28.259195    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod001b5d49_d123_4cec_85c4_e7f8927bf13a.slice/crio-1caaf3e8d598230ddcde69a56cde53d260384a1bf39db169f72bfd73d2ffaf57 WatchSource:0}: Error finding container 1caaf3e8d598230ddcde69a56cde53d260384a1bf39db169f72bfd73d2ffaf57: Status 404 returned error can't find the container with id 1caaf3e8d598230ddcde69a56cde53d260384a1bf39db169f72bfd73d2ffaf57
May 26 16:01:28.933020 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:28.932986    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-tkpb6" event={"ID":"001b5d49-d123-4cec-85c4-e7f8927bf13a","Type":"ContainerStarted","Data":"1caaf3e8d598230ddcde69a56cde53d260384a1bf39db169f72bfd73d2ffaf57"}
May 26 16:01:28.934553 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:28.934507    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-blmvt" event={"ID":"a131cc32-eb9c-4881-9af5-0985a6ad3b40","Type":"ContainerStarted","Data":"f6f044c84309ca79c123a3a23c137b4aa9c5b05dced8de93e6b38e777a73262d"}
May 26 16:01:30.940962 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:30.940934    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-tkpb6" event={"ID":"001b5d49-d123-4cec-85c4-e7f8927bf13a","Type":"ContainerStarted","Data":"947472eba48ca458b7172b71c16456b455929cbb809d8165c045f14a200fd8d1"}
May 26 16:01:30.942238 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:30.942212    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-blmvt" event={"ID":"a131cc32-eb9c-4881-9af5-0985a6ad3b40","Type":"ContainerStarted","Data":"d37c59fe3b33fc63385343708899d0cadf746afcacc579688150ce897a410f73"}
May 26 16:01:30.961827 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:30.961782    2556 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-tkpb6" podStartSLOduration=16.528604468 podStartE2EDuration="18.961767717s" podCreationTimestamp="2026-05-26 16:01:12 +0000 UTC" firstStartedPulling="2026-05-26 16:01:28.260977335 +0000 UTC m=+50.105594305" lastFinishedPulling="2026-05-26 16:01:30.694140584 +0000 UTC m=+52.538757554" observedRunningTime="2026-05-26 16:01:30.961324424 +0000 UTC m=+52.805941416" watchObservedRunningTime="2026-05-26 16:01:30.961767717 +0000 UTC m=+52.806384708"
May 26 16:01:31.946433 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:31.946398    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-blmvt" event={"ID":"a131cc32-eb9c-4881-9af5-0985a6ad3b40","Type":"ContainerStarted","Data":"654763d4f6e7b30b4d47a809eedb507c9024221ee177ca067f755bd30b509022"}
May 26 16:01:31.970696 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:31.970658    2556 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-dns/dns-default-blmvt" podStartSLOduration=17.527356095000002 podStartE2EDuration="19.97064544s" podCreationTimestamp="2026-05-26 16:01:12 +0000 UTC" firstStartedPulling="2026-05-26 16:01:28.24650612 +0000 UTC m=+50.091123091" lastFinishedPulling="2026-05-26 16:01:30.689795451 +0000 UTC m=+52.534412436" observedRunningTime="2026-05-26 16:01:31.970386063 +0000 UTC m=+53.815003052" watchObservedRunningTime="2026-05-26 16:01:31.97064544 +0000 UTC m=+53.815262422"
May 26 16:01:32.951257 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:32.950289    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-blmvt"
May 26 16:01:36.497775 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:36.497745    2556 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-cq5gn"]
May 26 16:01:36.523122 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:36.523081    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-cq5gn"]
May 26 16:01:36.523245 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:36.523228    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-cq5gn"
May 26 16:01:36.526256 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:36.526235    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
May 26 16:01:36.526374 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:36.526282    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
May 26 16:01:36.526492 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:36.526479    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-ltrjt\""
May 26 16:01:36.527561 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:36.527540    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
May 26 16:01:36.527661 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:36.527575    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
May 26 16:01:36.569315 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:36.569291    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e3f3da40-e5da-4eab-a2bb-b7f75f9546c8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cq5gn\" (UID: \"e3f3da40-e5da-4eab-a2bb-b7f75f9546c8\") " pod="openshift-insights/insights-runtime-extractor-cq5gn"
May 26 16:01:36.569423 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:36.569321    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e3f3da40-e5da-4eab-a2bb-b7f75f9546c8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cq5gn\" (UID: \"e3f3da40-e5da-4eab-a2bb-b7f75f9546c8\") " pod="openshift-insights/insights-runtime-extractor-cq5gn"
May 26 16:01:36.569423 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:36.569340    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fscvw\" (UniqueName: \"kubernetes.io/projected/e3f3da40-e5da-4eab-a2bb-b7f75f9546c8-kube-api-access-fscvw\") pod \"insights-runtime-extractor-cq5gn\" (UID: \"e3f3da40-e5da-4eab-a2bb-b7f75f9546c8\") " pod="openshift-insights/insights-runtime-extractor-cq5gn"
May 26 16:01:36.569423 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:36.569404    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e3f3da40-e5da-4eab-a2bb-b7f75f9546c8-data-volume\") pod \"insights-runtime-extractor-cq5gn\" (UID: \"e3f3da40-e5da-4eab-a2bb-b7f75f9546c8\") " pod="openshift-insights/insights-runtime-extractor-cq5gn"
May 26 16:01:36.569576 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:36.569451    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e3f3da40-e5da-4eab-a2bb-b7f75f9546c8-crio-socket\") pod \"insights-runtime-extractor-cq5gn\" (UID: \"e3f3da40-e5da-4eab-a2bb-b7f75f9546c8\") " pod="openshift-insights/insights-runtime-extractor-cq5gn"
May 26 16:01:36.598406 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:36.598383    2556 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-67474545cf-rgctv"]
May 26 16:01:36.619135 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:36.619109    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-67474545cf-rgctv"]
May 26 16:01:36.619234 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:36.619191    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-67474545cf-rgctv"
May 26 16:01:36.622071 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:36.622052    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
May 26 16:01:36.622667 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:36.622652    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
May 26 16:01:36.622788 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:36.622767    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-fjfhr\""
May 26 16:01:36.623894 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:36.623878    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
May 26 16:01:36.628578 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:36.628560    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
May 26 16:01:36.670604 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:36.670583    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4a9f9941-c5fa-4e05-97f4-23aeefb46d13-image-registry-private-configuration\") pod \"image-registry-67474545cf-rgctv\" (UID: \"4a9f9941-c5fa-4e05-97f4-23aeefb46d13\") " pod="openshift-image-registry/image-registry-67474545cf-rgctv"
May 26 16:01:36.670712 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:36.670635    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e3f3da40-e5da-4eab-a2bb-b7f75f9546c8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cq5gn\" (UID: \"e3f3da40-e5da-4eab-a2bb-b7f75f9546c8\") " pod="openshift-insights/insights-runtime-extractor-cq5gn"
May 26 16:01:36.670712 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:36.670653    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4a9f9941-c5fa-4e05-97f4-23aeefb46d13-registry-certificates\") pod \"image-registry-67474545cf-rgctv\" (UID: \"4a9f9941-c5fa-4e05-97f4-23aeefb46d13\") " pod="openshift-image-registry/image-registry-67474545cf-rgctv"
May 26 16:01:36.670712 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:36.670699    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcc5b\" (UniqueName: \"kubernetes.io/projected/4a9f9941-c5fa-4e05-97f4-23aeefb46d13-kube-api-access-dcc5b\") pod \"image-registry-67474545cf-rgctv\" (UID: \"4a9f9941-c5fa-4e05-97f4-23aeefb46d13\") " pod="openshift-image-registry/image-registry-67474545cf-rgctv"
May 26 16:01:36.670865 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:36.670754    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4a9f9941-c5fa-4e05-97f4-23aeefb46d13-registry-tls\") pod \"image-registry-67474545cf-rgctv\" (UID: \"4a9f9941-c5fa-4e05-97f4-23aeefb46d13\") " pod="openshift-image-registry/image-registry-67474545cf-rgctv"
May 26 16:01:36.670865 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:36.670773    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e3f3da40-e5da-4eab-a2bb-b7f75f9546c8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cq5gn\" (UID: \"e3f3da40-e5da-4eab-a2bb-b7f75f9546c8\") " pod="openshift-insights/insights-runtime-extractor-cq5gn"
May 26 16:01:36.670865 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:36.670792    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-fscvw\" (UniqueName: \"kubernetes.io/projected/e3f3da40-e5da-4eab-a2bb-b7f75f9546c8-kube-api-access-fscvw\") pod \"insights-runtime-extractor-cq5gn\" (UID: \"e3f3da40-e5da-4eab-a2bb-b7f75f9546c8\") " pod="openshift-insights/insights-runtime-extractor-cq5gn"
May 26 16:01:36.670865 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:36.670842    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e3f3da40-e5da-4eab-a2bb-b7f75f9546c8-data-volume\") pod \"insights-runtime-extractor-cq5gn\" (UID: \"e3f3da40-e5da-4eab-a2bb-b7f75f9546c8\") " pod="openshift-insights/insights-runtime-extractor-cq5gn"
May 26 16:01:36.671077 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:36.670887    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e3f3da40-e5da-4eab-a2bb-b7f75f9546c8-crio-socket\") pod \"insights-runtime-extractor-cq5gn\" (UID: \"e3f3da40-e5da-4eab-a2bb-b7f75f9546c8\") " pod="openshift-insights/insights-runtime-extractor-cq5gn"
May 26 16:01:36.671077 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:36.670919    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4a9f9941-c5fa-4e05-97f4-23aeefb46d13-installation-pull-secrets\") pod \"image-registry-67474545cf-rgctv\" (UID: \"4a9f9941-c5fa-4e05-97f4-23aeefb46d13\") " pod="openshift-image-registry/image-registry-67474545cf-rgctv"
May 26 16:01:36.671077 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:36.670945    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4a9f9941-c5fa-4e05-97f4-23aeefb46d13-ca-trust-extracted\") pod \"image-registry-67474545cf-rgctv\" (UID: \"4a9f9941-c5fa-4e05-97f4-23aeefb46d13\") " pod="openshift-image-registry/image-registry-67474545cf-rgctv"
May 26 16:01:36.671077 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:36.670969    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4a9f9941-c5fa-4e05-97f4-23aeefb46d13-trusted-ca\") pod \"image-registry-67474545cf-rgctv\" (UID: \"4a9f9941-c5fa-4e05-97f4-23aeefb46d13\") " pod="openshift-image-registry/image-registry-67474545cf-rgctv"
May 26 16:01:36.671077 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:36.670992    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4a9f9941-c5fa-4e05-97f4-23aeefb46d13-bound-sa-token\") pod \"image-registry-67474545cf-rgctv\" (UID: \"4a9f9941-c5fa-4e05-97f4-23aeefb46d13\") " pod="openshift-image-registry/image-registry-67474545cf-rgctv"
May 26 16:01:36.671077 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:36.671059    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e3f3da40-e5da-4eab-a2bb-b7f75f9546c8-crio-socket\") pod \"insights-runtime-extractor-cq5gn\" (UID: \"e3f3da40-e5da-4eab-a2bb-b7f75f9546c8\") " pod="openshift-insights/insights-runtime-extractor-cq5gn"
May 26 16:01:36.671380 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:36.671246    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e3f3da40-e5da-4eab-a2bb-b7f75f9546c8-data-volume\") pod \"insights-runtime-extractor-cq5gn\" (UID: \"e3f3da40-e5da-4eab-a2bb-b7f75f9546c8\") " pod="openshift-insights/insights-runtime-extractor-cq5gn"
May 26 16:01:36.671380 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:36.671302    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e3f3da40-e5da-4eab-a2bb-b7f75f9546c8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cq5gn\" (UID: \"e3f3da40-e5da-4eab-a2bb-b7f75f9546c8\") " pod="openshift-insights/insights-runtime-extractor-cq5gn"
May 26 16:01:36.673131 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:36.673115    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e3f3da40-e5da-4eab-a2bb-b7f75f9546c8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cq5gn\" (UID: \"e3f3da40-e5da-4eab-a2bb-b7f75f9546c8\") " pod="openshift-insights/insights-runtime-extractor-cq5gn"
May 26 16:01:36.684080 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:36.684057    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fscvw\" (UniqueName: \"kubernetes.io/projected/e3f3da40-e5da-4eab-a2bb-b7f75f9546c8-kube-api-access-fscvw\") pod \"insights-runtime-extractor-cq5gn\" (UID: \"e3f3da40-e5da-4eab-a2bb-b7f75f9546c8\") " pod="openshift-insights/insights-runtime-extractor-cq5gn"
May 26 16:01:36.772123 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:36.772060    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4a9f9941-c5fa-4e05-97f4-23aeefb46d13-registry-certificates\") pod \"image-registry-67474545cf-rgctv\" (UID: \"4a9f9941-c5fa-4e05-97f4-23aeefb46d13\") " pod="openshift-image-registry/image-registry-67474545cf-rgctv"
May 26 16:01:36.772123 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:36.772085    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-dcc5b\" (UniqueName: \"kubernetes.io/projected/4a9f9941-c5fa-4e05-97f4-23aeefb46d13-kube-api-access-dcc5b\") pod \"image-registry-67474545cf-rgctv\" (UID: \"4a9f9941-c5fa-4e05-97f4-23aeefb46d13\") " pod="openshift-image-registry/image-registry-67474545cf-rgctv"
May 26 16:01:36.772286 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:36.772123    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4a9f9941-c5fa-4e05-97f4-23aeefb46d13-registry-tls\") pod \"image-registry-67474545cf-rgctv\" (UID: \"4a9f9941-c5fa-4e05-97f4-23aeefb46d13\") " pod="openshift-image-registry/image-registry-67474545cf-rgctv"
May 26 16:01:36.772286 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:36.772238    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4a9f9941-c5fa-4e05-97f4-23aeefb46d13-installation-pull-secrets\") pod \"image-registry-67474545cf-rgctv\" (UID: \"4a9f9941-c5fa-4e05-97f4-23aeefb46d13\") " pod="openshift-image-registry/image-registry-67474545cf-rgctv"
May 26 16:01:36.772286 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:36.772274    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4a9f9941-c5fa-4e05-97f4-23aeefb46d13-ca-trust-extracted\") pod \"image-registry-67474545cf-rgctv\" (UID: \"4a9f9941-c5fa-4e05-97f4-23aeefb46d13\") " pod="openshift-image-registry/image-registry-67474545cf-rgctv"
May 26 16:01:36.772448 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:36.772299    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4a9f9941-c5fa-4e05-97f4-23aeefb46d13-trusted-ca\") pod \"image-registry-67474545cf-rgctv\" (UID: \"4a9f9941-c5fa-4e05-97f4-23aeefb46d13\") " pod="openshift-image-registry/image-registry-67474545cf-rgctv"
May 26 16:01:36.772448 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:36.772321    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4a9f9941-c5fa-4e05-97f4-23aeefb46d13-bound-sa-token\") pod \"image-registry-67474545cf-rgctv\" (UID: \"4a9f9941-c5fa-4e05-97f4-23aeefb46d13\") " pod="openshift-image-registry/image-registry-67474545cf-rgctv"
May 26 16:01:36.772448 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:36.772373    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4a9f9941-c5fa-4e05-97f4-23aeefb46d13-image-registry-private-configuration\") pod \"image-registry-67474545cf-rgctv\" (UID: \"4a9f9941-c5fa-4e05-97f4-23aeefb46d13\") " pod="openshift-image-registry/image-registry-67474545cf-rgctv"
May 26 16:01:36.772718 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:36.772693    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4a9f9941-c5fa-4e05-97f4-23aeefb46d13-ca-trust-extracted\") pod \"image-registry-67474545cf-rgctv\" (UID: \"4a9f9941-c5fa-4e05-97f4-23aeefb46d13\") " pod="openshift-image-registry/image-registry-67474545cf-rgctv"
May 26 16:01:36.772953 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:36.772927    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4a9f9941-c5fa-4e05-97f4-23aeefb46d13-registry-certificates\") pod \"image-registry-67474545cf-rgctv\" (UID: \"4a9f9941-c5fa-4e05-97f4-23aeefb46d13\") " pod="openshift-image-registry/image-registry-67474545cf-rgctv"
May 26 16:01:36.773252 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:36.773232    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4a9f9941-c5fa-4e05-97f4-23aeefb46d13-trusted-ca\") pod \"image-registry-67474545cf-rgctv\" (UID: \"4a9f9941-c5fa-4e05-97f4-23aeefb46d13\") " pod="openshift-image-registry/image-registry-67474545cf-rgctv"
May 26 16:01:36.774535 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:36.774513    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4a9f9941-c5fa-4e05-97f4-23aeefb46d13-installation-pull-secrets\") pod \"image-registry-67474545cf-rgctv\" (UID: \"4a9f9941-c5fa-4e05-97f4-23aeefb46d13\") " pod="openshift-image-registry/image-registry-67474545cf-rgctv"
May 26 16:01:36.774535 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:36.774529    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4a9f9941-c5fa-4e05-97f4-23aeefb46d13-registry-tls\") pod \"image-registry-67474545cf-rgctv\" (UID: \"4a9f9941-c5fa-4e05-97f4-23aeefb46d13\") " pod="openshift-image-registry/image-registry-67474545cf-rgctv"
May 26 16:01:36.774685 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:36.774664    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4a9f9941-c5fa-4e05-97f4-23aeefb46d13-image-registry-private-configuration\") pod \"image-registry-67474545cf-rgctv\" (UID: \"4a9f9941-c5fa-4e05-97f4-23aeefb46d13\") " pod="openshift-image-registry/image-registry-67474545cf-rgctv"
May 26 16:01:36.789216 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:36.789197    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4a9f9941-c5fa-4e05-97f4-23aeefb46d13-bound-sa-token\") pod \"image-registry-67474545cf-rgctv\" (UID: \"4a9f9941-c5fa-4e05-97f4-23aeefb46d13\") " pod="openshift-image-registry/image-registry-67474545cf-rgctv"
May 26 16:01:36.790969 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:36.790948    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcc5b\" (UniqueName: \"kubernetes.io/projected/4a9f9941-c5fa-4e05-97f4-23aeefb46d13-kube-api-access-dcc5b\") pod \"image-registry-67474545cf-rgctv\" (UID: \"4a9f9941-c5fa-4e05-97f4-23aeefb46d13\") " pod="openshift-image-registry/image-registry-67474545cf-rgctv"
May 26 16:01:36.831763 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:36.831743    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-cq5gn"
May 26 16:01:36.928322 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:36.928299    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-67474545cf-rgctv"
May 26 16:01:36.954902 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:36.954877    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-cq5gn"]
May 26 16:01:36.959656 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:01:36.959620    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3f3da40_e5da_4eab_a2bb_b7f75f9546c8.slice/crio-ccd77a1ea0f41eab2f973b3eae5f6e9f9b001a92ad9e1c444b037dc89704c644 WatchSource:0}: Error finding container ccd77a1ea0f41eab2f973b3eae5f6e9f9b001a92ad9e1c444b037dc89704c644: Status 404 returned error can't find the container with id ccd77a1ea0f41eab2f973b3eae5f6e9f9b001a92ad9e1c444b037dc89704c644
May 26 16:01:37.054716 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:37.054687    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-67474545cf-rgctv"]
May 26 16:01:37.057335 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:01:37.057309    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a9f9941_c5fa_4e05_97f4_23aeefb46d13.slice/crio-953ecdf4792dcf77a77c14b5816d75fdd5655a2458309c15ee1691135a9009eb WatchSource:0}: Error finding container 953ecdf4792dcf77a77c14b5816d75fdd5655a2458309c15ee1691135a9009eb: Status 404 returned error can't find the container with id 953ecdf4792dcf77a77c14b5816d75fdd5655a2458309c15ee1691135a9009eb
May 26 16:01:37.889193 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:37.889169    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-sk7ns"
May 26 16:01:37.963065 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:37.963043    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-cq5gn" event={"ID":"e3f3da40-e5da-4eab-a2bb-b7f75f9546c8","Type":"ContainerStarted","Data":"25646663b60404071af98da948fad2a1c1bcdb559fa25c3bb0b9d13711fc77e0"}
May 26 16:01:37.963184 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:37.963069    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-cq5gn" event={"ID":"e3f3da40-e5da-4eab-a2bb-b7f75f9546c8","Type":"ContainerStarted","Data":"db8b7b2247f465ce8e585be778906bcc01afe810a5990c65d9520a286e27e2d2"}
May 26 16:01:37.963184 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:37.963079    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-cq5gn" event={"ID":"e3f3da40-e5da-4eab-a2bb-b7f75f9546c8","Type":"ContainerStarted","Data":"ccd77a1ea0f41eab2f973b3eae5f6e9f9b001a92ad9e1c444b037dc89704c644"}
May 26 16:01:37.964239 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:37.964220    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-67474545cf-rgctv" event={"ID":"4a9f9941-c5fa-4e05-97f4-23aeefb46d13","Type":"ContainerStarted","Data":"aac49e5da81699f9b3da6376faeadff9646220e2c5bf3390cdd522fe4a6a1606"}
May 26 16:01:37.964327 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:37.964244    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-67474545cf-rgctv" event={"ID":"4a9f9941-c5fa-4e05-97f4-23aeefb46d13","Type":"ContainerStarted","Data":"953ecdf4792dcf77a77c14b5816d75fdd5655a2458309c15ee1691135a9009eb"}
May 26 16:01:37.964327 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:37.964296    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-67474545cf-rgctv"
May 26 16:01:37.987539 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:37.987477    2556 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-image-registry/image-registry-67474545cf-rgctv" podStartSLOduration=1.987466465 podStartE2EDuration="1.987466465s" podCreationTimestamp="2026-05-26 16:01:36 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 16:01:37.98711263 +0000 UTC m=+59.831729618" watchObservedRunningTime="2026-05-26 16:01:37.987466465 +0000 UTC m=+59.832083456"
May 26 16:01:39.972585 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:39.972547    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-cq5gn" event={"ID":"e3f3da40-e5da-4eab-a2bb-b7f75f9546c8","Type":"ContainerStarted","Data":"67e7effd83aa438379c50fe391887afc17ecac4baaf7c5a3ca6c6c5a8b9f2e0a"}
May 26 16:01:40.000804 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:40.000765    2556 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-cq5gn" podStartSLOduration=1.875402622 podStartE2EDuration="4.000752354s" podCreationTimestamp="2026-05-26 16:01:36 +0000 UTC" firstStartedPulling="2026-05-26 16:01:37.091882481 +0000 UTC m=+58.936499451" lastFinishedPulling="2026-05-26 16:01:39.217232211 +0000 UTC m=+61.061849183" observedRunningTime="2026-05-26 16:01:40.000544546 +0000 UTC m=+61.845161539" watchObservedRunningTime="2026-05-26 16:01:40.000752354 +0000 UTC m=+61.845369393"
May 26 16:01:42.955188 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:42.955152    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-blmvt"
May 26 16:01:44.419665 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:44.419638    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aed39145-780f-4642-af53-ddccbc03c34c-metrics-certs\") pod \"network-metrics-daemon-g9scb\" (UID: \"aed39145-780f-4642-af53-ddccbc03c34c\") " pod="openshift-multus/network-metrics-daemon-g9scb"
May 26 16:01:44.422784 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:44.422766    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
May 26 16:01:44.432377 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:44.432350    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aed39145-780f-4642-af53-ddccbc03c34c-metrics-certs\") pod \"network-metrics-daemon-g9scb\" (UID: \"aed39145-780f-4642-af53-ddccbc03c34c\") " pod="openshift-multus/network-metrics-daemon-g9scb"
May 26 16:01:44.520509 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:44.520482    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c0cc81a8-fc3f-4188-8e90-4640d751dfb5-original-pull-secret\") pod \"global-pull-secret-syncer-wc8kh\" (UID: \"c0cc81a8-fc3f-4188-8e90-4640d751dfb5\") " pod="kube-system/global-pull-secret-syncer-wc8kh"
May 26 16:01:44.523878 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:44.523862    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
May 26 16:01:44.533618 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:44.533591    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c0cc81a8-fc3f-4188-8e90-4640d751dfb5-original-pull-secret\") pod \"global-pull-secret-syncer-wc8kh\" (UID: \"c0cc81a8-fc3f-4188-8e90-4640d751dfb5\") " pod="kube-system/global-pull-secret-syncer-wc8kh"
May 26 16:01:44.579816 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:44.579790    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wc8kh"
May 26 16:01:44.595193 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:44.595173    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-f8kf8\""
May 26 16:01:44.602825 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:44.602806    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g9scb"
May 26 16:01:44.621055 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:44.621033    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-xdxbg\" (UniqueName: \"kubernetes.io/projected/115c1521-6997-403b-98d4-38a9198dd28e-kube-api-access-xdxbg\") pod \"network-check-target-twfp8\" (UID: \"115c1521-6997-403b-98d4-38a9198dd28e\") " pod="openshift-network-diagnostics/network-check-target-twfp8"
May 26 16:01:44.624703 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:44.624503    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
May 26 16:01:44.635243 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:44.635011    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
May 26 16:01:44.645042 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:44.645021    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdxbg\" (UniqueName: \"kubernetes.io/projected/115c1521-6997-403b-98d4-38a9198dd28e-kube-api-access-xdxbg\") pod \"network-check-target-twfp8\" (UID: \"115c1521-6997-403b-98d4-38a9198dd28e\") " pod="openshift-network-diagnostics/network-check-target-twfp8"
May 26 16:01:44.733178 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:44.732860    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-wc8kh"]
May 26 16:01:44.733418 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:01:44.733390    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0cc81a8_fc3f_4188_8e90_4640d751dfb5.slice/crio-a09de927f7424b2ca137e83832656e7db6c82cb52dcdbfcc76b57c3ede4fd718 WatchSource:0}: Error finding container a09de927f7424b2ca137e83832656e7db6c82cb52dcdbfcc76b57c3ede4fd718: Status 404 returned error can't find the container with id a09de927f7424b2ca137e83832656e7db6c82cb52dcdbfcc76b57c3ede4fd718
May 26 16:01:44.764744 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:44.764721    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-g9scb"]
May 26 16:01:44.767726 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:01:44.767698    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaed39145_780f_4642_af53_ddccbc03c34c.slice/crio-c38862eb391b4c934e0cb40f0ae312adcc3b4a1bd48e31b7657c5db72d72d736 WatchSource:0}: Error finding container c38862eb391b4c934e0cb40f0ae312adcc3b4a1bd48e31b7657c5db72d72d736: Status 404 returned error can't find the container with id c38862eb391b4c934e0cb40f0ae312adcc3b4a1bd48e31b7657c5db72d72d736
May 26 16:01:44.890103 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:44.890071    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-8h4d8\""
May 26 16:01:44.897568 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:44.897551    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-twfp8"
May 26 16:01:44.986897 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:44.986858    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-g9scb" event={"ID":"aed39145-780f-4642-af53-ddccbc03c34c","Type":"ContainerStarted","Data":"c38862eb391b4c934e0cb40f0ae312adcc3b4a1bd48e31b7657c5db72d72d736"}
May 26 16:01:44.987905 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:44.987882    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-wc8kh" event={"ID":"c0cc81a8-fc3f-4188-8e90-4640d751dfb5","Type":"ContainerStarted","Data":"a09de927f7424b2ca137e83832656e7db6c82cb52dcdbfcc76b57c3ede4fd718"}
May 26 16:01:45.030531 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:45.030498    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-twfp8"]
May 26 16:01:45.036519 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:01:45.036498    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod115c1521_6997_403b_98d4_38a9198dd28e.slice/crio-27570055eab416b08a760700ca5f2587c516a06b529e5716bce39a0af7be8b25 WatchSource:0}: Error finding container 27570055eab416b08a760700ca5f2587c516a06b529e5716bce39a0af7be8b25: Status 404 returned error can't find the container with id 27570055eab416b08a760700ca5f2587c516a06b529e5716bce39a0af7be8b25
May 26 16:01:45.992389 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:45.992352    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-twfp8" event={"ID":"115c1521-6997-403b-98d4-38a9198dd28e","Type":"ContainerStarted","Data":"27570055eab416b08a760700ca5f2587c516a06b529e5716bce39a0af7be8b25"}
May 26 16:01:46.998390 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:46.998352    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-g9scb" event={"ID":"aed39145-780f-4642-af53-ddccbc03c34c","Type":"ContainerStarted","Data":"3a727d20884d2a753a23ee08e938e0c27a75570394d532420c950ddb14c126ff"}
May 26 16:01:46.998390 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:46.998393    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-g9scb" event={"ID":"aed39145-780f-4642-af53-ddccbc03c34c","Type":"ContainerStarted","Data":"1a9bc8fc3586b12f0f4410acb97f71ba450e66be4490e57e2794ebb3be29f7b1"}
May 26 16:01:47.030296 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:47.030239    2556 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-g9scb" podStartSLOduration=67.541565977 podStartE2EDuration="1m9.030221619s" podCreationTimestamp="2026-05-26 16:00:38 +0000 UTC" firstStartedPulling="2026-05-26 16:01:44.769503901 +0000 UTC m=+66.614120870" lastFinishedPulling="2026-05-26 16:01:46.258159527 +0000 UTC m=+68.102776512" observedRunningTime="2026-05-26 16:01:47.025474445 +0000 UTC m=+68.870091431" watchObservedRunningTime="2026-05-26 16:01:47.030221619 +0000 UTC m=+68.874838612"
May 26 16:01:50.008263 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:50.008223    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-twfp8" event={"ID":"115c1521-6997-403b-98d4-38a9198dd28e","Type":"ContainerStarted","Data":"e9ce468cf0c2c7eefb4250c55f9ffc05ff5010dd10dbb0445294c5f5f5049fe4"}
May 26 16:01:50.008756 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:50.008294    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-twfp8"
May 26 16:01:50.009492 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:50.009471    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-wc8kh" event={"ID":"c0cc81a8-fc3f-4188-8e90-4640d751dfb5","Type":"ContainerStarted","Data":"e684b403cd4a56064087cca217b8e04c94ba8a311e90de43366ecd952f325591"}
May 26 16:01:50.030007 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:50.029966    2556 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-twfp8" podStartSLOduration=67.600882062 podStartE2EDuration="1m12.029955421s" podCreationTimestamp="2026-05-26 16:00:38 +0000 UTC" firstStartedPulling="2026-05-26 16:01:45.037882323 +0000 UTC m=+66.882499293" lastFinishedPulling="2026-05-26 16:01:49.466955666 +0000 UTC m=+71.311572652" observedRunningTime="2026-05-26 16:01:50.029665832 +0000 UTC m=+71.874282827" watchObservedRunningTime="2026-05-26 16:01:50.029955421 +0000 UTC m=+71.874572413"
May 26 16:01:50.051003 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:50.050957    2556 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-wc8kh" podStartSLOduration=66.314287511 podStartE2EDuration="1m11.050945383s" podCreationTimestamp="2026-05-26 16:00:39 +0000 UTC" firstStartedPulling="2026-05-26 16:01:44.735073011 +0000 UTC m=+66.579689985" lastFinishedPulling="2026-05-26 16:01:49.471730873 +0000 UTC m=+71.316347857" observedRunningTime="2026-05-26 16:01:50.050629977 +0000 UTC m=+71.895246972" watchObservedRunningTime="2026-05-26 16:01:50.050945383 +0000 UTC m=+71.895562376"
May 26 16:01:57.641889 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:57.641785    2556 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console/console-69c578695f-2xjn6"]
May 26 16:01:57.643519 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:57.643502    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-69c578695f-2xjn6"
May 26 16:01:57.648159 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:57.648138    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
May 26 16:01:57.648763 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:57.648737    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
May 26 16:01:57.649615 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:57.649400    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
May 26 16:01:57.649776 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:57.649755    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
May 26 16:01:57.649887 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:57.649868    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-tf8sq\""
May 26 16:01:57.649948 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:57.649902    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
May 26 16:01:57.650407 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:57.650392    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
May 26 16:01:57.653435 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:57.653421    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
May 26 16:01:57.671288 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:57.671243    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-69c578695f-2xjn6"]
May 26 16:01:57.708450 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:57.708427    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/48f886b1-5962-43d8-a61c-798284e98e29-console-config\") pod \"console-69c578695f-2xjn6\" (UID: \"48f886b1-5962-43d8-a61c-798284e98e29\") " pod="openshift-console/console-69c578695f-2xjn6"
May 26 16:01:57.708538 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:57.708458    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/48f886b1-5962-43d8-a61c-798284e98e29-console-oauth-config\") pod \"console-69c578695f-2xjn6\" (UID: \"48f886b1-5962-43d8-a61c-798284e98e29\") " pod="openshift-console/console-69c578695f-2xjn6"
May 26 16:01:57.708538 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:57.708475    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/48f886b1-5962-43d8-a61c-798284e98e29-oauth-serving-cert\") pod \"console-69c578695f-2xjn6\" (UID: \"48f886b1-5962-43d8-a61c-798284e98e29\") " pod="openshift-console/console-69c578695f-2xjn6"
May 26 16:01:57.708538 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:57.708490    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/48f886b1-5962-43d8-a61c-798284e98e29-service-ca\") pod \"console-69c578695f-2xjn6\" (UID: \"48f886b1-5962-43d8-a61c-798284e98e29\") " pod="openshift-console/console-69c578695f-2xjn6"
May 26 16:01:57.708647 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:57.708584    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zc65h\" (UniqueName: \"kubernetes.io/projected/48f886b1-5962-43d8-a61c-798284e98e29-kube-api-access-zc65h\") pod \"console-69c578695f-2xjn6\" (UID: \"48f886b1-5962-43d8-a61c-798284e98e29\") " pod="openshift-console/console-69c578695f-2xjn6"
May 26 16:01:57.708647 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:57.708625    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/48f886b1-5962-43d8-a61c-798284e98e29-console-serving-cert\") pod \"console-69c578695f-2xjn6\" (UID: \"48f886b1-5962-43d8-a61c-798284e98e29\") " pod="openshift-console/console-69c578695f-2xjn6"
May 26 16:01:57.809182 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:57.809157    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/48f886b1-5962-43d8-a61c-798284e98e29-console-oauth-config\") pod \"console-69c578695f-2xjn6\" (UID: \"48f886b1-5962-43d8-a61c-798284e98e29\") " pod="openshift-console/console-69c578695f-2xjn6"
May 26 16:01:57.809182 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:57.809183    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/48f886b1-5962-43d8-a61c-798284e98e29-oauth-serving-cert\") pod \"console-69c578695f-2xjn6\" (UID: \"48f886b1-5962-43d8-a61c-798284e98e29\") " pod="openshift-console/console-69c578695f-2xjn6"
May 26 16:01:57.809307 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:57.809201    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/48f886b1-5962-43d8-a61c-798284e98e29-service-ca\") pod \"console-69c578695f-2xjn6\" (UID: \"48f886b1-5962-43d8-a61c-798284e98e29\") " pod="openshift-console/console-69c578695f-2xjn6"
May 26 16:01:57.809353 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:57.809331    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-zc65h\" (UniqueName: \"kubernetes.io/projected/48f886b1-5962-43d8-a61c-798284e98e29-kube-api-access-zc65h\") pod \"console-69c578695f-2xjn6\" (UID: \"48f886b1-5962-43d8-a61c-798284e98e29\") " pod="openshift-console/console-69c578695f-2xjn6"
May 26 16:01:57.809403 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:57.809389    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/48f886b1-5962-43d8-a61c-798284e98e29-console-serving-cert\") pod \"console-69c578695f-2xjn6\" (UID: \"48f886b1-5962-43d8-a61c-798284e98e29\") " pod="openshift-console/console-69c578695f-2xjn6"
May 26 16:01:57.809457 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:57.809442    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/48f886b1-5962-43d8-a61c-798284e98e29-console-config\") pod \"console-69c578695f-2xjn6\" (UID: \"48f886b1-5962-43d8-a61c-798284e98e29\") " pod="openshift-console/console-69c578695f-2xjn6"
May 26 16:01:57.809822 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:57.809796    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/48f886b1-5962-43d8-a61c-798284e98e29-service-ca\") pod \"console-69c578695f-2xjn6\" (UID: \"48f886b1-5962-43d8-a61c-798284e98e29\") " pod="openshift-console/console-69c578695f-2xjn6"
May 26 16:01:57.809933 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:57.809912    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/48f886b1-5962-43d8-a61c-798284e98e29-oauth-serving-cert\") pod \"console-69c578695f-2xjn6\" (UID: \"48f886b1-5962-43d8-a61c-798284e98e29\") " pod="openshift-console/console-69c578695f-2xjn6"
May 26 16:01:57.810016 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:57.810001    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/48f886b1-5962-43d8-a61c-798284e98e29-console-config\") pod \"console-69c578695f-2xjn6\" (UID: \"48f886b1-5962-43d8-a61c-798284e98e29\") " pod="openshift-console/console-69c578695f-2xjn6"
May 26 16:01:57.811545 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:57.811527    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/48f886b1-5962-43d8-a61c-798284e98e29-console-oauth-config\") pod \"console-69c578695f-2xjn6\" (UID: \"48f886b1-5962-43d8-a61c-798284e98e29\") " pod="openshift-console/console-69c578695f-2xjn6"
May 26 16:01:57.811631 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:57.811608    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/48f886b1-5962-43d8-a61c-798284e98e29-console-serving-cert\") pod \"console-69c578695f-2xjn6\" (UID: \"48f886b1-5962-43d8-a61c-798284e98e29\") " pod="openshift-console/console-69c578695f-2xjn6"
May 26 16:01:57.819866 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:57.819840    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zc65h\" (UniqueName: \"kubernetes.io/projected/48f886b1-5962-43d8-a61c-798284e98e29-kube-api-access-zc65h\") pod \"console-69c578695f-2xjn6\" (UID: \"48f886b1-5962-43d8-a61c-798284e98e29\") " pod="openshift-console/console-69c578695f-2xjn6"
May 26 16:01:57.952287 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:57.952226    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-69c578695f-2xjn6"
May 26 16:01:58.080682 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:58.080650    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-69c578695f-2xjn6"]
May 26 16:01:58.083676 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:01:58.083640    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48f886b1_5962_43d8_a61c_798284e98e29.slice/crio-f4bf8453a98011a271bdb6cb35cb110265c871589c473973e8b07f9f9c31b83e WatchSource:0}: Error finding container f4bf8453a98011a271bdb6cb35cb110265c871589c473973e8b07f9f9c31b83e: Status 404 returned error can't find the container with id f4bf8453a98011a271bdb6cb35cb110265c871589c473973e8b07f9f9c31b83e
May 26 16:01:58.972149 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:58.972104    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-67474545cf-rgctv"
May 26 16:01:59.038722 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:01:59.038682    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-69c578695f-2xjn6" event={"ID":"48f886b1-5962-43d8-a61c-798284e98e29","Type":"ContainerStarted","Data":"f4bf8453a98011a271bdb6cb35cb110265c871589c473973e8b07f9f9c31b83e"}
May 26 16:02:02.048055 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:02:02.048012    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-69c578695f-2xjn6" event={"ID":"48f886b1-5962-43d8-a61c-798284e98e29","Type":"ContainerStarted","Data":"3fea6be77abbcfbcfc1947b6b26d4558b5f8d78a7916b3905b36ce2541d63eda"}
May 26 16:02:02.068519 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:02:02.068472    2556 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console/console-69c578695f-2xjn6" podStartSLOduration=2.044310361 podStartE2EDuration="5.068457816s" podCreationTimestamp="2026-05-26 16:01:57 +0000 UTC" firstStartedPulling="2026-05-26 16:01:58.085671555 +0000 UTC m=+79.930288528" lastFinishedPulling="2026-05-26 16:02:01.10981901 +0000 UTC m=+82.954435983" observedRunningTime="2026-05-26 16:02:02.067557472 +0000 UTC m=+83.912174461" watchObservedRunningTime="2026-05-26 16:02:02.068457816 +0000 UTC m=+83.913074808"
May 26 16:02:07.952698 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:02:07.952665    2556 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-69c578695f-2xjn6"
May 26 16:02:07.953222 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:02:07.952734    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-69c578695f-2xjn6"
May 26 16:02:07.957387 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:02:07.957368    2556 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-69c578695f-2xjn6"
May 26 16:02:08.067407 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:02:08.067387    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-69c578695f-2xjn6"
May 26 16:02:10.511214 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:02:10.511183    2556 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-69c578695f-2xjn6"]
May 26 16:02:13.953861 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:02:13.953826    2556 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-9q9lz"]
May 26 16:02:13.960813 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:02:13.960784    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-9q9lz"
May 26 16:02:13.963790 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:02:13.963765    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
May 26 16:02:13.963916 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:02:13.963765    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
May 26 16:02:13.963916 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:02:13.963869    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
May 26 16:02:13.964769 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:02:13.964751    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
May 26 16:02:13.965553 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:02:13.965536    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
May 26 16:02:13.966304 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:02:13.966287    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-4wtsn\""
May 26 16:02:13.968251 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:02:13.968235    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
May 26 16:02:14.022529 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:02:14.022488    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/88e521b6-1620-4720-8a54-35939b468812-node-exporter-textfile\") pod \"node-exporter-9q9lz\" (UID: \"88e521b6-1620-4720-8a54-35939b468812\") " pod="openshift-monitoring/node-exporter-9q9lz"
May 26 16:02:14.022729 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:02:14.022537    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/88e521b6-1620-4720-8a54-35939b468812-sys\") pod \"node-exporter-9q9lz\" (UID: \"88e521b6-1620-4720-8a54-35939b468812\") " pod="openshift-monitoring/node-exporter-9q9lz"
May 26 16:02:14.022729 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:02:14.022628    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/88e521b6-1620-4720-8a54-35939b468812-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9q9lz\" (UID: \"88e521b6-1620-4720-8a54-35939b468812\") " pod="openshift-monitoring/node-exporter-9q9lz"
May 26 16:02:14.022729 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:02:14.022657    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/88e521b6-1620-4720-8a54-35939b468812-node-exporter-accelerators-collector-config\") pod \"node-exporter-9q9lz\" (UID: \"88e521b6-1620-4720-8a54-35939b468812\") " pod="openshift-monitoring/node-exporter-9q9lz"
May 26 16:02:14.022729 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:02:14.022687    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rql8\" (UniqueName: \"kubernetes.io/projected/88e521b6-1620-4720-8a54-35939b468812-kube-api-access-7rql8\") pod \"node-exporter-9q9lz\" (UID: \"88e521b6-1620-4720-8a54-35939b468812\") " pod="openshift-monitoring/node-exporter-9q9lz"
May 26 16:02:14.022942 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:02:14.022745    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/88e521b6-1620-4720-8a54-35939b468812-node-exporter-tls\") pod \"node-exporter-9q9lz\" (UID: \"88e521b6-1620-4720-8a54-35939b468812\") " pod="openshift-monitoring/node-exporter-9q9lz"
May 26 16:02:14.022942 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:02:14.022773    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/88e521b6-1620-4720-8a54-35939b468812-node-exporter-wtmp\") pod \"node-exporter-9q9lz\" (UID: \"88e521b6-1620-4720-8a54-35939b468812\") " pod="openshift-monitoring/node-exporter-9q9lz"
May 26 16:02:14.022942 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:02:14.022802    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/88e521b6-1620-4720-8a54-35939b468812-metrics-client-ca\") pod \"node-exporter-9q9lz\" (UID: \"88e521b6-1620-4720-8a54-35939b468812\") " pod="openshift-monitoring/node-exporter-9q9lz"
May 26 16:02:14.022942 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:02:14.022830    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/88e521b6-1620-4720-8a54-35939b468812-root\") pod \"node-exporter-9q9lz\" (UID: \"88e521b6-1620-4720-8a54-35939b468812\") " pod="openshift-monitoring/node-exporter-9q9lz"
May 26 16:02:14.123877 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:02:14.123837    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/88e521b6-1620-4720-8a54-35939b468812-node-exporter-textfile\") pod \"node-exporter-9q9lz\" (UID: \"88e521b6-1620-4720-8a54-35939b468812\") " pod="openshift-monitoring/node-exporter-9q9lz"
May 26 16:02:14.124073 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:02:14.123880    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/88e521b6-1620-4720-8a54-35939b468812-sys\") pod \"node-exporter-9q9lz\" (UID: \"88e521b6-1620-4720-8a54-35939b468812\") " pod="openshift-monitoring/node-exporter-9q9lz"
May 26 16:02:14.124073 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:02:14.123936    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/88e521b6-1620-4720-8a54-35939b468812-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9q9lz\" (UID: \"88e521b6-1620-4720-8a54-35939b468812\") " pod="openshift-monitoring/node-exporter-9q9lz"
May 26 16:02:14.124073 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:02:14.123968    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/88e521b6-1620-4720-8a54-35939b468812-node-exporter-accelerators-collector-config\") pod \"node-exporter-9q9lz\" (UID: \"88e521b6-1620-4720-8a54-35939b468812\") " pod="openshift-monitoring/node-exporter-9q9lz"
May 26 16:02:14.124073 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:02:14.124005    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-7rql8\" (UniqueName: \"kubernetes.io/projected/88e521b6-1620-4720-8a54-35939b468812-kube-api-access-7rql8\") pod \"node-exporter-9q9lz\" (UID: \"88e521b6-1620-4720-8a54-35939b468812\") " pod="openshift-monitoring/node-exporter-9q9lz"
May 26 16:02:14.124073 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:02:14.124036    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/88e521b6-1620-4720-8a54-35939b468812-node-exporter-tls\") pod \"node-exporter-9q9lz\" (UID: \"88e521b6-1620-4720-8a54-35939b468812\") " pod="openshift-monitoring/node-exporter-9q9lz"
May 26 16:02:14.124073 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:02:14.124041    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/88e521b6-1620-4720-8a54-35939b468812-sys\") pod \"node-exporter-9q9lz\" (UID: \"88e521b6-1620-4720-8a54-35939b468812\") " pod="openshift-monitoring/node-exporter-9q9lz"
May 26 16:02:14.124073 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:02:14.124060    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/88e521b6-1620-4720-8a54-35939b468812-node-exporter-wtmp\") pod \"node-exporter-9q9lz\" (UID: \"88e521b6-1620-4720-8a54-35939b468812\") " pod="openshift-monitoring/node-exporter-9q9lz"
May 26 16:02:14.124459 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:02:14.124084    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/88e521b6-1620-4720-8a54-35939b468812-metrics-client-ca\") pod \"node-exporter-9q9lz\" (UID: \"88e521b6-1620-4720-8a54-35939b468812\") " pod="openshift-monitoring/node-exporter-9q9lz"
May 26 16:02:14.124459 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:02:14.124133    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/88e521b6-1620-4720-8a54-35939b468812-root\") pod \"node-exporter-9q9lz\" (UID: \"88e521b6-1620-4720-8a54-35939b468812\") " pod="openshift-monitoring/node-exporter-9q9lz"
May 26 16:02:14.124459 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:02:14.124223    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/88e521b6-1620-4720-8a54-35939b468812-root\") pod \"node-exporter-9q9lz\" (UID: \"88e521b6-1620-4720-8a54-35939b468812\") " pod="openshift-monitoring/node-exporter-9q9lz"
May 26 16:02:14.124459 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:02:14.124293    2556 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
May 26 16:02:14.124459 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:02:14.124326    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/88e521b6-1620-4720-8a54-35939b468812-node-exporter-textfile\") pod \"node-exporter-9q9lz\" (UID: \"88e521b6-1620-4720-8a54-35939b468812\") " pod="openshift-monitoring/node-exporter-9q9lz"
May 26 16:02:14.124459 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:02:14.124339    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/88e521b6-1620-4720-8a54-35939b468812-node-exporter-wtmp\") pod \"node-exporter-9q9lz\" (UID: \"88e521b6-1620-4720-8a54-35939b468812\") " pod="openshift-monitoring/node-exporter-9q9lz"
May 26 16:02:14.124459 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:02:14.124369    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/88e521b6-1620-4720-8a54-35939b468812-node-exporter-tls podName:88e521b6-1620-4720-8a54-35939b468812 nodeName:}" failed. No retries permitted until 2026-05-26 16:02:14.624347411 +0000 UTC m=+96.468964399 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/88e521b6-1620-4720-8a54-35939b468812-node-exporter-tls") pod "node-exporter-9q9lz" (UID: "88e521b6-1620-4720-8a54-35939b468812") : secret "node-exporter-tls" not found
May 26 16:02:14.124873 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:02:14.124853    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/88e521b6-1620-4720-8a54-35939b468812-node-exporter-accelerators-collector-config\") pod \"node-exporter-9q9lz\" (UID: \"88e521b6-1620-4720-8a54-35939b468812\") " pod="openshift-monitoring/node-exporter-9q9lz"
May 26 16:02:14.124927 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:02:14.124882    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/88e521b6-1620-4720-8a54-35939b468812-metrics-client-ca\") pod \"node-exporter-9q9lz\" (UID: \"88e521b6-1620-4720-8a54-35939b468812\") " pod="openshift-monitoring/node-exporter-9q9lz"
May 26 16:02:14.126430 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:02:14.126410    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/88e521b6-1620-4720-8a54-35939b468812-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9q9lz\" (UID: \"88e521b6-1620-4720-8a54-35939b468812\") " pod="openshift-monitoring/node-exporter-9q9lz"
May 26 16:02:14.134260 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:02:14.134237    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rql8\" (UniqueName: \"kubernetes.io/projected/88e521b6-1620-4720-8a54-35939b468812-kube-api-access-7rql8\") pod \"node-exporter-9q9lz\" (UID: \"88e521b6-1620-4720-8a54-35939b468812\") " pod="openshift-monitoring/node-exporter-9q9lz"
May 26 16:02:14.628795 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:02:14.628760    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/88e521b6-1620-4720-8a54-35939b468812-node-exporter-tls\") pod \"node-exporter-9q9lz\" (UID: \"88e521b6-1620-4720-8a54-35939b468812\") " pod="openshift-monitoring/node-exporter-9q9lz"
May 26 16:02:14.630978 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:02:14.630948    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/88e521b6-1620-4720-8a54-35939b468812-node-exporter-tls\") pod \"node-exporter-9q9lz\" (UID: \"88e521b6-1620-4720-8a54-35939b468812\") " pod="openshift-monitoring/node-exporter-9q9lz"
May 26 16:02:14.872007 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:02:14.871978    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-9q9lz"
May 26 16:02:14.881132 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:02:14.881056    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88e521b6_1620_4720_8a54_35939b468812.slice/crio-266abd5be68fd1b1e71caf48c34ac04be9117dcf0baa18dcab101e321681447e WatchSource:0}: Error finding container 266abd5be68fd1b1e71caf48c34ac04be9117dcf0baa18dcab101e321681447e: Status 404 returned error can't find the container with id 266abd5be68fd1b1e71caf48c34ac04be9117dcf0baa18dcab101e321681447e
May 26 16:02:15.081146 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:02:15.081107    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9q9lz" event={"ID":"88e521b6-1620-4720-8a54-35939b468812","Type":"ContainerStarted","Data":"266abd5be68fd1b1e71caf48c34ac04be9117dcf0baa18dcab101e321681447e"}
May 26 16:02:16.086707 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:02:16.086672    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9q9lz" event={"ID":"88e521b6-1620-4720-8a54-35939b468812","Type":"ContainerStarted","Data":"d02e1c8c76d85d94caf5bb27802f3024c7940828cc74327a34c0c57752b41be0"}
May 26 16:02:17.090823 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:02:17.090790    2556 generic.go:358] "Generic (PLEG): container finished" podID="88e521b6-1620-4720-8a54-35939b468812" containerID="d02e1c8c76d85d94caf5bb27802f3024c7940828cc74327a34c0c57752b41be0" exitCode=0
May 26 16:02:17.091214 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:02:17.090848    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9q9lz" event={"ID":"88e521b6-1620-4720-8a54-35939b468812","Type":"ContainerDied","Data":"d02e1c8c76d85d94caf5bb27802f3024c7940828cc74327a34c0c57752b41be0"}
May 26 16:02:17.760774 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:02:17.760749    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-tkpb6_001b5d49-d123-4cec-85c4-e7f8927bf13a/serve-healthcheck-canary/0.log"
May 26 16:02:18.095416 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:02:18.095328    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9q9lz" event={"ID":"88e521b6-1620-4720-8a54-35939b468812","Type":"ContainerStarted","Data":"ef9d17506809e9c9a385520fc3c0c1692f2753985358f8d601d7059a7cd72576"}
May 26 16:02:18.095416 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:02:18.095362    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9q9lz" event={"ID":"88e521b6-1620-4720-8a54-35939b468812","Type":"ContainerStarted","Data":"ddb47ca17bfa573b8803d2c52b04cdfd5ee30392d74bd94da4787974f53c52f6"}
May 26 16:02:18.749620 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:02:18.749567    2556 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-9q9lz" podStartSLOduration=4.60938318 podStartE2EDuration="5.749553236s" podCreationTimestamp="2026-05-26 16:02:13 +0000 UTC" firstStartedPulling="2026-05-26 16:02:14.883076596 +0000 UTC m=+96.727693566" lastFinishedPulling="2026-05-26 16:02:16.023246649 +0000 UTC m=+97.867863622" observedRunningTime="2026-05-26 16:02:18.125218344 +0000 UTC m=+99.969835348" watchObservedRunningTime="2026-05-26 16:02:18.749553236 +0000 UTC m=+100.594170228"
May 26 16:02:18.750377 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:02:18.750360    2556 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-665c9c8584-2v9mb"]
May 26 16:02:18.753397 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:02:18.753382    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-665c9c8584-2v9mb"
May 26 16:02:18.758416 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:02:18.758393    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
May 26 16:02:18.758549 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:02:18.758440    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"default-dockercfg-gnj8h\""
May 26 16:02:18.763966 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:02:18.763944    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-665c9c8584-2v9mb"]
May 26 16:02:18.859688 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:02:18.859658    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a06fc334-b590-4dad-a438-7ab5f9285f1f-monitoring-plugin-cert\") pod \"monitoring-plugin-665c9c8584-2v9mb\" (UID: \"a06fc334-b590-4dad-a438-7ab5f9285f1f\") " pod="openshift-monitoring/monitoring-plugin-665c9c8584-2v9mb"
May 26 16:02:18.960515 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:02:18.960485    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a06fc334-b590-4dad-a438-7ab5f9285f1f-monitoring-plugin-cert\") pod \"monitoring-plugin-665c9c8584-2v9mb\" (UID: \"a06fc334-b590-4dad-a438-7ab5f9285f1f\") " pod="openshift-monitoring/monitoring-plugin-665c9c8584-2v9mb"
May 26 16:02:18.960668 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:02:18.960598    2556 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found
May 26 16:02:18.960668 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:02:18.960650    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a06fc334-b590-4dad-a438-7ab5f9285f1f-monitoring-plugin-cert podName:a06fc334-b590-4dad-a438-7ab5f9285f1f nodeName:}" failed. No retries permitted until 2026-05-26 16:02:19.460634681 +0000 UTC m=+101.305251652 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/a06fc334-b590-4dad-a438-7ab5f9285f1f-monitoring-plugin-cert") pod "monitoring-plugin-665c9c8584-2v9mb" (UID: "a06fc334-b590-4dad-a438-7ab5f9285f1f") : secret "monitoring-plugin-cert" not found
May 26 16:02:19.463885 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:02:19.463854    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a06fc334-b590-4dad-a438-7ab5f9285f1f-monitoring-plugin-cert\") pod \"monitoring-plugin-665c9c8584-2v9mb\" (UID: \"a06fc334-b590-4dad-a438-7ab5f9285f1f\") " pod="openshift-monitoring/monitoring-plugin-665c9c8584-2v9mb"
May 26 16:02:19.466153 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:02:19.466132    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a06fc334-b590-4dad-a438-7ab5f9285f1f-monitoring-plugin-cert\") pod \"monitoring-plugin-665c9c8584-2v9mb\" (UID: \"a06fc334-b590-4dad-a438-7ab5f9285f1f\") " pod="openshift-monitoring/monitoring-plugin-665c9c8584-2v9mb"
May 26 16:02:19.661706 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:02:19.661679    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-665c9c8584-2v9mb"
May 26 16:02:19.788379 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:02:19.788351    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-665c9c8584-2v9mb"]
May 26 16:02:19.791692 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:02:19.791669    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda06fc334_b590_4dad_a438_7ab5f9285f1f.slice/crio-46bb67da9d4850dfd2f3c54fde102fc5d50b3e55943ec4cbd0860c8abc168f0c WatchSource:0}: Error finding container 46bb67da9d4850dfd2f3c54fde102fc5d50b3e55943ec4cbd0860c8abc168f0c: Status 404 returned error can't find the container with id 46bb67da9d4850dfd2f3c54fde102fc5d50b3e55943ec4cbd0860c8abc168f0c
May 26 16:02:20.101218 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:02:20.101134    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-665c9c8584-2v9mb" event={"ID":"a06fc334-b590-4dad-a438-7ab5f9285f1f","Type":"ContainerStarted","Data":"46bb67da9d4850dfd2f3c54fde102fc5d50b3e55943ec4cbd0860c8abc168f0c"}
May 26 16:02:21.015155 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:02:21.015061    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-twfp8"
May 26 16:02:22.108044 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:02:22.108003    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-665c9c8584-2v9mb" event={"ID":"a06fc334-b590-4dad-a438-7ab5f9285f1f","Type":"ContainerStarted","Data":"ca44fbdde9bc7b66ff3f6b329f3a32ec8fff57093a69fec79c6898bcbeeefde5"}
May 26 16:02:22.108470 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:02:22.108260    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-665c9c8584-2v9mb"
May 26 16:02:22.112983 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:02:22.112963    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-665c9c8584-2v9mb"
May 26 16:02:22.216607 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:02:22.216562    2556 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-665c9c8584-2v9mb" podStartSLOduration=2.491251095 podStartE2EDuration="4.21654986s" podCreationTimestamp="2026-05-26 16:02:18 +0000 UTC" firstStartedPulling="2026-05-26 16:02:19.793536113 +0000 UTC m=+101.638153083" lastFinishedPulling="2026-05-26 16:02:21.518834877 +0000 UTC m=+103.363451848" observedRunningTime="2026-05-26 16:02:22.181860544 +0000 UTC m=+104.026477537" watchObservedRunningTime="2026-05-26 16:02:22.21654986 +0000 UTC m=+104.061166852"
May 26 16:02:36.088735 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:02:36.088697    2556 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-console/console-69c578695f-2xjn6" podUID="48f886b1-5962-43d8-a61c-798284e98e29" containerName="console" containerID="cri-o://3fea6be77abbcfbcfc1947b6b26d4558b5f8d78a7916b3905b36ce2541d63eda" gracePeriod=15
May 26 16:02:36.321168 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:02:36.321148    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-69c578695f-2xjn6_48f886b1-5962-43d8-a61c-798284e98e29/console/0.log"
May 26 16:02:36.321264 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:02:36.321219    2556 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-69c578695f-2xjn6"
May 26 16:02:36.476162 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:02:36.476129    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/48f886b1-5962-43d8-a61c-798284e98e29-service-ca\") pod \"48f886b1-5962-43d8-a61c-798284e98e29\" (UID: \"48f886b1-5962-43d8-a61c-798284e98e29\") "
May 26 16:02:36.476295 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:02:36.476185    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/48f886b1-5962-43d8-a61c-798284e98e29-oauth-serving-cert\") pod \"48f886b1-5962-43d8-a61c-798284e98e29\" (UID: \"48f886b1-5962-43d8-a61c-798284e98e29\") "
May 26 16:02:36.476295 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:02:36.476213    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zc65h\" (UniqueName: \"kubernetes.io/projected/48f886b1-5962-43d8-a61c-798284e98e29-kube-api-access-zc65h\") pod \"48f886b1-5962-43d8-a61c-798284e98e29\" (UID: \"48f886b1-5962-43d8-a61c-798284e98e29\") "
May 26 16:02:36.476373 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:02:36.476313    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/48f886b1-5962-43d8-a61c-798284e98e29-console-serving-cert\") pod \"48f886b1-5962-43d8-a61c-798284e98e29\" (UID: \"48f886b1-5962-43d8-a61c-798284e98e29\") "
May 26 16:02:36.476373 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:02:36.476357    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/48f886b1-5962-43d8-a61c-798284e98e29-console-oauth-config\") pod \"48f886b1-5962-43d8-a61c-798284e98e29\" (UID: \"48f886b1-5962-43d8-a61c-798284e98e29\") "
May 26 16:02:36.476446 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:02:36.476394    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/48f886b1-5962-43d8-a61c-798284e98e29-console-config\") pod \"48f886b1-5962-43d8-a61c-798284e98e29\" (UID: \"48f886b1-5962-43d8-a61c-798284e98e29\") "
May 26 16:02:36.476572 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:02:36.476540    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48f886b1-5962-43d8-a61c-798284e98e29-service-ca" (OuterVolumeSpecName: "service-ca") pod "48f886b1-5962-43d8-a61c-798284e98e29" (UID: "48f886b1-5962-43d8-a61c-798284e98e29"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 16:02:36.476696 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:02:36.476643    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48f886b1-5962-43d8-a61c-798284e98e29-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "48f886b1-5962-43d8-a61c-798284e98e29" (UID: "48f886b1-5962-43d8-a61c-798284e98e29"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 16:02:36.476866 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:02:36.476841    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48f886b1-5962-43d8-a61c-798284e98e29-console-config" (OuterVolumeSpecName: "console-config") pod "48f886b1-5962-43d8-a61c-798284e98e29" (UID: "48f886b1-5962-43d8-a61c-798284e98e29"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 16:02:36.478491 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:02:36.478467    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48f886b1-5962-43d8-a61c-798284e98e29-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "48f886b1-5962-43d8-a61c-798284e98e29" (UID: "48f886b1-5962-43d8-a61c-798284e98e29"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 16:02:36.478676 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:02:36.478654    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48f886b1-5962-43d8-a61c-798284e98e29-kube-api-access-zc65h" (OuterVolumeSpecName: "kube-api-access-zc65h") pod "48f886b1-5962-43d8-a61c-798284e98e29" (UID: "48f886b1-5962-43d8-a61c-798284e98e29"). InnerVolumeSpecName "kube-api-access-zc65h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 16:02:36.478772 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:02:36.478657    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48f886b1-5962-43d8-a61c-798284e98e29-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "48f886b1-5962-43d8-a61c-798284e98e29" (UID: "48f886b1-5962-43d8-a61c-798284e98e29"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 16:02:36.577269 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:02:36.577246    2556 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/48f886b1-5962-43d8-a61c-798284e98e29-oauth-serving-cert\") on node \"ip-10-0-134-19.ec2.internal\" DevicePath \"\""
May 26 16:02:36.577269 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:02:36.577267    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zc65h\" (UniqueName: \"kubernetes.io/projected/48f886b1-5962-43d8-a61c-798284e98e29-kube-api-access-zc65h\") on node \"ip-10-0-134-19.ec2.internal\" DevicePath \"\""
May 26 16:02:36.577388 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:02:36.577277    2556 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/48f886b1-5962-43d8-a61c-798284e98e29-console-serving-cert\") on node \"ip-10-0-134-19.ec2.internal\" DevicePath \"\""
May 26 16:02:36.577388 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:02:36.577286    2556 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/48f886b1-5962-43d8-a61c-798284e98e29-console-oauth-config\") on node \"ip-10-0-134-19.ec2.internal\" DevicePath \"\""
May 26 16:02:36.577388 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:02:36.577296    2556 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/48f886b1-5962-43d8-a61c-798284e98e29-console-config\") on node \"ip-10-0-134-19.ec2.internal\" DevicePath \"\""
May 26 16:02:36.577388 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:02:36.577305    2556 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/48f886b1-5962-43d8-a61c-798284e98e29-service-ca\") on node \"ip-10-0-134-19.ec2.internal\" DevicePath \"\""
May 26 16:02:37.144366 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:02:37.144343    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-69c578695f-2xjn6_48f886b1-5962-43d8-a61c-798284e98e29/console/0.log"
May 26 16:02:37.144794 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:02:37.144382    2556 generic.go:358] "Generic (PLEG): container finished" podID="48f886b1-5962-43d8-a61c-798284e98e29" containerID="3fea6be77abbcfbcfc1947b6b26d4558b5f8d78a7916b3905b36ce2541d63eda" exitCode=2
May 26 16:02:37.144794 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:02:37.144418    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-69c578695f-2xjn6" event={"ID":"48f886b1-5962-43d8-a61c-798284e98e29","Type":"ContainerDied","Data":"3fea6be77abbcfbcfc1947b6b26d4558b5f8d78a7916b3905b36ce2541d63eda"}
May 26 16:02:37.144794 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:02:37.144467    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-69c578695f-2xjn6" event={"ID":"48f886b1-5962-43d8-a61c-798284e98e29","Type":"ContainerDied","Data":"f4bf8453a98011a271bdb6cb35cb110265c871589c473973e8b07f9f9c31b83e"}
May 26 16:02:37.144794 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:02:37.144477    2556 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-69c578695f-2xjn6"
May 26 16:02:37.144794 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:02:37.144493    2556 scope.go:117] "RemoveContainer" containerID="3fea6be77abbcfbcfc1947b6b26d4558b5f8d78a7916b3905b36ce2541d63eda"
May 26 16:02:37.151998 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:02:37.151982    2556 scope.go:117] "RemoveContainer" containerID="3fea6be77abbcfbcfc1947b6b26d4558b5f8d78a7916b3905b36ce2541d63eda"
May 26 16:02:37.152269 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:02:37.152241    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3fea6be77abbcfbcfc1947b6b26d4558b5f8d78a7916b3905b36ce2541d63eda\": container with ID starting with 3fea6be77abbcfbcfc1947b6b26d4558b5f8d78a7916b3905b36ce2541d63eda not found: ID does not exist" containerID="3fea6be77abbcfbcfc1947b6b26d4558b5f8d78a7916b3905b36ce2541d63eda"
May 26 16:02:37.152365 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:02:37.152270    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3fea6be77abbcfbcfc1947b6b26d4558b5f8d78a7916b3905b36ce2541d63eda"} err="failed to get container status \"3fea6be77abbcfbcfc1947b6b26d4558b5f8d78a7916b3905b36ce2541d63eda\": rpc error: code = NotFound desc = could not find container \"3fea6be77abbcfbcfc1947b6b26d4558b5f8d78a7916b3905b36ce2541d63eda\": container with ID starting with 3fea6be77abbcfbcfc1947b6b26d4558b5f8d78a7916b3905b36ce2541d63eda not found: ID does not exist"
May 26 16:02:37.172545 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:02:37.172517    2556 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-69c578695f-2xjn6"]
May 26 16:02:37.182477 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:02:37.182456    2556 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-69c578695f-2xjn6"]
May 26 16:02:38.769711 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:02:38.769682    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="48f886b1-5962-43d8-a61c-798284e98e29" path="/var/lib/kubelet/pods/48f886b1-5962-43d8-a61c-798284e98e29/volumes"
May 26 16:03:38.366907 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:03:38.366830    2556 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-65b6b594d5-w7prx"]
May 26 16:03:38.367328 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:03:38.367051    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48f886b1-5962-43d8-a61c-798284e98e29" containerName="console"
May 26 16:03:38.367328 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:03:38.367061    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="48f886b1-5962-43d8-a61c-798284e98e29" containerName="console"
May 26 16:03:38.367328 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:03:38.367133    2556 memory_manager.go:356] "RemoveStaleState removing state" podUID="48f886b1-5962-43d8-a61c-798284e98e29" containerName="console"
May 26 16:03:38.369726 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:03:38.369710    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-65b6b594d5-w7prx"
May 26 16:03:38.372513 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:03:38.372491    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
May 26 16:03:38.372652 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:03:38.372546    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-48m6s\""
May 26 16:03:38.373183 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:03:38.373158    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
May 26 16:03:38.373280 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:03:38.373241    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
May 26 16:03:38.373280 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:03:38.373246    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
May 26 16:03:38.373409 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:03:38.373314    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
May 26 16:03:38.379641 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:03:38.379622    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
May 26 16:03:38.380029 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:03:38.380003    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5604c4cf-75ec-4fd0-989b-9e0316b35972-serving-certs-ca-bundle\") pod \"telemeter-client-65b6b594d5-w7prx\" (UID: \"5604c4cf-75ec-4fd0-989b-9e0316b35972\") " pod="openshift-monitoring/telemeter-client-65b6b594d5-w7prx"
May 26 16:03:38.380168 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:03:38.380048    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5604c4cf-75ec-4fd0-989b-9e0316b35972-metrics-client-ca\") pod \"telemeter-client-65b6b594d5-w7prx\" (UID: \"5604c4cf-75ec-4fd0-989b-9e0316b35972\") " pod="openshift-monitoring/telemeter-client-65b6b594d5-w7prx"
May 26 16:03:38.380168 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:03:38.380104    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/5604c4cf-75ec-4fd0-989b-9e0316b35972-secret-telemeter-client\") pod \"telemeter-client-65b6b594d5-w7prx\" (UID: \"5604c4cf-75ec-4fd0-989b-9e0316b35972\") " pod="openshift-monitoring/telemeter-client-65b6b594d5-w7prx"
May 26 16:03:38.380168 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:03:38.380133    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/5604c4cf-75ec-4fd0-989b-9e0316b35972-telemeter-client-tls\") pod \"telemeter-client-65b6b594d5-w7prx\" (UID: \"5604c4cf-75ec-4fd0-989b-9e0316b35972\") " pod="openshift-monitoring/telemeter-client-65b6b594d5-w7prx"
May 26 16:03:38.380629 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:03:38.380191    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2znnq\" (UniqueName: \"kubernetes.io/projected/5604c4cf-75ec-4fd0-989b-9e0316b35972-kube-api-access-2znnq\") pod \"telemeter-client-65b6b594d5-w7prx\" (UID: \"5604c4cf-75ec-4fd0-989b-9e0316b35972\") " pod="openshift-monitoring/telemeter-client-65b6b594d5-w7prx"
May 26 16:03:38.380629 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:03:38.380311    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/5604c4cf-75ec-4fd0-989b-9e0316b35972-federate-client-tls\") pod \"telemeter-client-65b6b594d5-w7prx\" (UID: \"5604c4cf-75ec-4fd0-989b-9e0316b35972\") " pod="openshift-monitoring/telemeter-client-65b6b594d5-w7prx"
May 26 16:03:38.380629 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:03:38.380349    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5604c4cf-75ec-4fd0-989b-9e0316b35972-telemeter-trusted-ca-bundle\") pod \"telemeter-client-65b6b594d5-w7prx\" (UID: \"5604c4cf-75ec-4fd0-989b-9e0316b35972\") " pod="openshift-monitoring/telemeter-client-65b6b594d5-w7prx"
May 26 16:03:38.380629 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:03:38.380383    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5604c4cf-75ec-4fd0-989b-9e0316b35972-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-65b6b594d5-w7prx\" (UID: \"5604c4cf-75ec-4fd0-989b-9e0316b35972\") " pod="openshift-monitoring/telemeter-client-65b6b594d5-w7prx"
May 26 16:03:38.385074 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:03:38.385053    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-65b6b594d5-w7prx"]
May 26 16:03:38.481005 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:03:38.480977    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/5604c4cf-75ec-4fd0-989b-9e0316b35972-federate-client-tls\") pod \"telemeter-client-65b6b594d5-w7prx\" (UID: \"5604c4cf-75ec-4fd0-989b-9e0316b35972\") " pod="openshift-monitoring/telemeter-client-65b6b594d5-w7prx"
May 26 16:03:38.481180 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:03:38.481018    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5604c4cf-75ec-4fd0-989b-9e0316b35972-telemeter-trusted-ca-bundle\") pod \"telemeter-client-65b6b594d5-w7prx\" (UID: \"5604c4cf-75ec-4fd0-989b-9e0316b35972\") " pod="openshift-monitoring/telemeter-client-65b6b594d5-w7prx"
May 26 16:03:38.481180 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:03:38.481158    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5604c4cf-75ec-4fd0-989b-9e0316b35972-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-65b6b594d5-w7prx\" (UID: \"5604c4cf-75ec-4fd0-989b-9e0316b35972\") " pod="openshift-monitoring/telemeter-client-65b6b594d5-w7prx"
May 26 16:03:38.481291 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:03:38.481205    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5604c4cf-75ec-4fd0-989b-9e0316b35972-serving-certs-ca-bundle\") pod \"telemeter-client-65b6b594d5-w7prx\" (UID: \"5604c4cf-75ec-4fd0-989b-9e0316b35972\") " pod="openshift-monitoring/telemeter-client-65b6b594d5-w7prx"
May 26 16:03:38.481291 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:03:38.481230    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5604c4cf-75ec-4fd0-989b-9e0316b35972-metrics-client-ca\") pod \"telemeter-client-65b6b594d5-w7prx\" (UID: \"5604c4cf-75ec-4fd0-989b-9e0316b35972\") " pod="openshift-monitoring/telemeter-client-65b6b594d5-w7prx"
May 26 16:03:38.481291 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:03:38.481258    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/5604c4cf-75ec-4fd0-989b-9e0316b35972-secret-telemeter-client\") pod \"telemeter-client-65b6b594d5-w7prx\" (UID: \"5604c4cf-75ec-4fd0-989b-9e0316b35972\") " pod="openshift-monitoring/telemeter-client-65b6b594d5-w7prx"
May 26 16:03:38.481291 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:03:38.481276    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/5604c4cf-75ec-4fd0-989b-9e0316b35972-telemeter-client-tls\") pod \"telemeter-client-65b6b594d5-w7prx\" (UID: \"5604c4cf-75ec-4fd0-989b-9e0316b35972\") " pod="openshift-monitoring/telemeter-client-65b6b594d5-w7prx"
May 26 16:03:38.481495 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:03:38.481308    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-2znnq\" (UniqueName: \"kubernetes.io/projected/5604c4cf-75ec-4fd0-989b-9e0316b35972-kube-api-access-2znnq\") pod \"telemeter-client-65b6b594d5-w7prx\" (UID: \"5604c4cf-75ec-4fd0-989b-9e0316b35972\") " pod="openshift-monitoring/telemeter-client-65b6b594d5-w7prx"
May 26 16:03:38.481887 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:03:38.481865    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5604c4cf-75ec-4fd0-989b-9e0316b35972-serving-certs-ca-bundle\") pod \"telemeter-client-65b6b594d5-w7prx\" (UID: \"5604c4cf-75ec-4fd0-989b-9e0316b35972\") " pod="openshift-monitoring/telemeter-client-65b6b594d5-w7prx"
May 26 16:03:38.482011 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:03:38.481990    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5604c4cf-75ec-4fd0-989b-9e0316b35972-telemeter-trusted-ca-bundle\") pod \"telemeter-client-65b6b594d5-w7prx\" (UID: \"5604c4cf-75ec-4fd0-989b-9e0316b35972\") " pod="openshift-monitoring/telemeter-client-65b6b594d5-w7prx"
May 26 16:03:38.482306 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:03:38.482285    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5604c4cf-75ec-4fd0-989b-9e0316b35972-metrics-client-ca\") pod \"telemeter-client-65b6b594d5-w7prx\" (UID: \"5604c4cf-75ec-4fd0-989b-9e0316b35972\") " pod="openshift-monitoring/telemeter-client-65b6b594d5-w7prx"
May 26 16:03:38.483786 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:03:38.483760    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/5604c4cf-75ec-4fd0-989b-9e0316b35972-secret-telemeter-client\") pod \"telemeter-client-65b6b594d5-w7prx\" (UID: \"5604c4cf-75ec-4fd0-989b-9e0316b35972\") " pod="openshift-monitoring/telemeter-client-65b6b594d5-w7prx"
May 26 16:03:38.483867 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:03:38.483815    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/5604c4cf-75ec-4fd0-989b-9e0316b35972-telemeter-client-tls\") pod \"telemeter-client-65b6b594d5-w7prx\" (UID: \"5604c4cf-75ec-4fd0-989b-9e0316b35972\") " pod="openshift-monitoring/telemeter-client-65b6b594d5-w7prx"
May 26 16:03:38.483867 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:03:38.483843    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/5604c4cf-75ec-4fd0-989b-9e0316b35972-federate-client-tls\") pod \"telemeter-client-65b6b594d5-w7prx\" (UID: \"5604c4cf-75ec-4fd0-989b-9e0316b35972\") " pod="openshift-monitoring/telemeter-client-65b6b594d5-w7prx"
May 26 16:03:38.483937 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:03:38.483900    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5604c4cf-75ec-4fd0-989b-9e0316b35972-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-65b6b594d5-w7prx\" (UID: \"5604c4cf-75ec-4fd0-989b-9e0316b35972\") " pod="openshift-monitoring/telemeter-client-65b6b594d5-w7prx"
May 26 16:03:38.491547 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:03:38.491529    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2znnq\" (UniqueName: \"kubernetes.io/projected/5604c4cf-75ec-4fd0-989b-9e0316b35972-kube-api-access-2znnq\") pod \"telemeter-client-65b6b594d5-w7prx\" (UID: \"5604c4cf-75ec-4fd0-989b-9e0316b35972\") " pod="openshift-monitoring/telemeter-client-65b6b594d5-w7prx"
May 26 16:03:38.685540 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:03:38.685477    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-48m6s\""
May 26 16:03:38.693160 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:03:38.693143    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-65b6b594d5-w7prx"
May 26 16:03:38.849141 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:03:38.849106    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5604c4cf_75ec_4fd0_989b_9e0316b35972.slice/crio-e07ad6f071c8fc5be8d3ba5d715e9e88b71b70c102c70cdb01de3e0409471ee2 WatchSource:0}: Error finding container e07ad6f071c8fc5be8d3ba5d715e9e88b71b70c102c70cdb01de3e0409471ee2: Status 404 returned error can't find the container with id e07ad6f071c8fc5be8d3ba5d715e9e88b71b70c102c70cdb01de3e0409471ee2
May 26 16:03:38.849280 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:03:38.849250    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-65b6b594d5-w7prx"]
May 26 16:03:39.296222 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:03:39.296183    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-65b6b594d5-w7prx" event={"ID":"5604c4cf-75ec-4fd0-989b-9e0316b35972","Type":"ContainerStarted","Data":"e07ad6f071c8fc5be8d3ba5d715e9e88b71b70c102c70cdb01de3e0409471ee2"}
May 26 16:03:41.302832 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:03:41.302795    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-65b6b594d5-w7prx" event={"ID":"5604c4cf-75ec-4fd0-989b-9e0316b35972","Type":"ContainerStarted","Data":"93e012b39bbda8bef39897d8d81a39467224b5311665b53fb93841ad2d05c984"}
May 26 16:03:42.307151 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:03:42.307110    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-65b6b594d5-w7prx" event={"ID":"5604c4cf-75ec-4fd0-989b-9e0316b35972","Type":"ContainerStarted","Data":"fe6d8e0f022cb4d44c8e82b77c54de799842732ab6521ea731e3bb1f8bd5e301"}
May 26 16:03:42.307151 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:03:42.307148    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-65b6b594d5-w7prx" event={"ID":"5604c4cf-75ec-4fd0-989b-9e0316b35972","Type":"ContainerStarted","Data":"44ac6b0be56da158e9f31a4c544419324ac8a4f1fd312bb47baaa34c2cfd04ee"}
May 26 16:03:42.341420 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:03:42.341375    2556 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-65b6b594d5-w7prx" podStartSLOduration=1.713208781 podStartE2EDuration="4.341360963s" podCreationTimestamp="2026-05-26 16:03:38 +0000 UTC" firstStartedPulling="2026-05-26 16:03:38.851102817 +0000 UTC m=+180.695719802" lastFinishedPulling="2026-05-26 16:03:41.479255009 +0000 UTC m=+183.323871984" observedRunningTime="2026-05-26 16:03:42.340673879 +0000 UTC m=+184.185290872" watchObservedRunningTime="2026-05-26 16:03:42.341360963 +0000 UTC m=+184.185977955"
May 26 16:03:43.016171 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:03:43.016138    2556 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console/console-5cf464fbd6-bz877"]
May 26 16:03:43.019161 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:03:43.019145    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5cf464fbd6-bz877"
May 26 16:03:43.025033 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:03:43.024733    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-tf8sq\""
May 26 16:03:43.026079 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:03:43.026060    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
May 26 16:03:43.026189 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:03:43.026143    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
May 26 16:03:43.026189 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:03:43.026144    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
May 26 16:03:43.026777 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:03:43.026761    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
May 26 16:03:43.026869 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:03:43.026853    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
May 26 16:03:43.027402 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:03:43.027388    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
May 26 16:03:43.033354 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:03:43.033337    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
May 26 16:03:43.037498 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:03:43.037479    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5cf464fbd6-bz877"]
May 26 16:03:43.045489 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:03:43.045469    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
May 26 16:03:43.112274 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:03:43.112242    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d6d4c27c-74b8-4c80-ae01-7b7775cd7b27-console-oauth-config\") pod \"console-5cf464fbd6-bz877\" (UID: \"d6d4c27c-74b8-4c80-ae01-7b7775cd7b27\") " pod="openshift-console/console-5cf464fbd6-bz877"
May 26 16:03:43.112410 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:03:43.112289    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d6d4c27c-74b8-4c80-ae01-7b7775cd7b27-service-ca\") pod \"console-5cf464fbd6-bz877\" (UID: \"d6d4c27c-74b8-4c80-ae01-7b7775cd7b27\") " pod="openshift-console/console-5cf464fbd6-bz877"
May 26 16:03:43.112410 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:03:43.112331    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d6d4c27c-74b8-4c80-ae01-7b7775cd7b27-console-config\") pod \"console-5cf464fbd6-bz877\" (UID: \"d6d4c27c-74b8-4c80-ae01-7b7775cd7b27\") " pod="openshift-console/console-5cf464fbd6-bz877"
May 26 16:03:43.112410 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:03:43.112354    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjprv\" (UniqueName: \"kubernetes.io/projected/d6d4c27c-74b8-4c80-ae01-7b7775cd7b27-kube-api-access-pjprv\") pod \"console-5cf464fbd6-bz877\" (UID: \"d6d4c27c-74b8-4c80-ae01-7b7775cd7b27\") " pod="openshift-console/console-5cf464fbd6-bz877"
May 26 16:03:43.112410 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:03:43.112380    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d6d4c27c-74b8-4c80-ae01-7b7775cd7b27-console-serving-cert\") pod \"console-5cf464fbd6-bz877\" (UID: \"d6d4c27c-74b8-4c80-ae01-7b7775cd7b27\") " pod="openshift-console/console-5cf464fbd6-bz877"
May 26 16:03:43.112540 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:03:43.112434    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d6d4c27c-74b8-4c80-ae01-7b7775cd7b27-trusted-ca-bundle\") pod \"console-5cf464fbd6-bz877\" (UID: \"d6d4c27c-74b8-4c80-ae01-7b7775cd7b27\") " pod="openshift-console/console-5cf464fbd6-bz877"
May 26 16:03:43.112540 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:03:43.112470    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d6d4c27c-74b8-4c80-ae01-7b7775cd7b27-oauth-serving-cert\") pod \"console-5cf464fbd6-bz877\" (UID: \"d6d4c27c-74b8-4c80-ae01-7b7775cd7b27\") " pod="openshift-console/console-5cf464fbd6-bz877"
May 26 16:03:43.213660 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:03:43.213634    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d6d4c27c-74b8-4c80-ae01-7b7775cd7b27-service-ca\") pod \"console-5cf464fbd6-bz877\" (UID: \"d6d4c27c-74b8-4c80-ae01-7b7775cd7b27\") " pod="openshift-console/console-5cf464fbd6-bz877"
May 26 16:03:43.213806 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:03:43.213666    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d6d4c27c-74b8-4c80-ae01-7b7775cd7b27-console-config\") pod \"console-5cf464fbd6-bz877\" (UID: \"d6d4c27c-74b8-4c80-ae01-7b7775cd7b27\") " pod="openshift-console/console-5cf464fbd6-bz877"
May 26 16:03:43.213806 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:03:43.213687    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-pjprv\" (UniqueName: \"kubernetes.io/projected/d6d4c27c-74b8-4c80-ae01-7b7775cd7b27-kube-api-access-pjprv\") pod \"console-5cf464fbd6-bz877\" (UID: \"d6d4c27c-74b8-4c80-ae01-7b7775cd7b27\") " pod="openshift-console/console-5cf464fbd6-bz877"
May 26 16:03:43.213806 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:03:43.213712    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d6d4c27c-74b8-4c80-ae01-7b7775cd7b27-console-serving-cert\") pod \"console-5cf464fbd6-bz877\" (UID: \"d6d4c27c-74b8-4c80-ae01-7b7775cd7b27\") " pod="openshift-console/console-5cf464fbd6-bz877"
May 26 16:03:43.213806 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:03:43.213760    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d6d4c27c-74b8-4c80-ae01-7b7775cd7b27-trusted-ca-bundle\") pod \"console-5cf464fbd6-bz877\" (UID: \"d6d4c27c-74b8-4c80-ae01-7b7775cd7b27\") " pod="openshift-console/console-5cf464fbd6-bz877"
May 26 16:03:43.213806 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:03:43.213792    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d6d4c27c-74b8-4c80-ae01-7b7775cd7b27-oauth-serving-cert\") pod \"console-5cf464fbd6-bz877\" (UID: \"d6d4c27c-74b8-4c80-ae01-7b7775cd7b27\") " pod="openshift-console/console-5cf464fbd6-bz877"
May 26 16:03:43.214033 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:03:43.213829    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d6d4c27c-74b8-4c80-ae01-7b7775cd7b27-console-oauth-config\") pod \"console-5cf464fbd6-bz877\" (UID: \"d6d4c27c-74b8-4c80-ae01-7b7775cd7b27\") " pod="openshift-console/console-5cf464fbd6-bz877"
May 26 16:03:43.214422 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:03:43.214395    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d6d4c27c-74b8-4c80-ae01-7b7775cd7b27-service-ca\") pod \"console-5cf464fbd6-bz877\" (UID: \"d6d4c27c-74b8-4c80-ae01-7b7775cd7b27\") " pod="openshift-console/console-5cf464fbd6-bz877"
May 26 16:03:43.214545 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:03:43.214486    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d6d4c27c-74b8-4c80-ae01-7b7775cd7b27-oauth-serving-cert\") pod \"console-5cf464fbd6-bz877\" (UID: \"d6d4c27c-74b8-4c80-ae01-7b7775cd7b27\") " pod="openshift-console/console-5cf464fbd6-bz877"
May 26 16:03:43.214545 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:03:43.214500    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d6d4c27c-74b8-4c80-ae01-7b7775cd7b27-console-config\") pod \"console-5cf464fbd6-bz877\" (UID: \"d6d4c27c-74b8-4c80-ae01-7b7775cd7b27\") " pod="openshift-console/console-5cf464fbd6-bz877"
May 26 16:03:43.214861 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:03:43.214838    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d6d4c27c-74b8-4c80-ae01-7b7775cd7b27-trusted-ca-bundle\") pod \"console-5cf464fbd6-bz877\" (UID: \"d6d4c27c-74b8-4c80-ae01-7b7775cd7b27\") " pod="openshift-console/console-5cf464fbd6-bz877"
May 26 16:03:43.216312 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:03:43.216288    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d6d4c27c-74b8-4c80-ae01-7b7775cd7b27-console-oauth-config\") pod \"console-5cf464fbd6-bz877\" (UID: \"d6d4c27c-74b8-4c80-ae01-7b7775cd7b27\") " pod="openshift-console/console-5cf464fbd6-bz877"
May 26 16:03:43.216400 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:03:43.216354    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d6d4c27c-74b8-4c80-ae01-7b7775cd7b27-console-serving-cert\") pod \"console-5cf464fbd6-bz877\" (UID: \"d6d4c27c-74b8-4c80-ae01-7b7775cd7b27\") " pod="openshift-console/console-5cf464fbd6-bz877"
May 26 16:03:43.224542 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:03:43.224521    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjprv\" (UniqueName: \"kubernetes.io/projected/d6d4c27c-74b8-4c80-ae01-7b7775cd7b27-kube-api-access-pjprv\") pod \"console-5cf464fbd6-bz877\" (UID: \"d6d4c27c-74b8-4c80-ae01-7b7775cd7b27\") " pod="openshift-console/console-5cf464fbd6-bz877"
May 26 16:03:43.327559 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:03:43.327506    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5cf464fbd6-bz877"
May 26 16:03:43.459636 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:03:43.459610    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5cf464fbd6-bz877"]
May 26 16:03:43.461885 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:03:43.461854    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6d4c27c_74b8_4c80_ae01_7b7775cd7b27.slice/crio-54cac6c5ac5a40d091567338c957bbfdebf462c142a0837596f50fc3e4888ebd WatchSource:0}: Error finding container 54cac6c5ac5a40d091567338c957bbfdebf462c142a0837596f50fc3e4888ebd: Status 404 returned error can't find the container with id 54cac6c5ac5a40d091567338c957bbfdebf462c142a0837596f50fc3e4888ebd
May 26 16:03:44.313474 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:03:44.313440    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5cf464fbd6-bz877" event={"ID":"d6d4c27c-74b8-4c80-ae01-7b7775cd7b27","Type":"ContainerStarted","Data":"fabdf0b7f225877655494ba03714d0d42f57787032b578f4ed2106c9abffb667"}
May 26 16:03:44.313474 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:03:44.313478    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5cf464fbd6-bz877" event={"ID":"d6d4c27c-74b8-4c80-ae01-7b7775cd7b27","Type":"ContainerStarted","Data":"54cac6c5ac5a40d091567338c957bbfdebf462c142a0837596f50fc3e4888ebd"}
May 26 16:03:53.327869 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:03:53.327840    2556 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5cf464fbd6-bz877"
May 26 16:03:53.328425 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:03:53.327927    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5cf464fbd6-bz877"
May 26 16:03:53.333538 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:03:53.333518    2556 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5cf464fbd6-bz877"
May 26 16:03:53.339961 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:03:53.339946    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5cf464fbd6-bz877"
May 26 16:03:53.357710 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:03:53.357671    2556 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console/console-5cf464fbd6-bz877" podStartSLOduration=11.357659992 podStartE2EDuration="11.357659992s" podCreationTimestamp="2026-05-26 16:03:42 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 16:03:44.345240962 +0000 UTC m=+186.189857965" watchObservedRunningTime="2026-05-26 16:03:53.357659992 +0000 UTC m=+195.202276984"
May 26 16:03:53.602573 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:03:53.602502    2556 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console/console-7645f78b96-bfcs8"]
May 26 16:03:53.605686 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:03:53.605669    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7645f78b96-bfcs8"
May 26 16:03:53.629871 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:03:53.629840    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7645f78b96-bfcs8"]
May 26 16:03:53.688320 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:03:53.688290    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1e228518-0a79-4c9c-bda1-79a5e2bd6688-console-config\") pod \"console-7645f78b96-bfcs8\" (UID: \"1e228518-0a79-4c9c-bda1-79a5e2bd6688\") " pod="openshift-console/console-7645f78b96-bfcs8"
May 26 16:03:53.688433 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:03:53.688328    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1e228518-0a79-4c9c-bda1-79a5e2bd6688-console-oauth-config\") pod \"console-7645f78b96-bfcs8\" (UID: \"1e228518-0a79-4c9c-bda1-79a5e2bd6688\") " pod="openshift-console/console-7645f78b96-bfcs8"
May 26 16:03:53.688433 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:03:53.688367    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1e228518-0a79-4c9c-bda1-79a5e2bd6688-trusted-ca-bundle\") pod \"console-7645f78b96-bfcs8\" (UID: \"1e228518-0a79-4c9c-bda1-79a5e2bd6688\") " pod="openshift-console/console-7645f78b96-bfcs8"
May 26 16:03:53.688433 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:03:53.688383    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hqqf\" (UniqueName: \"kubernetes.io/projected/1e228518-0a79-4c9c-bda1-79a5e2bd6688-kube-api-access-7hqqf\") pod \"console-7645f78b96-bfcs8\" (UID: \"1e228518-0a79-4c9c-bda1-79a5e2bd6688\") " pod="openshift-console/console-7645f78b96-bfcs8"
May 26 16:03:53.688433 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:03:53.688406    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1e228518-0a79-4c9c-bda1-79a5e2bd6688-service-ca\") pod \"console-7645f78b96-bfcs8\" (UID: \"1e228518-0a79-4c9c-bda1-79a5e2bd6688\") " pod="openshift-console/console-7645f78b96-bfcs8"
May 26 16:03:53.688433 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:03:53.688424    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1e228518-0a79-4c9c-bda1-79a5e2bd6688-console-serving-cert\") pod \"console-7645f78b96-bfcs8\" (UID: \"1e228518-0a79-4c9c-bda1-79a5e2bd6688\") " pod="openshift-console/console-7645f78b96-bfcs8"
May 26 16:03:53.688617 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:03:53.688441    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1e228518-0a79-4c9c-bda1-79a5e2bd6688-oauth-serving-cert\") pod \"console-7645f78b96-bfcs8\" (UID: \"1e228518-0a79-4c9c-bda1-79a5e2bd6688\") " pod="openshift-console/console-7645f78b96-bfcs8"
May 26 16:03:53.789349 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:03:53.789319    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1e228518-0a79-4c9c-bda1-79a5e2bd6688-trusted-ca-bundle\") pod \"console-7645f78b96-bfcs8\" (UID: \"1e228518-0a79-4c9c-bda1-79a5e2bd6688\") " pod="openshift-console/console-7645f78b96-bfcs8"
May 26 16:03:53.789468 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:03:53.789352    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-7hqqf\" (UniqueName: \"kubernetes.io/projected/1e228518-0a79-4c9c-bda1-79a5e2bd6688-kube-api-access-7hqqf\") pod \"console-7645f78b96-bfcs8\" (UID: \"1e228518-0a79-4c9c-bda1-79a5e2bd6688\") " pod="openshift-console/console-7645f78b96-bfcs8"
May 26 16:03:53.789468 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:03:53.789386    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1e228518-0a79-4c9c-bda1-79a5e2bd6688-service-ca\") pod \"console-7645f78b96-bfcs8\" (UID: \"1e228518-0a79-4c9c-bda1-79a5e2bd6688\") " pod="openshift-console/console-7645f78b96-bfcs8"
May 26 16:03:53.789468 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:03:53.789411    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1e228518-0a79-4c9c-bda1-79a5e2bd6688-console-serving-cert\") pod \"console-7645f78b96-bfcs8\" (UID: \"1e228518-0a79-4c9c-bda1-79a5e2bd6688\") " pod="openshift-console/console-7645f78b96-bfcs8"
May 26 16:03:53.789468 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:03:53.789434    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1e228518-0a79-4c9c-bda1-79a5e2bd6688-oauth-serving-cert\") pod \"console-7645f78b96-bfcs8\" (UID: \"1e228518-0a79-4c9c-bda1-79a5e2bd6688\") " pod="openshift-console/console-7645f78b96-bfcs8"
May 26 16:03:53.789668 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:03:53.789469    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1e228518-0a79-4c9c-bda1-79a5e2bd6688-console-config\") pod \"console-7645f78b96-bfcs8\" (UID: \"1e228518-0a79-4c9c-bda1-79a5e2bd6688\") " pod="openshift-console/console-7645f78b96-bfcs8"
May 26 16:03:53.789668 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:03:53.789510    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1e228518-0a79-4c9c-bda1-79a5e2bd6688-console-oauth-config\") pod \"console-7645f78b96-bfcs8\" (UID: \"1e228518-0a79-4c9c-bda1-79a5e2bd6688\") " pod="openshift-console/console-7645f78b96-bfcs8"
May 26 16:03:53.790221 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:03:53.790197    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1e228518-0a79-4c9c-bda1-79a5e2bd6688-service-ca\") pod \"console-7645f78b96-bfcs8\" (UID: \"1e228518-0a79-4c9c-bda1-79a5e2bd6688\") " pod="openshift-console/console-7645f78b96-bfcs8"
May 26 16:03:53.790308 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:03:53.790211    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1e228518-0a79-4c9c-bda1-79a5e2bd6688-oauth-serving-cert\") pod \"console-7645f78b96-bfcs8\" (UID: \"1e228518-0a79-4c9c-bda1-79a5e2bd6688\") " pod="openshift-console/console-7645f78b96-bfcs8"
May 26 16:03:53.790308 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:03:53.790261    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1e228518-0a79-4c9c-bda1-79a5e2bd6688-console-config\") pod \"console-7645f78b96-bfcs8\" (UID: \"1e228518-0a79-4c9c-bda1-79a5e2bd6688\") " pod="openshift-console/console-7645f78b96-bfcs8"
May 26 16:03:53.790523 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:03:53.790498    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1e228518-0a79-4c9c-bda1-79a5e2bd6688-trusted-ca-bundle\") pod \"console-7645f78b96-bfcs8\" (UID: \"1e228518-0a79-4c9c-bda1-79a5e2bd6688\") " pod="openshift-console/console-7645f78b96-bfcs8"
May 26 16:03:53.791963 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:03:53.791934    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1e228518-0a79-4c9c-bda1-79a5e2bd6688-console-serving-cert\") pod \"console-7645f78b96-bfcs8\" (UID: \"1e228518-0a79-4c9c-bda1-79a5e2bd6688\") " pod="openshift-console/console-7645f78b96-bfcs8"
May 26 16:03:53.792054 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:03:53.791969    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1e228518-0a79-4c9c-bda1-79a5e2bd6688-console-oauth-config\") pod \"console-7645f78b96-bfcs8\" (UID: \"1e228518-0a79-4c9c-bda1-79a5e2bd6688\") " pod="openshift-console/console-7645f78b96-bfcs8"
May 26 16:03:53.799178 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:03:53.799160    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hqqf\" (UniqueName: \"kubernetes.io/projected/1e228518-0a79-4c9c-bda1-79a5e2bd6688-kube-api-access-7hqqf\") pod \"console-7645f78b96-bfcs8\" (UID: \"1e228518-0a79-4c9c-bda1-79a5e2bd6688\") " pod="openshift-console/console-7645f78b96-bfcs8"
May 26 16:03:53.917885 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:03:53.917857    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7645f78b96-bfcs8"
May 26 16:03:54.040533 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:03:54.040457    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7645f78b96-bfcs8"]
May 26 16:03:54.042901 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:03:54.042874    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e228518_0a79_4c9c_bda1_79a5e2bd6688.slice/crio-298dd8249d09dd3fe67f80f1ddadbb03afc157d40c4e8459f65d9cee3ab5f09b WatchSource:0}: Error finding container 298dd8249d09dd3fe67f80f1ddadbb03afc157d40c4e8459f65d9cee3ab5f09b: Status 404 returned error can't find the container with id 298dd8249d09dd3fe67f80f1ddadbb03afc157d40c4e8459f65d9cee3ab5f09b
May 26 16:03:54.340707 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:03:54.340618    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7645f78b96-bfcs8" event={"ID":"1e228518-0a79-4c9c-bda1-79a5e2bd6688","Type":"ContainerStarted","Data":"85ed881fef26e5e84248b0804992ac85675e7430ab07d97c1557276a6d9fcb42"}
May 26 16:03:54.340707 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:03:54.340663    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7645f78b96-bfcs8" event={"ID":"1e228518-0a79-4c9c-bda1-79a5e2bd6688","Type":"ContainerStarted","Data":"298dd8249d09dd3fe67f80f1ddadbb03afc157d40c4e8459f65d9cee3ab5f09b"}
May 26 16:03:54.364958 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:03:54.364917    2556 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console/console-7645f78b96-bfcs8" podStartSLOduration=1.364904293 podStartE2EDuration="1.364904293s" podCreationTimestamp="2026-05-26 16:03:53 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 16:03:54.362783933 +0000 UTC m=+196.207400926" watchObservedRunningTime="2026-05-26 16:03:54.364904293 +0000 UTC m=+196.209521286"
May 26 16:04:03.918514 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:04:03.918467    2556 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7645f78b96-bfcs8"
May 26 16:04:03.918514 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:04:03.918525    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7645f78b96-bfcs8"
May 26 16:04:03.923134 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:04:03.923113    2556 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7645f78b96-bfcs8"
May 26 16:04:04.370673 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:04:04.370649    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7645f78b96-bfcs8"
May 26 16:04:04.427524 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:04:04.427496    2556 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5cf464fbd6-bz877"]
May 26 16:04:29.449436 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:04:29.449395    2556 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-console/console-5cf464fbd6-bz877" podUID="d6d4c27c-74b8-4c80-ae01-7b7775cd7b27" containerName="console" containerID="cri-o://fabdf0b7f225877655494ba03714d0d42f57787032b578f4ed2106c9abffb667" gracePeriod=15
May 26 16:04:29.686675 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:04:29.686656    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-5cf464fbd6-bz877_d6d4c27c-74b8-4c80-ae01-7b7775cd7b27/console/0.log"
May 26 16:04:29.686771 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:04:29.686714    2556 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5cf464fbd6-bz877"
May 26 16:04:29.836305 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:04:29.836218    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pjprv\" (UniqueName: \"kubernetes.io/projected/d6d4c27c-74b8-4c80-ae01-7b7775cd7b27-kube-api-access-pjprv\") pod \"d6d4c27c-74b8-4c80-ae01-7b7775cd7b27\" (UID: \"d6d4c27c-74b8-4c80-ae01-7b7775cd7b27\") "
May 26 16:04:29.836305 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:04:29.836272    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d6d4c27c-74b8-4c80-ae01-7b7775cd7b27-oauth-serving-cert\") pod \"d6d4c27c-74b8-4c80-ae01-7b7775cd7b27\" (UID: \"d6d4c27c-74b8-4c80-ae01-7b7775cd7b27\") "
May 26 16:04:29.836505 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:04:29.836304    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d6d4c27c-74b8-4c80-ae01-7b7775cd7b27-console-config\") pod \"d6d4c27c-74b8-4c80-ae01-7b7775cd7b27\" (UID: \"d6d4c27c-74b8-4c80-ae01-7b7775cd7b27\") "
May 26 16:04:29.836505 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:04:29.836341    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d6d4c27c-74b8-4c80-ae01-7b7775cd7b27-console-oauth-config\") pod \"d6d4c27c-74b8-4c80-ae01-7b7775cd7b27\" (UID: \"d6d4c27c-74b8-4c80-ae01-7b7775cd7b27\") "
May 26 16:04:29.836505 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:04:29.836365    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d6d4c27c-74b8-4c80-ae01-7b7775cd7b27-trusted-ca-bundle\") pod \"d6d4c27c-74b8-4c80-ae01-7b7775cd7b27\" (UID: \"d6d4c27c-74b8-4c80-ae01-7b7775cd7b27\") "
May 26 16:04:29.836505 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:04:29.836387    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d6d4c27c-74b8-4c80-ae01-7b7775cd7b27-console-serving-cert\") pod \"d6d4c27c-74b8-4c80-ae01-7b7775cd7b27\" (UID: \"d6d4c27c-74b8-4c80-ae01-7b7775cd7b27\") "
May 26 16:04:29.836505 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:04:29.836438    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d6d4c27c-74b8-4c80-ae01-7b7775cd7b27-service-ca\") pod \"d6d4c27c-74b8-4c80-ae01-7b7775cd7b27\" (UID: \"d6d4c27c-74b8-4c80-ae01-7b7775cd7b27\") "
May 26 16:04:29.836870 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:04:29.836843    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6d4c27c-74b8-4c80-ae01-7b7775cd7b27-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "d6d4c27c-74b8-4c80-ae01-7b7775cd7b27" (UID: "d6d4c27c-74b8-4c80-ae01-7b7775cd7b27"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 16:04:29.837031 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:04:29.836841    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6d4c27c-74b8-4c80-ae01-7b7775cd7b27-console-config" (OuterVolumeSpecName: "console-config") pod "d6d4c27c-74b8-4c80-ae01-7b7775cd7b27" (UID: "d6d4c27c-74b8-4c80-ae01-7b7775cd7b27"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 16:04:29.837031 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:04:29.836860    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6d4c27c-74b8-4c80-ae01-7b7775cd7b27-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "d6d4c27c-74b8-4c80-ae01-7b7775cd7b27" (UID: "d6d4c27c-74b8-4c80-ae01-7b7775cd7b27"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 16:04:29.837031 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:04:29.836940    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6d4c27c-74b8-4c80-ae01-7b7775cd7b27-service-ca" (OuterVolumeSpecName: "service-ca") pod "d6d4c27c-74b8-4c80-ae01-7b7775cd7b27" (UID: "d6d4c27c-74b8-4c80-ae01-7b7775cd7b27"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 16:04:29.838495 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:04:29.838461    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6d4c27c-74b8-4c80-ae01-7b7775cd7b27-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "d6d4c27c-74b8-4c80-ae01-7b7775cd7b27" (UID: "d6d4c27c-74b8-4c80-ae01-7b7775cd7b27"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 16:04:29.838576 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:04:29.838508    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6d4c27c-74b8-4c80-ae01-7b7775cd7b27-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "d6d4c27c-74b8-4c80-ae01-7b7775cd7b27" (UID: "d6d4c27c-74b8-4c80-ae01-7b7775cd7b27"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 16:04:29.838576 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:04:29.838542    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6d4c27c-74b8-4c80-ae01-7b7775cd7b27-kube-api-access-pjprv" (OuterVolumeSpecName: "kube-api-access-pjprv") pod "d6d4c27c-74b8-4c80-ae01-7b7775cd7b27" (UID: "d6d4c27c-74b8-4c80-ae01-7b7775cd7b27"). InnerVolumeSpecName "kube-api-access-pjprv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 16:04:29.937821 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:04:29.937797    2556 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d6d4c27c-74b8-4c80-ae01-7b7775cd7b27-service-ca\") on node \"ip-10-0-134-19.ec2.internal\" DevicePath \"\""
May 26 16:04:29.937821 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:04:29.937819    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pjprv\" (UniqueName: \"kubernetes.io/projected/d6d4c27c-74b8-4c80-ae01-7b7775cd7b27-kube-api-access-pjprv\") on node \"ip-10-0-134-19.ec2.internal\" DevicePath \"\""
May 26 16:04:29.937957 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:04:29.937829    2556 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d6d4c27c-74b8-4c80-ae01-7b7775cd7b27-oauth-serving-cert\") on node \"ip-10-0-134-19.ec2.internal\" DevicePath \"\""
May 26 16:04:29.937957 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:04:29.937838    2556 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d6d4c27c-74b8-4c80-ae01-7b7775cd7b27-console-config\") on node \"ip-10-0-134-19.ec2.internal\" DevicePath \"\""
May 26 16:04:29.937957 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:04:29.937847    2556 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d6d4c27c-74b8-4c80-ae01-7b7775cd7b27-console-oauth-config\") on node \"ip-10-0-134-19.ec2.internal\" DevicePath \"\""
May 26 16:04:29.937957 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:04:29.937856    2556 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d6d4c27c-74b8-4c80-ae01-7b7775cd7b27-trusted-ca-bundle\") on node \"ip-10-0-134-19.ec2.internal\" DevicePath \"\""
May 26 16:04:29.937957 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:04:29.937865    2556 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d6d4c27c-74b8-4c80-ae01-7b7775cd7b27-console-serving-cert\") on node \"ip-10-0-134-19.ec2.internal\" DevicePath \"\""
May 26 16:04:30.437999 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:04:30.437974    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-5cf464fbd6-bz877_d6d4c27c-74b8-4c80-ae01-7b7775cd7b27/console/0.log"
May 26 16:04:30.438218 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:04:30.438012    2556 generic.go:358] "Generic (PLEG): container finished" podID="d6d4c27c-74b8-4c80-ae01-7b7775cd7b27" containerID="fabdf0b7f225877655494ba03714d0d42f57787032b578f4ed2106c9abffb667" exitCode=2
May 26 16:04:30.438218 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:04:30.438047    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5cf464fbd6-bz877" event={"ID":"d6d4c27c-74b8-4c80-ae01-7b7775cd7b27","Type":"ContainerDied","Data":"fabdf0b7f225877655494ba03714d0d42f57787032b578f4ed2106c9abffb667"}
May 26 16:04:30.438218 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:04:30.438083    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5cf464fbd6-bz877" event={"ID":"d6d4c27c-74b8-4c80-ae01-7b7775cd7b27","Type":"ContainerDied","Data":"54cac6c5ac5a40d091567338c957bbfdebf462c142a0837596f50fc3e4888ebd"}
May 26 16:04:30.438218 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:04:30.438119    2556 scope.go:117] "RemoveContainer" containerID="fabdf0b7f225877655494ba03714d0d42f57787032b578f4ed2106c9abffb667"
May 26 16:04:30.438218 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:04:30.438143    2556 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5cf464fbd6-bz877"
May 26 16:04:30.446065 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:04:30.446047    2556 scope.go:117] "RemoveContainer" containerID="fabdf0b7f225877655494ba03714d0d42f57787032b578f4ed2106c9abffb667"
May 26 16:04:30.446360 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:04:30.446340    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fabdf0b7f225877655494ba03714d0d42f57787032b578f4ed2106c9abffb667\": container with ID starting with fabdf0b7f225877655494ba03714d0d42f57787032b578f4ed2106c9abffb667 not found: ID does not exist" containerID="fabdf0b7f225877655494ba03714d0d42f57787032b578f4ed2106c9abffb667"
May 26 16:04:30.446443 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:04:30.446369    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fabdf0b7f225877655494ba03714d0d42f57787032b578f4ed2106c9abffb667"} err="failed to get container status \"fabdf0b7f225877655494ba03714d0d42f57787032b578f4ed2106c9abffb667\": rpc error: code = NotFound desc = could not find container \"fabdf0b7f225877655494ba03714d0d42f57787032b578f4ed2106c9abffb667\": container with ID starting with fabdf0b7f225877655494ba03714d0d42f57787032b578f4ed2106c9abffb667 not found: ID does not exist"
May 26 16:04:30.462741 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:04:30.462720    2556 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5cf464fbd6-bz877"]
May 26 16:04:30.467974 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:04:30.467955    2556 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5cf464fbd6-bz877"]
May 26 16:04:30.770685 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:04:30.770609    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d6d4c27c-74b8-4c80-ae01-7b7775cd7b27" path="/var/lib/kubelet/pods/d6d4c27c-74b8-4c80-ae01-7b7775cd7b27/volumes"
May 26 16:04:58.660137 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:04:58.660039    2556 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29chbqw5"]
May 26 16:04:58.660582 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:04:58.660411    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d6d4c27c-74b8-4c80-ae01-7b7775cd7b27" containerName="console"
May 26 16:04:58.660582 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:04:58.660428    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="d6d4c27c-74b8-4c80-ae01-7b7775cd7b27" containerName="console"
May 26 16:04:58.660582 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:04:58.660499    2556 memory_manager.go:356] "RemoveStaleState removing state" podUID="d6d4c27c-74b8-4c80-ae01-7b7775cd7b27" containerName="console"
May 26 16:04:58.663285 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:04:58.663271    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29chbqw5"
May 26 16:04:58.666158 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:04:58.666135    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
May 26 16:04:58.667384 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:04:58.667366    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
May 26 16:04:58.667475 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:04:58.667458    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-2v46n\""
May 26 16:04:58.673086 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:04:58.673064    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29chbqw5"]
May 26 16:04:58.829986 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:04:58.829956    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2ea159d1-0314-41db-bb23-64e357f5111e-util\") pod \"59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29chbqw5\" (UID: \"2ea159d1-0314-41db-bb23-64e357f5111e\") " pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29chbqw5"
May 26 16:04:58.830148 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:04:58.830014    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76vrm\" (UniqueName: \"kubernetes.io/projected/2ea159d1-0314-41db-bb23-64e357f5111e-kube-api-access-76vrm\") pod \"59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29chbqw5\" (UID: \"2ea159d1-0314-41db-bb23-64e357f5111e\") " pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29chbqw5"
May 26 16:04:58.830148 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:04:58.830071    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2ea159d1-0314-41db-bb23-64e357f5111e-bundle\") pod \"59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29chbqw5\" (UID: \"2ea159d1-0314-41db-bb23-64e357f5111e\") " pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29chbqw5"
May 26 16:04:58.931453 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:04:58.931373    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2ea159d1-0314-41db-bb23-64e357f5111e-util\") pod \"59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29chbqw5\" (UID: \"2ea159d1-0314-41db-bb23-64e357f5111e\") " pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29chbqw5"
May 26 16:04:58.931453 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:04:58.931441    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-76vrm\" (UniqueName: \"kubernetes.io/projected/2ea159d1-0314-41db-bb23-64e357f5111e-kube-api-access-76vrm\") pod \"59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29chbqw5\" (UID: \"2ea159d1-0314-41db-bb23-64e357f5111e\") " pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29chbqw5"
May 26 16:04:58.931628 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:04:58.931480    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2ea159d1-0314-41db-bb23-64e357f5111e-bundle\") pod \"59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29chbqw5\" (UID: \"2ea159d1-0314-41db-bb23-64e357f5111e\") " pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29chbqw5"
May 26 16:04:58.931728 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:04:58.931709    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2ea159d1-0314-41db-bb23-64e357f5111e-util\") pod \"59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29chbqw5\" (UID: \"2ea159d1-0314-41db-bb23-64e357f5111e\") " pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29chbqw5"
May 26 16:04:58.931810 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:04:58.931794    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2ea159d1-0314-41db-bb23-64e357f5111e-bundle\") pod \"59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29chbqw5\" (UID: \"2ea159d1-0314-41db-bb23-64e357f5111e\") " pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29chbqw5"
May 26 16:04:58.943383 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:04:58.943364    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-76vrm\" (UniqueName: \"kubernetes.io/projected/2ea159d1-0314-41db-bb23-64e357f5111e-kube-api-access-76vrm\") pod \"59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29chbqw5\" (UID: \"2ea159d1-0314-41db-bb23-64e357f5111e\") " pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29chbqw5"
May 26 16:04:58.971609 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:04:58.971592    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29chbqw5"
May 26 16:04:59.094583 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:04:59.094549    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29chbqw5"]
May 26 16:04:59.097311 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:04:59.097283    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ea159d1_0314_41db_bb23_64e357f5111e.slice/crio-f3ceeafb0cb5e9a83103accf5badd09a3ec142cd46c38210907f5c65f2240aee WatchSource:0}: Error finding container f3ceeafb0cb5e9a83103accf5badd09a3ec142cd46c38210907f5c65f2240aee: Status 404 returned error can't find the container with id f3ceeafb0cb5e9a83103accf5badd09a3ec142cd46c38210907f5c65f2240aee
May 26 16:04:59.511870 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:04:59.511840    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29chbqw5" event={"ID":"2ea159d1-0314-41db-bb23-64e357f5111e","Type":"ContainerStarted","Data":"f3ceeafb0cb5e9a83103accf5badd09a3ec142cd46c38210907f5c65f2240aee"}
May 26 16:05:05.529962 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:05:05.529927    2556 generic.go:358] "Generic (PLEG): container finished" podID="2ea159d1-0314-41db-bb23-64e357f5111e" containerID="a98fb5963cf6821cb76ba616a47fee6cd2e109477877dc226efb065ea6c0de2a" exitCode=0
May 26 16:05:05.530370 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:05:05.530000    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29chbqw5" event={"ID":"2ea159d1-0314-41db-bb23-64e357f5111e","Type":"ContainerDied","Data":"a98fb5963cf6821cb76ba616a47fee6cd2e109477877dc226efb065ea6c0de2a"}
May 26 16:05:08.540735 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:05:08.540697    2556 generic.go:358] "Generic (PLEG): container finished" podID="2ea159d1-0314-41db-bb23-64e357f5111e" containerID="18535beb6bdde95ad99aecac6f9919b5d41cab7208f1f693fa9aaf89f3187c9b" exitCode=0
May 26 16:05:08.541216 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:05:08.540786    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29chbqw5" event={"ID":"2ea159d1-0314-41db-bb23-64e357f5111e","Type":"ContainerDied","Data":"18535beb6bdde95ad99aecac6f9919b5d41cab7208f1f693fa9aaf89f3187c9b"}
May 26 16:05:15.562254 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:05:15.562217    2556 generic.go:358] "Generic (PLEG): container finished" podID="2ea159d1-0314-41db-bb23-64e357f5111e" containerID="77f38d058535c530ee5a312e67c7267ceceb6f922beda6f2907d4a6c7e5814ba" exitCode=0
May 26 16:05:15.562619 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:05:15.562257    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29chbqw5" event={"ID":"2ea159d1-0314-41db-bb23-64e357f5111e","Type":"ContainerDied","Data":"77f38d058535c530ee5a312e67c7267ceceb6f922beda6f2907d4a6c7e5814ba"}
May 26 16:05:16.692488 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:05:16.692466    2556 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29chbqw5"
May 26 16:05:16.859188 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:05:16.859163    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-76vrm\" (UniqueName: \"kubernetes.io/projected/2ea159d1-0314-41db-bb23-64e357f5111e-kube-api-access-76vrm\") pod \"2ea159d1-0314-41db-bb23-64e357f5111e\" (UID: \"2ea159d1-0314-41db-bb23-64e357f5111e\") "
May 26 16:05:16.859321 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:05:16.859208    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2ea159d1-0314-41db-bb23-64e357f5111e-util\") pod \"2ea159d1-0314-41db-bb23-64e357f5111e\" (UID: \"2ea159d1-0314-41db-bb23-64e357f5111e\") "
May 26 16:05:16.859321 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:05:16.859241    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2ea159d1-0314-41db-bb23-64e357f5111e-bundle\") pod \"2ea159d1-0314-41db-bb23-64e357f5111e\" (UID: \"2ea159d1-0314-41db-bb23-64e357f5111e\") "
May 26 16:05:16.859826 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:05:16.859795    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ea159d1-0314-41db-bb23-64e357f5111e-bundle" (OuterVolumeSpecName: "bundle") pod "2ea159d1-0314-41db-bb23-64e357f5111e" (UID: "2ea159d1-0314-41db-bb23-64e357f5111e"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 16:05:16.861218 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:05:16.861196    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ea159d1-0314-41db-bb23-64e357f5111e-kube-api-access-76vrm" (OuterVolumeSpecName: "kube-api-access-76vrm") pod "2ea159d1-0314-41db-bb23-64e357f5111e" (UID: "2ea159d1-0314-41db-bb23-64e357f5111e"). InnerVolumeSpecName "kube-api-access-76vrm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 16:05:16.863443 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:05:16.863421    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ea159d1-0314-41db-bb23-64e357f5111e-util" (OuterVolumeSpecName: "util") pod "2ea159d1-0314-41db-bb23-64e357f5111e" (UID: "2ea159d1-0314-41db-bb23-64e357f5111e"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 16:05:16.960568 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:05:16.960547    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76vrm\" (UniqueName: \"kubernetes.io/projected/2ea159d1-0314-41db-bb23-64e357f5111e-kube-api-access-76vrm\") on node \"ip-10-0-134-19.ec2.internal\" DevicePath \"\""
May 26 16:05:16.960568 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:05:16.960566    2556 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2ea159d1-0314-41db-bb23-64e357f5111e-util\") on node \"ip-10-0-134-19.ec2.internal\" DevicePath \"\""
May 26 16:05:16.960687 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:05:16.960576    2556 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2ea159d1-0314-41db-bb23-64e357f5111e-bundle\") on node \"ip-10-0-134-19.ec2.internal\" DevicePath \"\""
May 26 16:05:17.569509 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:05:17.569483    2556 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29chbqw5"
May 26 16:05:17.569660 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:05:17.569482    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29chbqw5" event={"ID":"2ea159d1-0314-41db-bb23-64e357f5111e","Type":"ContainerDied","Data":"f3ceeafb0cb5e9a83103accf5badd09a3ec142cd46c38210907f5c65f2240aee"}
May 26 16:05:17.569660 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:05:17.569593    2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f3ceeafb0cb5e9a83103accf5badd09a3ec142cd46c38210907f5c65f2240aee"
May 26 16:05:21.218550 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:05:21.218511    2556 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-tzntb"]
May 26 16:05:21.219025 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:05:21.218896    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ea159d1-0314-41db-bb23-64e357f5111e" containerName="util"
May 26 16:05:21.219025 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:05:21.218913    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ea159d1-0314-41db-bb23-64e357f5111e" containerName="util"
May 26 16:05:21.219025 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:05:21.218923    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ea159d1-0314-41db-bb23-64e357f5111e" containerName="pull"
May 26 16:05:21.219025 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:05:21.218930    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ea159d1-0314-41db-bb23-64e357f5111e" containerName="pull"
May 26 16:05:21.219025 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:05:21.218946    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ea159d1-0314-41db-bb23-64e357f5111e" containerName="extract"
May 26 16:05:21.219025 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:05:21.218954    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ea159d1-0314-41db-bb23-64e357f5111e" containerName="extract"
May 26 16:05:21.219025 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:05:21.219022    2556 memory_manager.go:356] "RemoveStaleState removing state" podUID="2ea159d1-0314-41db-bb23-64e357f5111e" containerName="extract"
May 26 16:05:21.223883 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:05:21.223862    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-tzntb"
May 26 16:05:21.227821 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:05:21.227786    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
May 26 16:05:21.228264 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:05:21.228235    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
May 26 16:05:21.228549 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:05:21.228453    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
May 26 16:05:21.233788 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:05:21.233751    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"custom-metrics-autoscaler-operator-dockercfg-zzv45\""
May 26 16:05:21.235292 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:05:21.235266    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-tzntb"]
May 26 16:05:21.289363 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:05:21.289340    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88mbq\" (UniqueName: \"kubernetes.io/projected/5e005ad8-e706-4ece-b9f1-8662c96060f1-kube-api-access-88mbq\") pod \"custom-metrics-autoscaler-operator-bbf89fd5d-tzntb\" (UID: \"5e005ad8-e706-4ece-b9f1-8662c96060f1\") " pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-tzntb"
May 26 16:05:21.289450 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:05:21.289387    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/5e005ad8-e706-4ece-b9f1-8662c96060f1-certificates\") pod \"custom-metrics-autoscaler-operator-bbf89fd5d-tzntb\" (UID: \"5e005ad8-e706-4ece-b9f1-8662c96060f1\") " pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-tzntb"
May 26 16:05:21.390477 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:05:21.390456    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-88mbq\" (UniqueName: \"kubernetes.io/projected/5e005ad8-e706-4ece-b9f1-8662c96060f1-kube-api-access-88mbq\") pod \"custom-metrics-autoscaler-operator-bbf89fd5d-tzntb\" (UID: \"5e005ad8-e706-4ece-b9f1-8662c96060f1\") " pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-tzntb"
May 26 16:05:21.390560 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:05:21.390497    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/5e005ad8-e706-4ece-b9f1-8662c96060f1-certificates\") pod \"custom-metrics-autoscaler-operator-bbf89fd5d-tzntb\" (UID: \"5e005ad8-e706-4ece-b9f1-8662c96060f1\") " pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-tzntb"
May 26 16:05:21.392632 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:05:21.392609    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/5e005ad8-e706-4ece-b9f1-8662c96060f1-certificates\") pod \"custom-metrics-autoscaler-operator-bbf89fd5d-tzntb\" (UID: \"5e005ad8-e706-4ece-b9f1-8662c96060f1\") " pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-tzntb"
May 26 16:05:21.401057 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:05:21.401038    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-88mbq\" (UniqueName: \"kubernetes.io/projected/5e005ad8-e706-4ece-b9f1-8662c96060f1-kube-api-access-88mbq\") pod \"custom-metrics-autoscaler-operator-bbf89fd5d-tzntb\" (UID: \"5e005ad8-e706-4ece-b9f1-8662c96060f1\") " pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-tzntb"
May 26 16:05:21.539238 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:05:21.539187    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-tzntb"
May 26 16:05:21.664500 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:05:21.664471    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-tzntb"]
May 26 16:05:21.668996 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:05:21.668962    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e005ad8_e706_4ece_b9f1_8662c96060f1.slice/crio-a6ebe803bbfc1f8771f0f68cb872320df6522cc92a904a4c8899db5a8ab87cd9 WatchSource:0}: Error finding container a6ebe803bbfc1f8771f0f68cb872320df6522cc92a904a4c8899db5a8ab87cd9: Status 404 returned error can't find the container with id a6ebe803bbfc1f8771f0f68cb872320df6522cc92a904a4c8899db5a8ab87cd9
May 26 16:05:22.584378 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:05:22.584335    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-tzntb" event={"ID":"5e005ad8-e706-4ece-b9f1-8662c96060f1","Type":"ContainerStarted","Data":"a6ebe803bbfc1f8771f0f68cb872320df6522cc92a904a4c8899db5a8ab87cd9"}
May 26 16:05:25.593473 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:05:25.593428    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-tzntb" event={"ID":"5e005ad8-e706-4ece-b9f1-8662c96060f1","Type":"ContainerStarted","Data":"7f5acdf1bc472f0ba6ee5d8a3fb721cb2b27de4c3279598c923cada424d38d55"}
May 26 16:05:25.593804 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:05:25.593501    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-tzntb"
May 26 16:05:25.617921 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:05:25.617881    2556 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-tzntb" podStartSLOduration=0.90517078 podStartE2EDuration="4.617868726s" podCreationTimestamp="2026-05-26 16:05:21 +0000 UTC" firstStartedPulling="2026-05-26 16:05:21.67118672 +0000 UTC m=+283.515803694" lastFinishedPulling="2026-05-26 16:05:25.383884667 +0000 UTC m=+287.228501640" observedRunningTime="2026-05-26 16:05:25.616467571 +0000 UTC m=+287.461084574" watchObservedRunningTime="2026-05-26 16:05:25.617868726 +0000 UTC m=+287.462485718"
May 26 16:05:25.928907 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:05:25.928868    2556 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-operator-ffbb595cb-wp9dn"]
May 26 16:05:25.931830 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:05:25.931815    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-ffbb595cb-wp9dn"
May 26 16:05:25.956148 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:05:25.956123    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-certs\""
May 26 16:05:25.956299 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:05:25.956161    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-vt5lh\""
May 26 16:05:25.957208 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:05:25.957190    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-ffbb595cb-wp9dn"]
May 26 16:05:25.984208 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:05:25.984189    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
May 26 16:05:26.025178 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:05:26.025159    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/1aa1642f-716c-4b3c-8dd6-4aec383835eb-certificates\") pod \"keda-operator-ffbb595cb-wp9dn\" (UID: \"1aa1642f-716c-4b3c-8dd6-4aec383835eb\") " pod="openshift-keda/keda-operator-ffbb595cb-wp9dn"
May 26 16:05:26.025269 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:05:26.025201    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtz7g\" (UniqueName: \"kubernetes.io/projected/1aa1642f-716c-4b3c-8dd6-4aec383835eb-kube-api-access-jtz7g\") pod \"keda-operator-ffbb595cb-wp9dn\" (UID: \"1aa1642f-716c-4b3c-8dd6-4aec383835eb\") " pod="openshift-keda/keda-operator-ffbb595cb-wp9dn"
May 26 16:05:26.025269 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:05:26.025222    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/1aa1642f-716c-4b3c-8dd6-4aec383835eb-cabundle0\") pod \"keda-operator-ffbb595cb-wp9dn\" (UID: \"1aa1642f-716c-4b3c-8dd6-4aec383835eb\") " pod="openshift-keda/keda-operator-ffbb595cb-wp9dn"
May 26 16:05:26.126177 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:05:26.126152    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-jtz7g\" (UniqueName: \"kubernetes.io/projected/1aa1642f-716c-4b3c-8dd6-4aec383835eb-kube-api-access-jtz7g\") pod \"keda-operator-ffbb595cb-wp9dn\" (UID: \"1aa1642f-716c-4b3c-8dd6-4aec383835eb\") " pod="openshift-keda/keda-operator-ffbb595cb-wp9dn"
May 26 16:05:26.126263 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:05:26.126189    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/1aa1642f-716c-4b3c-8dd6-4aec383835eb-cabundle0\") pod \"keda-operator-ffbb595cb-wp9dn\" (UID: \"1aa1642f-716c-4b3c-8dd6-4aec383835eb\") " pod="openshift-keda/keda-operator-ffbb595cb-wp9dn"
May 26 16:05:26.126263 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:05:26.126228    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/1aa1642f-716c-4b3c-8dd6-4aec383835eb-certificates\") pod \"keda-operator-ffbb595cb-wp9dn\" (UID: \"1aa1642f-716c-4b3c-8dd6-4aec383835eb\") " pod="openshift-keda/keda-operator-ffbb595cb-wp9dn"
May 26 16:05:26.126352 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:05:26.126340    2556 secret.go:281] references non-existent secret key: ca.crt
May 26 16:05:26.126385 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:05:26.126358    2556 projected.go:277] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
May 26 16:05:26.126385 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:05:26.126368    2556 projected.go:194] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-ffbb595cb-wp9dn: references non-existent secret key: ca.crt
May 26 16:05:26.126450 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:05:26.126426    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1aa1642f-716c-4b3c-8dd6-4aec383835eb-certificates podName:1aa1642f-716c-4b3c-8dd6-4aec383835eb nodeName:}" failed. No retries permitted until 2026-05-26 16:05:26.626406749 +0000 UTC m=+288.471023721 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/1aa1642f-716c-4b3c-8dd6-4aec383835eb-certificates") pod "keda-operator-ffbb595cb-wp9dn" (UID: "1aa1642f-716c-4b3c-8dd6-4aec383835eb") : references non-existent secret key: ca.crt
May 26 16:05:26.126833 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:05:26.126818    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/1aa1642f-716c-4b3c-8dd6-4aec383835eb-cabundle0\") pod \"keda-operator-ffbb595cb-wp9dn\" (UID: \"1aa1642f-716c-4b3c-8dd6-4aec383835eb\") " pod="openshift-keda/keda-operator-ffbb595cb-wp9dn"
May 26 16:05:26.135750 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:05:26.135724    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtz7g\" (UniqueName: \"kubernetes.io/projected/1aa1642f-716c-4b3c-8dd6-4aec383835eb-kube-api-access-jtz7g\") pod \"keda-operator-ffbb595cb-wp9dn\" (UID: \"1aa1642f-716c-4b3c-8dd6-4aec383835eb\") " pod="openshift-keda/keda-operator-ffbb595cb-wp9dn"
May 26 16:05:26.239482 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:05:26.239430    2556 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-metrics-apiserver-7c9f485588-krtd2"]
May 26 16:05:26.242555 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:05:26.242541    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-7c9f485588-krtd2"
May 26 16:05:26.245304 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:05:26.245287    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-metrics-apiserver-certs\""
May 26 16:05:26.252069 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:05:26.252047    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-7c9f485588-krtd2"]
May 26 16:05:26.327808 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:05:26.327784    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9jsg\" (UniqueName: \"kubernetes.io/projected/74b5b840-d957-47c6-92a6-645474079404-kube-api-access-w9jsg\") pod \"keda-metrics-apiserver-7c9f485588-krtd2\" (UID: \"74b5b840-d957-47c6-92a6-645474079404\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-krtd2"
May 26 16:05:26.327909 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:05:26.327837    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/74b5b840-d957-47c6-92a6-645474079404-temp-vol\") pod \"keda-metrics-apiserver-7c9f485588-krtd2\" (UID: \"74b5b840-d957-47c6-92a6-645474079404\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-krtd2"
May 26 16:05:26.327972 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:05:26.327951    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/74b5b840-d957-47c6-92a6-645474079404-certificates\") pod \"keda-metrics-apiserver-7c9f485588-krtd2\" (UID: \"74b5b840-d957-47c6-92a6-645474079404\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-krtd2"
May 26 16:05:26.398949 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:05:26.398924    2556 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-admission-cf49989db-6ghn5"]
May 26 16:05:26.406932 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:05:26.402563    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-cf49989db-6ghn5"
May 26 16:05:26.406932 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:05:26.406210    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-admission-webhooks-certs\""
May 26 16:05:26.411598 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:05:26.411577    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-cf49989db-6ghn5"]
May 26 16:05:26.428475 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:05:26.428456    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/74b5b840-d957-47c6-92a6-645474079404-certificates\") pod \"keda-metrics-apiserver-7c9f485588-krtd2\" (UID: \"74b5b840-d957-47c6-92a6-645474079404\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-krtd2"
May 26 16:05:26.428578 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:05:26.428497    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/c401d1a7-6017-418a-92c2-e088ea6764f8-certificates\") pod \"keda-admission-cf49989db-6ghn5\" (UID: \"c401d1a7-6017-418a-92c2-e088ea6764f8\") " pod="openshift-keda/keda-admission-cf49989db-6ghn5"
May 26 16:05:26.428578 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:05:26.428519    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-w9jsg\" (UniqueName: \"kubernetes.io/projected/74b5b840-d957-47c6-92a6-645474079404-kube-api-access-w9jsg\") pod \"keda-metrics-apiserver-7c9f485588-krtd2\" (UID: \"74b5b840-d957-47c6-92a6-645474079404\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-krtd2"
May 26 16:05:26.428578 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:05:26.428556    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfxf4\" (UniqueName: \"kubernetes.io/projected/c401d1a7-6017-418a-92c2-e088ea6764f8-kube-api-access-dfxf4\") pod \"keda-admission-cf49989db-6ghn5\" (UID: \"c401d1a7-6017-418a-92c2-e088ea6764f8\") " pod="openshift-keda/keda-admission-cf49989db-6ghn5"
May 26 16:05:26.428740 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:05:26.428585    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/74b5b840-d957-47c6-92a6-645474079404-temp-vol\") pod \"keda-metrics-apiserver-7c9f485588-krtd2\" (UID: \"74b5b840-d957-47c6-92a6-645474079404\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-krtd2"
May 26 16:05:26.428740 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:05:26.428598    2556 secret.go:281] references non-existent secret key: tls.crt
May 26 16:05:26.428740 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:05:26.428620    2556 projected.go:277] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: tls.crt
May 26 16:05:26.428740 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:05:26.428640    2556 projected.go:194] Error preparing data for projected volume certificates for pod openshift-keda/keda-metrics-apiserver-7c9f485588-krtd2: references non-existent secret key: tls.crt
May 26 16:05:26.428740 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:05:26.428707    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/74b5b840-d957-47c6-92a6-645474079404-certificates podName:74b5b840-d957-47c6-92a6-645474079404 nodeName:}" failed. No retries permitted until 2026-05-26 16:05:26.928689207 +0000 UTC m=+288.773306194 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/74b5b840-d957-47c6-92a6-645474079404-certificates") pod "keda-metrics-apiserver-7c9f485588-krtd2" (UID: "74b5b840-d957-47c6-92a6-645474079404") : references non-existent secret key: tls.crt
May 26 16:05:26.428969 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:05:26.428879    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/74b5b840-d957-47c6-92a6-645474079404-temp-vol\") pod \"keda-metrics-apiserver-7c9f485588-krtd2\" (UID: \"74b5b840-d957-47c6-92a6-645474079404\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-krtd2"
May 26 16:05:26.442668 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:05:26.442645    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9jsg\" (UniqueName: \"kubernetes.io/projected/74b5b840-d957-47c6-92a6-645474079404-kube-api-access-w9jsg\") pod \"keda-metrics-apiserver-7c9f485588-krtd2\" (UID: \"74b5b840-d957-47c6-92a6-645474079404\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-krtd2"
May 26 16:05:26.529812 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:05:26.529765    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/c401d1a7-6017-418a-92c2-e088ea6764f8-certificates\") pod \"keda-admission-cf49989db-6ghn5\" (UID: \"c401d1a7-6017-418a-92c2-e088ea6764f8\") " pod="openshift-keda/keda-admission-cf49989db-6ghn5"
May 26 16:05:26.529812 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:05:26.529798    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-dfxf4\" (UniqueName: \"kubernetes.io/projected/c401d1a7-6017-418a-92c2-e088ea6764f8-kube-api-access-dfxf4\") pod \"keda-admission-cf49989db-6ghn5\" (UID: \"c401d1a7-6017-418a-92c2-e088ea6764f8\") " pod="openshift-keda/keda-admission-cf49989db-6ghn5"
May 26 16:05:26.529922 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:05:26.529885    2556 projected.go:264] Couldn't get secret openshift-keda/keda-admission-webhooks-certs: secret "keda-admission-webhooks-certs" not found
May 26 16:05:26.529922 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:05:26.529904    2556 projected.go:194] Error preparing data for projected volume certificates for pod openshift-keda/keda-admission-cf49989db-6ghn5: secret "keda-admission-webhooks-certs" not found
May 26 16:05:26.530011 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:05:26.529956    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c401d1a7-6017-418a-92c2-e088ea6764f8-certificates podName:c401d1a7-6017-418a-92c2-e088ea6764f8 nodeName:}" failed. No retries permitted until 2026-05-26 16:05:27.029943308 +0000 UTC m=+288.874560278 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/c401d1a7-6017-418a-92c2-e088ea6764f8-certificates") pod "keda-admission-cf49989db-6ghn5" (UID: "c401d1a7-6017-418a-92c2-e088ea6764f8") : secret "keda-admission-webhooks-certs" not found
May 26 16:05:26.544231 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:05:26.544213    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfxf4\" (UniqueName: \"kubernetes.io/projected/c401d1a7-6017-418a-92c2-e088ea6764f8-kube-api-access-dfxf4\") pod \"keda-admission-cf49989db-6ghn5\" (UID: \"c401d1a7-6017-418a-92c2-e088ea6764f8\") " pod="openshift-keda/keda-admission-cf49989db-6ghn5"
May 26 16:05:26.630852 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:05:26.630832    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/1aa1642f-716c-4b3c-8dd6-4aec383835eb-certificates\") pod \"keda-operator-ffbb595cb-wp9dn\" (UID: \"1aa1642f-716c-4b3c-8dd6-4aec383835eb\") " pod="openshift-keda/keda-operator-ffbb595cb-wp9dn"
May 26 16:05:26.631149 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:05:26.630955    2556 secret.go:281] references non-existent secret key: ca.crt
May 26 16:05:26.631149 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:05:26.630974    2556 projected.go:277] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
May 26 16:05:26.631149 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:05:26.630985    2556 projected.go:194] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-ffbb595cb-wp9dn: references non-existent secret key: ca.crt
May 26 16:05:26.631149 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:05:26.631069    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1aa1642f-716c-4b3c-8dd6-4aec383835eb-certificates podName:1aa1642f-716c-4b3c-8dd6-4aec383835eb nodeName:}" failed. No retries permitted until 2026-05-26 16:05:27.631050511 +0000 UTC m=+289.475667483 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/1aa1642f-716c-4b3c-8dd6-4aec383835eb-certificates") pod "keda-operator-ffbb595cb-wp9dn" (UID: "1aa1642f-716c-4b3c-8dd6-4aec383835eb") : references non-existent secret key: ca.crt
May 26 16:05:26.932319 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:05:26.932296    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/74b5b840-d957-47c6-92a6-645474079404-certificates\") pod \"keda-metrics-apiserver-7c9f485588-krtd2\" (UID: \"74b5b840-d957-47c6-92a6-645474079404\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-krtd2"
May 26 16:05:26.932447 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:05:26.932383    2556 secret.go:281] references non-existent secret key: tls.crt
May 26 16:05:26.932447 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:05:26.932394    2556 projected.go:277] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: tls.crt
May 26 16:05:26.932447 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:05:26.932410    2556 projected.go:194] Error preparing data for projected volume certificates for pod openshift-keda/keda-metrics-apiserver-7c9f485588-krtd2: references non-existent secret key: tls.crt
May 26 16:05:26.932546 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:05:26.932451    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/74b5b840-d957-47c6-92a6-645474079404-certificates podName:74b5b840-d957-47c6-92a6-645474079404 nodeName:}" failed. No retries permitted until 2026-05-26 16:05:27.932439709 +0000 UTC m=+289.777056678 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/74b5b840-d957-47c6-92a6-645474079404-certificates") pod "keda-metrics-apiserver-7c9f485588-krtd2" (UID: "74b5b840-d957-47c6-92a6-645474079404") : references non-existent secret key: tls.crt
May 26 16:05:27.033164 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:05:27.033132    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/c401d1a7-6017-418a-92c2-e088ea6764f8-certificates\") pod \"keda-admission-cf49989db-6ghn5\" (UID: \"c401d1a7-6017-418a-92c2-e088ea6764f8\") " pod="openshift-keda/keda-admission-cf49989db-6ghn5"
May 26 16:05:27.035389 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:05:27.035367    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/c401d1a7-6017-418a-92c2-e088ea6764f8-certificates\") pod \"keda-admission-cf49989db-6ghn5\" (UID: \"c401d1a7-6017-418a-92c2-e088ea6764f8\") " pod="openshift-keda/keda-admission-cf49989db-6ghn5"
May 26 16:05:27.317418 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:05:27.317356    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-cf49989db-6ghn5"
May 26 16:05:27.482012 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:05:27.481957    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-cf49989db-6ghn5"]
May 26 16:05:27.484550 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:05:27.484513    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc401d1a7_6017_418a_92c2_e088ea6764f8.slice/crio-12d23b6c1bfd29734b7f5600d6df523119f45e682d64ad252b10544c7fb2a18d WatchSource:0}: Error finding container 12d23b6c1bfd29734b7f5600d6df523119f45e682d64ad252b10544c7fb2a18d: Status 404 returned error can't find the container with id 12d23b6c1bfd29734b7f5600d6df523119f45e682d64ad252b10544c7fb2a18d
May 26 16:05:27.600598 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:05:27.600533    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-cf49989db-6ghn5" event={"ID":"c401d1a7-6017-418a-92c2-e088ea6764f8","Type":"ContainerStarted","Data":"12d23b6c1bfd29734b7f5600d6df523119f45e682d64ad252b10544c7fb2a18d"}
May 26 16:05:27.637547 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:05:27.637525    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/1aa1642f-716c-4b3c-8dd6-4aec383835eb-certificates\") pod \"keda-operator-ffbb595cb-wp9dn\" (UID: \"1aa1642f-716c-4b3c-8dd6-4aec383835eb\") " pod="openshift-keda/keda-operator-ffbb595cb-wp9dn"
May 26 16:05:27.637811 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:05:27.637657    2556 secret.go:281] references non-existent secret key: ca.crt
May 26 16:05:27.637811 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:05:27.637675    2556 projected.go:277] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
May 26 16:05:27.637811 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:05:27.637686    2556 projected.go:194] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-ffbb595cb-wp9dn: references non-existent secret key: ca.crt
May 26 16:05:27.637811 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:05:27.637731    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1aa1642f-716c-4b3c-8dd6-4aec383835eb-certificates podName:1aa1642f-716c-4b3c-8dd6-4aec383835eb nodeName:}" failed. No retries permitted until 2026-05-26 16:05:29.637717419 +0000 UTC m=+291.482334390 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/1aa1642f-716c-4b3c-8dd6-4aec383835eb-certificates") pod "keda-operator-ffbb595cb-wp9dn" (UID: "1aa1642f-716c-4b3c-8dd6-4aec383835eb") : references non-existent secret key: ca.crt
May 26 16:05:27.939656 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:05:27.939628    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/74b5b840-d957-47c6-92a6-645474079404-certificates\") pod \"keda-metrics-apiserver-7c9f485588-krtd2\" (UID: \"74b5b840-d957-47c6-92a6-645474079404\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-krtd2"
May 26 16:05:27.939784 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:05:27.939745    2556 secret.go:281] references non-existent secret key: tls.crt
May 26 16:05:27.939784 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:05:27.939761    2556 projected.go:277] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: tls.crt
May 26 16:05:27.939784 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:05:27.939781    2556 projected.go:194] Error preparing data for projected volume certificates for pod openshift-keda/keda-metrics-apiserver-7c9f485588-krtd2: references non-existent secret key: tls.crt
May 26 16:05:27.939933 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:05:27.939841    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/74b5b840-d957-47c6-92a6-645474079404-certificates podName:74b5b840-d957-47c6-92a6-645474079404 nodeName:}" failed. No retries permitted until 2026-05-26 16:05:29.939822951 +0000 UTC m=+291.784439926 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/74b5b840-d957-47c6-92a6-645474079404-certificates") pod "keda-metrics-apiserver-7c9f485588-krtd2" (UID: "74b5b840-d957-47c6-92a6-645474079404") : references non-existent secret key: tls.crt
May 26 16:05:29.608121 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:05:29.608072    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-cf49989db-6ghn5" event={"ID":"c401d1a7-6017-418a-92c2-e088ea6764f8","Type":"ContainerStarted","Data":"a8d528a0ffbea50bbb86fed435a1a8f7a56b6ac2024a192caccc69c84b5aca16"}
May 26 16:05:29.608642 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:05:29.608621    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-admission-cf49989db-6ghn5"
May 26 16:05:29.642222 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:05:29.642183    2556 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-keda/keda-admission-cf49989db-6ghn5" podStartSLOduration=2.084333193 podStartE2EDuration="3.64217181s" podCreationTimestamp="2026-05-26 16:05:26 +0000 UTC" firstStartedPulling="2026-05-26 16:05:27.486200572 +0000 UTC m=+289.330817545" lastFinishedPulling="2026-05-26 16:05:29.044039187 +0000 UTC m=+290.888656162" observedRunningTime="2026-05-26 16:05:29.640082872 +0000 UTC m=+291.484699876" watchObservedRunningTime="2026-05-26 16:05:29.64217181 +0000 UTC m=+291.486788801"
May 26 16:05:29.652362 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:05:29.652343    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/1aa1642f-716c-4b3c-8dd6-4aec383835eb-certificates\") pod \"keda-operator-ffbb595cb-wp9dn\" (UID: \"1aa1642f-716c-4b3c-8dd6-4aec383835eb\") " pod="openshift-keda/keda-operator-ffbb595cb-wp9dn"
May 26 16:05:29.654633 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:05:29.654606    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/1aa1642f-716c-4b3c-8dd6-4aec383835eb-certificates\") pod \"keda-operator-ffbb595cb-wp9dn\" (UID: \"1aa1642f-716c-4b3c-8dd6-4aec383835eb\") " pod="openshift-keda/keda-operator-ffbb595cb-wp9dn"
May 26 16:05:29.841456 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:05:29.841434    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-ffbb595cb-wp9dn"
May 26 16:05:29.954273 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:05:29.954247    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/74b5b840-d957-47c6-92a6-645474079404-certificates\") pod \"keda-metrics-apiserver-7c9f485588-krtd2\" (UID: \"74b5b840-d957-47c6-92a6-645474079404\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-krtd2"
May 26 16:05:29.956469 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:05:29.956449    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/74b5b840-d957-47c6-92a6-645474079404-certificates\") pod \"keda-metrics-apiserver-7c9f485588-krtd2\" (UID: \"74b5b840-d957-47c6-92a6-645474079404\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-krtd2"
May 26 16:05:29.962343 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:05:29.962320    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-ffbb595cb-wp9dn"]
May 26 16:05:29.965220 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:05:29.965196    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1aa1642f_716c_4b3c_8dd6_4aec383835eb.slice/crio-617d67fecaf807aafc09654a2b574b28bb9cafe6d046339c6baa31cc59fe6303 WatchSource:0}: Error finding container 617d67fecaf807aafc09654a2b574b28bb9cafe6d046339c6baa31cc59fe6303: Status 404 returned error can't find the container with id 617d67fecaf807aafc09654a2b574b28bb9cafe6d046339c6baa31cc59fe6303
May 26 16:05:30.153312 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:05:30.153259    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-7c9f485588-krtd2"
May 26 16:05:30.278425 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:05:30.278398    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-7c9f485588-krtd2"]
May 26 16:05:30.281135 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:05:30.281103    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74b5b840_d957_47c6_92a6_645474079404.slice/crio-144b155760d0de6bd957c358a3c0370df099371c40ebc3ba8c468c6977e86983 WatchSource:0}: Error finding container 144b155760d0de6bd957c358a3c0370df099371c40ebc3ba8c468c6977e86983: Status 404 returned error can't find the container with id 144b155760d0de6bd957c358a3c0370df099371c40ebc3ba8c468c6977e86983
May 26 16:05:30.615790 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:05:30.615707    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-7c9f485588-krtd2" event={"ID":"74b5b840-d957-47c6-92a6-645474079404","Type":"ContainerStarted","Data":"144b155760d0de6bd957c358a3c0370df099371c40ebc3ba8c468c6977e86983"}
May 26 16:05:30.616698 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:05:30.616670    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-ffbb595cb-wp9dn" event={"ID":"1aa1642f-716c-4b3c-8dd6-4aec383835eb","Type":"ContainerStarted","Data":"617d67fecaf807aafc09654a2b574b28bb9cafe6d046339c6baa31cc59fe6303"}
May 26 16:05:34.631065 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:05:34.631026    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-ffbb595cb-wp9dn" event={"ID":"1aa1642f-716c-4b3c-8dd6-4aec383835eb","Type":"ContainerStarted","Data":"8f46ee757fbf683f6e3654127005ed1ff75cccb240fee7deb4471ea9edf97dee"}
May 26 16:05:34.631476 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:05:34.631139    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-operator-ffbb595cb-wp9dn"
May 26 16:05:34.632239 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:05:34.632217    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-7c9f485588-krtd2" event={"ID":"74b5b840-d957-47c6-92a6-645474079404","Type":"ContainerStarted","Data":"2d45c3d3c2c86904598ae2818705a88df8a566d89b3796e3e5269ff9b6f97c41"}
May 26 16:05:34.632370 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:05:34.632357    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-metrics-apiserver-7c9f485588-krtd2"
May 26 16:05:34.652164 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:05:34.652115    2556 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-keda/keda-operator-ffbb595cb-wp9dn" podStartSLOduration=5.646156063 podStartE2EDuration="9.652081843s" podCreationTimestamp="2026-05-26 16:05:25 +0000 UTC" firstStartedPulling="2026-05-26 16:05:29.966493623 +0000 UTC m=+291.811110594" lastFinishedPulling="2026-05-26 16:05:33.972419402 +0000 UTC m=+295.817036374" observedRunningTime="2026-05-26 16:05:34.65062465 +0000 UTC m=+296.495241643" watchObservedRunningTime="2026-05-26 16:05:34.652081843 +0000 UTC m=+296.496698836"
May 26 16:05:34.673544 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:05:34.673506    2556 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-keda/keda-metrics-apiserver-7c9f485588-krtd2" podStartSLOduration=4.988894716 podStartE2EDuration="8.673493502s" podCreationTimestamp="2026-05-26 16:05:26 +0000 UTC" firstStartedPulling="2026-05-26 16:05:30.282401234 +0000 UTC m=+292.127018210" lastFinishedPulling="2026-05-26 16:05:33.967000023 +0000 UTC m=+295.811616996" observedRunningTime="2026-05-26 16:05:34.671687502 +0000 UTC m=+296.516304495" watchObservedRunningTime="2026-05-26 16:05:34.673493502 +0000 UTC m=+296.518110495"
May 26 16:05:38.637533 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:05:38.637498    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sk7ns_4887c0d3-11fb-4287-bb94-a876a6c38803/ovn-acl-logging/0.log"
May 26 16:05:38.639903 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:05:38.639876    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sk7ns_4887c0d3-11fb-4287-bb94-a876a6c38803/ovn-acl-logging/0.log"
May 26 16:05:38.645418 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:05:38.645399    2556 kubelet.go:1628] "Image garbage collection succeeded"
May 26 16:05:45.639978 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:05:45.639946    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-metrics-apiserver-7c9f485588-krtd2"
May 26 16:05:46.598165 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:05:46.598137    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-tzntb"
May 26 16:05:51.622391 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:05:51.622355    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-admission-cf49989db-6ghn5"
May 26 16:05:55.637799 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:05:55.637762    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-operator-ffbb595cb-wp9dn"
May 26 16:06:34.470705 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:06:34.470635    2556 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-6f7545b5b9-9xj7j"]
May 26 16:06:34.472845 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:06:34.472828    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-6f7545b5b9-9xj7j"
May 26 16:06:34.475890 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:06:34.475867    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-5k26f\""
May 26 16:06:34.476003 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:06:34.475873    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
May 26 16:06:34.476003 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:06:34.475875    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\""
May 26 16:06:34.476826 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:06:34.476810    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
May 26 16:06:34.479170 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:06:34.479146    2556 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve/llmisvc-controller-manager-6b94ff949c-szqfm"]
May 26 16:06:34.481694 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:06:34.481673    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-6b94ff949c-szqfm"
May 26 16:06:34.483494 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:06:34.483473    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-6f7545b5b9-9xj7j"]
May 26 16:06:34.484449 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:06:34.484433    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-webhook-server-cert\""
May 26 16:06:34.484583 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:06:34.484453    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-controller-manager-dockercfg-6qjct\""
May 26 16:06:34.498275 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:06:34.498250    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-6b94ff949c-szqfm"]
May 26 16:06:34.505059 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:06:34.505036    2556 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-79f457656b-796rp"]
May 26 16:06:34.507004 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:06:34.506987    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-79f457656b-796rp"
May 26 16:06:34.510040 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:06:34.510023    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-5qt54\""
May 26 16:06:34.510336 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:06:34.510317    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
May 26 16:06:34.519124 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:06:34.519085    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-79f457656b-796rp"]
May 26 16:06:34.603410 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:06:34.603388    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/be62ba6f-4cd8-486d-8d5c-b20fd810d803-data\") pod \"seaweedfs-79f457656b-796rp\" (UID: \"be62ba6f-4cd8-486d-8d5c-b20fd810d803\") " pod="kserve/seaweedfs-79f457656b-796rp"
May 26 16:06:34.603547 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:06:34.603424    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cb5nw\" (UniqueName: \"kubernetes.io/projected/be62ba6f-4cd8-486d-8d5c-b20fd810d803-kube-api-access-cb5nw\") pod \"seaweedfs-79f457656b-796rp\" (UID: \"be62ba6f-4cd8-486d-8d5c-b20fd810d803\") " pod="kserve/seaweedfs-79f457656b-796rp"
May 26 16:06:34.603547 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:06:34.603456    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e628d10c-677b-48df-b75f-6efffbabe7fa-cert\") pod \"llmisvc-controller-manager-6b94ff949c-szqfm\" (UID: \"e628d10c-677b-48df-b75f-6efffbabe7fa\") " pod="kserve/llmisvc-controller-manager-6b94ff949c-szqfm"
May 26 16:06:34.603547 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:06:34.603512    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8bb67de7-b77c-48e4-8fdb-a6be94128e01-cert\") pod \"kserve-controller-manager-6f7545b5b9-9xj7j\" (UID: \"8bb67de7-b77c-48e4-8fdb-a6be94128e01\") " pod="kserve/kserve-controller-manager-6f7545b5b9-9xj7j"
May 26 16:06:34.603706 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:06:34.603580    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzc6x\" (UniqueName: \"kubernetes.io/projected/8bb67de7-b77c-48e4-8fdb-a6be94128e01-kube-api-access-vzc6x\") pod \"kserve-controller-manager-6f7545b5b9-9xj7j\" (UID: \"8bb67de7-b77c-48e4-8fdb-a6be94128e01\") " pod="kserve/kserve-controller-manager-6f7545b5b9-9xj7j"
May 26 16:06:34.603706 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:06:34.603616    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgmmx\" (UniqueName: \"kubernetes.io/projected/e628d10c-677b-48df-b75f-6efffbabe7fa-kube-api-access-sgmmx\") pod \"llmisvc-controller-manager-6b94ff949c-szqfm\" (UID: \"e628d10c-677b-48df-b75f-6efffbabe7fa\") " pod="kserve/llmisvc-controller-manager-6b94ff949c-szqfm"
May 26 16:06:34.704327 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:06:34.704306    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/be62ba6f-4cd8-486d-8d5c-b20fd810d803-data\") pod \"seaweedfs-79f457656b-796rp\" (UID: \"be62ba6f-4cd8-486d-8d5c-b20fd810d803\") " pod="kserve/seaweedfs-79f457656b-796rp"
May 26 16:06:34.704450 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:06:34.704334    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-cb5nw\" (UniqueName: \"kubernetes.io/projected/be62ba6f-4cd8-486d-8d5c-b20fd810d803-kube-api-access-cb5nw\") pod \"seaweedfs-79f457656b-796rp\" (UID: \"be62ba6f-4cd8-486d-8d5c-b20fd810d803\") " pod="kserve/seaweedfs-79f457656b-796rp"
May 26 16:06:34.704450 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:06:34.704360    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e628d10c-677b-48df-b75f-6efffbabe7fa-cert\") pod \"llmisvc-controller-manager-6b94ff949c-szqfm\" (UID: \"e628d10c-677b-48df-b75f-6efffbabe7fa\") " pod="kserve/llmisvc-controller-manager-6b94ff949c-szqfm"
May 26 16:06:34.704450 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:06:34.704377    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8bb67de7-b77c-48e4-8fdb-a6be94128e01-cert\") pod \"kserve-controller-manager-6f7545b5b9-9xj7j\" (UID: \"8bb67de7-b77c-48e4-8fdb-a6be94128e01\") " pod="kserve/kserve-controller-manager-6f7545b5b9-9xj7j"
May 26 16:06:34.704450 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:06:34.704412    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-vzc6x\" (UniqueName: \"kubernetes.io/projected/8bb67de7-b77c-48e4-8fdb-a6be94128e01-kube-api-access-vzc6x\") pod \"kserve-controller-manager-6f7545b5b9-9xj7j\" (UID: \"8bb67de7-b77c-48e4-8fdb-a6be94128e01\") " pod="kserve/kserve-controller-manager-6f7545b5b9-9xj7j"
May 26 16:06:34.704450 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:06:34.704440    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-sgmmx\" (UniqueName: \"kubernetes.io/projected/e628d10c-677b-48df-b75f-6efffbabe7fa-kube-api-access-sgmmx\") pod \"llmisvc-controller-manager-6b94ff949c-szqfm\" (UID: \"e628d10c-677b-48df-b75f-6efffbabe7fa\") " pod="kserve/llmisvc-controller-manager-6b94ff949c-szqfm"
May 26 16:06:34.704703 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:06:34.704691    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/be62ba6f-4cd8-486d-8d5c-b20fd810d803-data\") pod \"seaweedfs-79f457656b-796rp\" (UID: \"be62ba6f-4cd8-486d-8d5c-b20fd810d803\") " pod="kserve/seaweedfs-79f457656b-796rp"
May 26 16:06:34.706794 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:06:34.706769    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8bb67de7-b77c-48e4-8fdb-a6be94128e01-cert\") pod \"kserve-controller-manager-6f7545b5b9-9xj7j\" (UID: \"8bb67de7-b77c-48e4-8fdb-a6be94128e01\") " pod="kserve/kserve-controller-manager-6f7545b5b9-9xj7j"
May 26 16:06:34.706874 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:06:34.706813    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e628d10c-677b-48df-b75f-6efffbabe7fa-cert\") pod \"llmisvc-controller-manager-6b94ff949c-szqfm\" (UID: \"e628d10c-677b-48df-b75f-6efffbabe7fa\") " pod="kserve/llmisvc-controller-manager-6b94ff949c-szqfm"
May 26 16:06:34.715351 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:06:34.715333    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgmmx\" (UniqueName: \"kubernetes.io/projected/e628d10c-677b-48df-b75f-6efffbabe7fa-kube-api-access-sgmmx\") pod \"llmisvc-controller-manager-6b94ff949c-szqfm\" (UID: \"e628d10c-677b-48df-b75f-6efffbabe7fa\") " pod="kserve/llmisvc-controller-manager-6b94ff949c-szqfm"
May 26 16:06:34.715440 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:06:34.715400    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzc6x\" (UniqueName: \"kubernetes.io/projected/8bb67de7-b77c-48e4-8fdb-a6be94128e01-kube-api-access-vzc6x\") pod \"kserve-controller-manager-6f7545b5b9-9xj7j\" (UID: \"8bb67de7-b77c-48e4-8fdb-a6be94128e01\") " pod="kserve/kserve-controller-manager-6f7545b5b9-9xj7j"
May 26 16:06:34.715440 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:06:34.715428    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cb5nw\" (UniqueName: \"kubernetes.io/projected/be62ba6f-4cd8-486d-8d5c-b20fd810d803-kube-api-access-cb5nw\") pod \"seaweedfs-79f457656b-796rp\" (UID: \"be62ba6f-4cd8-486d-8d5c-b20fd810d803\") " pod="kserve/seaweedfs-79f457656b-796rp"
May 26 16:06:34.785778 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:06:34.785724    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-6f7545b5b9-9xj7j"
May 26 16:06:34.802391 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:06:34.802374    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-6b94ff949c-szqfm"
May 26 16:06:34.815575 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:06:34.815554    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-79f457656b-796rp"
May 26 16:06:34.936771 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:06:34.936741    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-6f7545b5b9-9xj7j"]
May 26 16:06:34.940267 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:06:34.940241    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bb67de7_b77c_48e4_8fdb_a6be94128e01.slice/crio-e6e06bd7adbef785a96deecc02e1de9a116247640f7aff35f03140e3fbc76928 WatchSource:0}: Error finding container e6e06bd7adbef785a96deecc02e1de9a116247640f7aff35f03140e3fbc76928: Status 404 returned error can't find the container with id e6e06bd7adbef785a96deecc02e1de9a116247640f7aff35f03140e3fbc76928
May 26 16:06:34.941607 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:06:34.941590    2556 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 26 16:06:34.957231 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:06:34.957188    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-6b94ff949c-szqfm"]
May 26 16:06:34.961351 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:06:34.961318    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pode628d10c_677b_48df_b75f_6efffbabe7fa.slice/crio-90bf7f1b6c11c222d34d7f8824e80ef7fdd42dbdcc1f81739429e298fe52ec76 WatchSource:0}: Error finding container 90bf7f1b6c11c222d34d7f8824e80ef7fdd42dbdcc1f81739429e298fe52ec76: Status 404 returned error can't find the container with id 90bf7f1b6c11c222d34d7f8824e80ef7fdd42dbdcc1f81739429e298fe52ec76
May 26 16:06:34.976247 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:06:34.976220    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-79f457656b-796rp"]
May 26 16:06:34.978693 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:06:34.978663    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe62ba6f_4cd8_486d_8d5c_b20fd810d803.slice/crio-583dbc166adfddbcb4d00f78548cadd2ec53df2a6ad68901bde16dc4419985d2 WatchSource:0}: Error finding container 583dbc166adfddbcb4d00f78548cadd2ec53df2a6ad68901bde16dc4419985d2: Status 404 returned error can't find the container with id 583dbc166adfddbcb4d00f78548cadd2ec53df2a6ad68901bde16dc4419985d2
May 26 16:06:35.804825 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:06:35.804765    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-79f457656b-796rp" event={"ID":"be62ba6f-4cd8-486d-8d5c-b20fd810d803","Type":"ContainerStarted","Data":"583dbc166adfddbcb4d00f78548cadd2ec53df2a6ad68901bde16dc4419985d2"}
May 26 16:06:35.806423 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:06:35.806393    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-6f7545b5b9-9xj7j" event={"ID":"8bb67de7-b77c-48e4-8fdb-a6be94128e01","Type":"ContainerStarted","Data":"e6e06bd7adbef785a96deecc02e1de9a116247640f7aff35f03140e3fbc76928"}
May 26 16:06:35.807605 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:06:35.807581    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-6b94ff949c-szqfm" event={"ID":"e628d10c-677b-48df-b75f-6efffbabe7fa","Type":"ContainerStarted","Data":"90bf7f1b6c11c222d34d7f8824e80ef7fdd42dbdcc1f81739429e298fe52ec76"}
May 26 16:06:39.672237 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:06:39.672217    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
May 26 16:06:39.823491 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:06:39.823456    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-6f7545b5b9-9xj7j" event={"ID":"8bb67de7-b77c-48e4-8fdb-a6be94128e01","Type":"ContainerStarted","Data":"a2ef09a90dbc2094a2645381fa1ae15aac554dff4429174d2f68aae63e9f4f9d"}
May 26 16:06:39.823664 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:06:39.823583    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-6f7545b5b9-9xj7j"
May 26 16:06:39.824768 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:06:39.824740    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-6b94ff949c-szqfm" event={"ID":"e628d10c-677b-48df-b75f-6efffbabe7fa","Type":"ContainerStarted","Data":"9734d75b4207068a0882a2884589f8c10caf64a2f72271b67ce89105afc577e9"}
May 26 16:06:39.824875 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:06:39.824796    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-6b94ff949c-szqfm"
May 26 16:06:39.825899 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:06:39.825877    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-79f457656b-796rp" event={"ID":"be62ba6f-4cd8-486d-8d5c-b20fd810d803","Type":"ContainerStarted","Data":"09ec40e6137a952c221556b92eb9665aaba7e366926c8041412339db110c186d"}
May 26 16:06:39.826007 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:06:39.825996    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-79f457656b-796rp"
May 26 16:06:39.887536 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:06:39.887487    2556 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve/kserve-controller-manager-6f7545b5b9-9xj7j" podStartSLOduration=1.338319932 podStartE2EDuration="5.887472039s" podCreationTimestamp="2026-05-26 16:06:34 +0000 UTC" firstStartedPulling="2026-05-26 16:06:34.941716264 +0000 UTC m=+356.786333233" lastFinishedPulling="2026-05-26 16:06:39.49086837 +0000 UTC m=+361.335485340" observedRunningTime="2026-05-26 16:06:39.857169737 +0000 UTC m=+361.701786728" watchObservedRunningTime="2026-05-26 16:06:39.887472039 +0000 UTC m=+361.732089042"
May 26 16:06:39.887679 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:06:39.887563    2556 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve/seaweedfs-79f457656b-796rp" podStartSLOduration=1.198416985 podStartE2EDuration="5.887558411s" podCreationTimestamp="2026-05-26 16:06:34 +0000 UTC" firstStartedPulling="2026-05-26 16:06:34.97991999 +0000 UTC m=+356.824536961" lastFinishedPulling="2026-05-26 16:06:39.669061409 +0000 UTC m=+361.513678387" observedRunningTime="2026-05-26 16:06:39.885787627 +0000 UTC m=+361.730404632" watchObservedRunningTime="2026-05-26 16:06:39.887558411 +0000 UTC m=+361.732175403"
May 26 16:06:39.909418 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:06:39.909339    2556 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve/llmisvc-controller-manager-6b94ff949c-szqfm" podStartSLOduration=1.23581593 podStartE2EDuration="5.909328705s" podCreationTimestamp="2026-05-26 16:06:34 +0000 UTC" firstStartedPulling="2026-05-26 16:06:34.962696302 +0000 UTC m=+356.807313276" lastFinishedPulling="2026-05-26 16:06:39.636209078 +0000 UTC m=+361.480826051" observedRunningTime="2026-05-26 16:06:39.907624616 +0000 UTC m=+361.752241607" watchObservedRunningTime="2026-05-26 16:06:39.909328705 +0000 UTC m=+361.753945697"
May 26 16:06:45.831148 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:06:45.831116    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-79f457656b-796rp"
May 26 16:07:10.831075 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:07:10.831041    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-6b94ff949c-szqfm"
May 26 16:07:10.834064 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:07:10.834043    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-6f7545b5b9-9xj7j"
May 26 16:07:12.507742 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:07:12.507706    2556 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-6f7545b5b9-9xj7j"]
May 26 16:07:12.508169 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:07:12.507913    2556 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve/kserve-controller-manager-6f7545b5b9-9xj7j" podUID="8bb67de7-b77c-48e4-8fdb-a6be94128e01" containerName="manager" containerID="cri-o://a2ef09a90dbc2094a2645381fa1ae15aac554dff4429174d2f68aae63e9f4f9d" gracePeriod=10
May 26 16:07:12.760789 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:07:12.760732    2556 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-6f7545b5b9-9xj7j"
May 26 16:07:12.772341 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:07:12.772319    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8bb67de7-b77c-48e4-8fdb-a6be94128e01-cert\") pod \"8bb67de7-b77c-48e4-8fdb-a6be94128e01\" (UID: \"8bb67de7-b77c-48e4-8fdb-a6be94128e01\") "
May 26 16:07:12.772449 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:07:12.772372    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vzc6x\" (UniqueName: \"kubernetes.io/projected/8bb67de7-b77c-48e4-8fdb-a6be94128e01-kube-api-access-vzc6x\") pod \"8bb67de7-b77c-48e4-8fdb-a6be94128e01\" (UID: \"8bb67de7-b77c-48e4-8fdb-a6be94128e01\") "
May 26 16:07:12.774396 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:07:12.774374    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8bb67de7-b77c-48e4-8fdb-a6be94128e01-cert" (OuterVolumeSpecName: "cert") pod "8bb67de7-b77c-48e4-8fdb-a6be94128e01" (UID: "8bb67de7-b77c-48e4-8fdb-a6be94128e01"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 16:07:12.774396 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:07:12.774381    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bb67de7-b77c-48e4-8fdb-a6be94128e01-kube-api-access-vzc6x" (OuterVolumeSpecName: "kube-api-access-vzc6x") pod "8bb67de7-b77c-48e4-8fdb-a6be94128e01" (UID: "8bb67de7-b77c-48e4-8fdb-a6be94128e01"). InnerVolumeSpecName "kube-api-access-vzc6x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 16:07:12.873272 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:07:12.873238    2556 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8bb67de7-b77c-48e4-8fdb-a6be94128e01-cert\") on node \"ip-10-0-134-19.ec2.internal\" DevicePath \"\""
May 26 16:07:12.873272 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:07:12.873264    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vzc6x\" (UniqueName: \"kubernetes.io/projected/8bb67de7-b77c-48e4-8fdb-a6be94128e01-kube-api-access-vzc6x\") on node \"ip-10-0-134-19.ec2.internal\" DevicePath \"\""
May 26 16:07:12.918803 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:07:12.918770    2556 generic.go:358] "Generic (PLEG): container finished" podID="8bb67de7-b77c-48e4-8fdb-a6be94128e01" containerID="a2ef09a90dbc2094a2645381fa1ae15aac554dff4429174d2f68aae63e9f4f9d" exitCode=0
May 26 16:07:12.918940 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:07:12.918842    2556 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-6f7545b5b9-9xj7j"
May 26 16:07:12.918940 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:07:12.918841    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-6f7545b5b9-9xj7j" event={"ID":"8bb67de7-b77c-48e4-8fdb-a6be94128e01","Type":"ContainerDied","Data":"a2ef09a90dbc2094a2645381fa1ae15aac554dff4429174d2f68aae63e9f4f9d"}
May 26 16:07:12.919047 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:07:12.918958    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-6f7545b5b9-9xj7j" event={"ID":"8bb67de7-b77c-48e4-8fdb-a6be94128e01","Type":"ContainerDied","Data":"e6e06bd7adbef785a96deecc02e1de9a116247640f7aff35f03140e3fbc76928"}
May 26 16:07:12.919047 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:07:12.918974    2556 scope.go:117] "RemoveContainer" containerID="a2ef09a90dbc2094a2645381fa1ae15aac554dff4429174d2f68aae63e9f4f9d"
May 26 16:07:12.927490 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:07:12.927476    2556 scope.go:117] "RemoveContainer" containerID="a2ef09a90dbc2094a2645381fa1ae15aac554dff4429174d2f68aae63e9f4f9d"
May 26 16:07:12.927717 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:07:12.927697    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2ef09a90dbc2094a2645381fa1ae15aac554dff4429174d2f68aae63e9f4f9d\": container with ID starting with a2ef09a90dbc2094a2645381fa1ae15aac554dff4429174d2f68aae63e9f4f9d not found: ID does not exist" containerID="a2ef09a90dbc2094a2645381fa1ae15aac554dff4429174d2f68aae63e9f4f9d"
May 26 16:07:12.927777 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:07:12.927729    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2ef09a90dbc2094a2645381fa1ae15aac554dff4429174d2f68aae63e9f4f9d"} err="failed to get container status \"a2ef09a90dbc2094a2645381fa1ae15aac554dff4429174d2f68aae63e9f4f9d\": rpc error: code = NotFound desc = could not find container \"a2ef09a90dbc2094a2645381fa1ae15aac554dff4429174d2f68aae63e9f4f9d\": container with ID starting with a2ef09a90dbc2094a2645381fa1ae15aac554dff4429174d2f68aae63e9f4f9d not found: ID does not exist"
May 26 16:07:12.950408 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:07:12.950384    2556 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-6f7545b5b9-9xj7j"]
May 26 16:07:12.951567 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:07:12.951547    2556 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve/kserve-controller-manager-6f7545b5b9-9xj7j"]
May 26 16:07:14.770526 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:07:14.770492    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8bb67de7-b77c-48e4-8fdb-a6be94128e01" path="/var/lib/kubelet/pods/8bb67de7-b77c-48e4-8fdb-a6be94128e01/volumes"
May 26 16:07:48.380062 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:07:48.380025    2556 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-86f7b4b499-rkhfm"]
May 26 16:07:48.380648 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:07:48.380332    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8bb67de7-b77c-48e4-8fdb-a6be94128e01" containerName="manager"
May 26 16:07:48.380648 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:07:48.380344    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="8bb67de7-b77c-48e4-8fdb-a6be94128e01" containerName="manager"
May 26 16:07:48.380648 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:07:48.380404    2556 memory_manager.go:356] "RemoveStaleState removing state" podUID="8bb67de7-b77c-48e4-8fdb-a6be94128e01" containerName="manager"
May 26 16:07:48.382381 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:07:48.382352    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-86f7b4b499-rkhfm"
May 26 16:07:48.386186 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:07:48.386154    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-s4h9s\""
May 26 16:07:48.386186 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:07:48.386164    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
May 26 16:07:48.387714 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:07:48.387694    2556 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-696fc77849-fhsd8"]
May 26 16:07:48.389991 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:07:48.389971    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-696fc77849-fhsd8"
May 26 16:07:48.392631 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:07:48.392607    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
May 26 16:07:48.392923 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:07:48.392903    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-ts647\""
May 26 16:07:48.393919 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:07:48.393901    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-86f7b4b499-rkhfm"]
May 26 16:07:48.401943 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:07:48.401919    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-696fc77849-fhsd8"]
May 26 16:07:48.414771 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:07:48.414752    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jzzq\" (UniqueName: \"kubernetes.io/projected/053a18f4-17b7-4dd4-9108-0e418f05b305-kube-api-access-6jzzq\") pod \"odh-model-controller-696fc77849-fhsd8\" (UID: \"053a18f4-17b7-4dd4-9108-0e418f05b305\") " pod="kserve/odh-model-controller-696fc77849-fhsd8"
May 26 16:07:48.414864 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:07:48.414778    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/053a18f4-17b7-4dd4-9108-0e418f05b305-cert\") pod \"odh-model-controller-696fc77849-fhsd8\" (UID: \"053a18f4-17b7-4dd4-9108-0e418f05b305\") " pod="kserve/odh-model-controller-696fc77849-fhsd8"
May 26 16:07:48.414864 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:07:48.414803    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwkfk\" (UniqueName: \"kubernetes.io/projected/a2c73ac5-470d-4e56-9b23-03e3bdf4dfa8-kube-api-access-mwkfk\") pod \"model-serving-api-86f7b4b499-rkhfm\" (UID: \"a2c73ac5-470d-4e56-9b23-03e3bdf4dfa8\") " pod="kserve/model-serving-api-86f7b4b499-rkhfm"
May 26 16:07:48.414864 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:07:48.414861    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2c73ac5-470d-4e56-9b23-03e3bdf4dfa8-tls-certs\") pod \"model-serving-api-86f7b4b499-rkhfm\" (UID: \"a2c73ac5-470d-4e56-9b23-03e3bdf4dfa8\") " pod="kserve/model-serving-api-86f7b4b499-rkhfm"
May 26 16:07:48.515814 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:07:48.515790    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-6jzzq\" (UniqueName: \"kubernetes.io/projected/053a18f4-17b7-4dd4-9108-0e418f05b305-kube-api-access-6jzzq\") pod \"odh-model-controller-696fc77849-fhsd8\" (UID: \"053a18f4-17b7-4dd4-9108-0e418f05b305\") " pod="kserve/odh-model-controller-696fc77849-fhsd8"
May 26 16:07:48.515957 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:07:48.515825    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/053a18f4-17b7-4dd4-9108-0e418f05b305-cert\") pod \"odh-model-controller-696fc77849-fhsd8\" (UID: \"053a18f4-17b7-4dd4-9108-0e418f05b305\") " pod="kserve/odh-model-controller-696fc77849-fhsd8"
May 26 16:07:48.515957 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:07:48.515861    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-mwkfk\" (UniqueName: \"kubernetes.io/projected/a2c73ac5-470d-4e56-9b23-03e3bdf4dfa8-kube-api-access-mwkfk\") pod \"model-serving-api-86f7b4b499-rkhfm\" (UID: \"a2c73ac5-470d-4e56-9b23-03e3bdf4dfa8\") " pod="kserve/model-serving-api-86f7b4b499-rkhfm"
May 26 16:07:48.515957 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:07:48.515900    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2c73ac5-470d-4e56-9b23-03e3bdf4dfa8-tls-certs\") pod \"model-serving-api-86f7b4b499-rkhfm\" (UID: \"a2c73ac5-470d-4e56-9b23-03e3bdf4dfa8\") " pod="kserve/model-serving-api-86f7b4b499-rkhfm"
May 26 16:07:48.518154 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:07:48.518132    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2c73ac5-470d-4e56-9b23-03e3bdf4dfa8-tls-certs\") pod \"model-serving-api-86f7b4b499-rkhfm\" (UID: \"a2c73ac5-470d-4e56-9b23-03e3bdf4dfa8\") " pod="kserve/model-serving-api-86f7b4b499-rkhfm"
May 26 16:07:48.518250 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:07:48.518171    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/053a18f4-17b7-4dd4-9108-0e418f05b305-cert\") pod \"odh-model-controller-696fc77849-fhsd8\" (UID: \"053a18f4-17b7-4dd4-9108-0e418f05b305\") " pod="kserve/odh-model-controller-696fc77849-fhsd8"
May 26 16:07:48.526472 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:07:48.526449    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jzzq\" (UniqueName: \"kubernetes.io/projected/053a18f4-17b7-4dd4-9108-0e418f05b305-kube-api-access-6jzzq\") pod \"odh-model-controller-696fc77849-fhsd8\" (UID: \"053a18f4-17b7-4dd4-9108-0e418f05b305\") " pod="kserve/odh-model-controller-696fc77849-fhsd8"
May 26 16:07:48.526572 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:07:48.526554    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwkfk\" (UniqueName: \"kubernetes.io/projected/a2c73ac5-470d-4e56-9b23-03e3bdf4dfa8-kube-api-access-mwkfk\") pod \"model-serving-api-86f7b4b499-rkhfm\" (UID: \"a2c73ac5-470d-4e56-9b23-03e3bdf4dfa8\") " pod="kserve/model-serving-api-86f7b4b499-rkhfm"
May 26 16:07:48.694038 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:07:48.693984    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-86f7b4b499-rkhfm"
May 26 16:07:48.703821 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:07:48.703786    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-696fc77849-fhsd8"
May 26 16:07:48.832418 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:07:48.832393    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-86f7b4b499-rkhfm"]
May 26 16:07:48.835267 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:07:48.835240    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2c73ac5_470d_4e56_9b23_03e3bdf4dfa8.slice/crio-495914ef2bdc5c89d76793bc21385c571b295b0f56fcf8c18cc676107714c803 WatchSource:0}: Error finding container 495914ef2bdc5c89d76793bc21385c571b295b0f56fcf8c18cc676107714c803: Status 404 returned error can't find the container with id 495914ef2bdc5c89d76793bc21385c571b295b0f56fcf8c18cc676107714c803
May 26 16:07:48.847869 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:07:48.847849    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-696fc77849-fhsd8"]
May 26 16:07:48.849817 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:07:48.849792    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod053a18f4_17b7_4dd4_9108_0e418f05b305.slice/crio-3663bbb974c62ceb4a81f8b248f26388f70d8a486f1211836eb3c76ea3c40ba2 WatchSource:0}: Error finding container 3663bbb974c62ceb4a81f8b248f26388f70d8a486f1211836eb3c76ea3c40ba2: Status 404 returned error can't find the container with id 3663bbb974c62ceb4a81f8b248f26388f70d8a486f1211836eb3c76ea3c40ba2
May 26 16:07:49.026697 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:07:49.026626    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-86f7b4b499-rkhfm" event={"ID":"a2c73ac5-470d-4e56-9b23-03e3bdf4dfa8","Type":"ContainerStarted","Data":"495914ef2bdc5c89d76793bc21385c571b295b0f56fcf8c18cc676107714c803"}
May 26 16:07:49.027546 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:07:49.027516    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-696fc77849-fhsd8" event={"ID":"053a18f4-17b7-4dd4-9108-0e418f05b305","Type":"ContainerStarted","Data":"3663bbb974c62ceb4a81f8b248f26388f70d8a486f1211836eb3c76ea3c40ba2"}
May 26 16:07:53.044327 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:07:53.044260    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-696fc77849-fhsd8" event={"ID":"053a18f4-17b7-4dd4-9108-0e418f05b305","Type":"ContainerStarted","Data":"068bda9c368825beb257680f37d0bc483dbbf0db139e2df7253b6834a007a5c6"}
May 26 16:07:53.044770 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:07:53.044372    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-696fc77849-fhsd8"
May 26 16:07:53.045611 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:07:53.045592    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-86f7b4b499-rkhfm" event={"ID":"a2c73ac5-470d-4e56-9b23-03e3bdf4dfa8","Type":"ContainerStarted","Data":"94171e76005d65be71ba8d9123f1972c5f4077c05c70802e811199733a4b8627"}
May 26 16:07:53.045730 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:07:53.045720    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-86f7b4b499-rkhfm"
May 26 16:07:53.062554 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:07:53.062519    2556 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve/odh-model-controller-696fc77849-fhsd8" podStartSLOduration=1.46965807 podStartE2EDuration="5.062509923s" podCreationTimestamp="2026-05-26 16:07:48 +0000 UTC" firstStartedPulling="2026-05-26 16:07:48.851018503 +0000 UTC m=+430.695635476" lastFinishedPulling="2026-05-26 16:07:52.443870346 +0000 UTC m=+434.288487329" observedRunningTime="2026-05-26 16:07:53.062340027 +0000 UTC m=+434.906957016" watchObservedRunningTime="2026-05-26 16:07:53.062509923 +0000 UTC m=+434.907126914"
May 26 16:07:53.081758 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:07:53.081722    2556 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve/model-serving-api-86f7b4b499-rkhfm" podStartSLOduration=1.475443824 podStartE2EDuration="5.081710767s" podCreationTimestamp="2026-05-26 16:07:48 +0000 UTC" firstStartedPulling="2026-05-26 16:07:48.837118339 +0000 UTC m=+430.681735316" lastFinishedPulling="2026-05-26 16:07:52.44338529 +0000 UTC m=+434.288002259" observedRunningTime="2026-05-26 16:07:53.079410665 +0000 UTC m=+434.924027667" watchObservedRunningTime="2026-05-26 16:07:53.081710767 +0000 UTC m=+434.926327758"
May 26 16:08:04.051261 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:08:04.051234    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-696fc77849-fhsd8"
May 26 16:08:04.052930 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:08:04.052908    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-86f7b4b499-rkhfm"
May 26 16:08:24.945107 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:08:24.945063    2556 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-91b4a-predictor-78f8d8cdc7-bv7nf"]
May 26 16:08:24.952879 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:08:24.952855    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-91b4a-predictor-78f8d8cdc7-bv7nf"
May 26 16:08:24.955975 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:08:24.955950    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-raw-sklearn-batcher-91b4a-predictor-serving-cert\""
May 26 16:08:24.956112 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:08:24.956018    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
May 26 16:08:24.956112 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:08:24.956026    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-llsfr\""
May 26 16:08:24.956112 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:08:24.956029    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-raw-sklearn-batcher-91b4a-kube-rbac-proxy-sar-config\""
May 26 16:08:24.957370 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:08:24.957347    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
May 26 16:08:24.959436 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:08:24.959415    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-91b4a-predictor-78f8d8cdc7-bv7nf"]
May 26 16:08:25.108600 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:08:25.108563    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/151ad032-daaf-4774-b54b-c4f98d55891d-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-91b4a-predictor-78f8d8cdc7-bv7nf\" (UID: \"151ad032-daaf-4774-b54b-c4f98d55891d\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-91b4a-predictor-78f8d8cdc7-bv7nf"
May 26 16:08:25.108783 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:08:25.108629    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/151ad032-daaf-4774-b54b-c4f98d55891d-proxy-tls\") pod \"isvc-raw-sklearn-batcher-91b4a-predictor-78f8d8cdc7-bv7nf\" (UID: \"151ad032-daaf-4774-b54b-c4f98d55891d\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-91b4a-predictor-78f8d8cdc7-bv7nf"
May 26 16:08:25.108783 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:08:25.108653    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfht8\" (UniqueName: \"kubernetes.io/projected/151ad032-daaf-4774-b54b-c4f98d55891d-kube-api-access-kfht8\") pod \"isvc-raw-sklearn-batcher-91b4a-predictor-78f8d8cdc7-bv7nf\" (UID: \"151ad032-daaf-4774-b54b-c4f98d55891d\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-91b4a-predictor-78f8d8cdc7-bv7nf"
May 26 16:08:25.108783 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:08:25.108693    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-raw-sklearn-batcher-91b4a-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/151ad032-daaf-4774-b54b-c4f98d55891d-isvc-raw-sklearn-batcher-91b4a-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-91b4a-predictor-78f8d8cdc7-bv7nf\" (UID: \"151ad032-daaf-4774-b54b-c4f98d55891d\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-91b4a-predictor-78f8d8cdc7-bv7nf"
May 26 16:08:25.209588 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:08:25.209497    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-raw-sklearn-batcher-91b4a-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/151ad032-daaf-4774-b54b-c4f98d55891d-isvc-raw-sklearn-batcher-91b4a-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-91b4a-predictor-78f8d8cdc7-bv7nf\" (UID: \"151ad032-daaf-4774-b54b-c4f98d55891d\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-91b4a-predictor-78f8d8cdc7-bv7nf"
May 26 16:08:25.209588 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:08:25.209568    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/151ad032-daaf-4774-b54b-c4f98d55891d-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-91b4a-predictor-78f8d8cdc7-bv7nf\" (UID: \"151ad032-daaf-4774-b54b-c4f98d55891d\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-91b4a-predictor-78f8d8cdc7-bv7nf"
May 26 16:08:25.209827 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:08:25.209624    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/151ad032-daaf-4774-b54b-c4f98d55891d-proxy-tls\") pod \"isvc-raw-sklearn-batcher-91b4a-predictor-78f8d8cdc7-bv7nf\" (UID: \"151ad032-daaf-4774-b54b-c4f98d55891d\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-91b4a-predictor-78f8d8cdc7-bv7nf"
May 26 16:08:25.209827 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:08:25.209660    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-kfht8\" (UniqueName: \"kubernetes.io/projected/151ad032-daaf-4774-b54b-c4f98d55891d-kube-api-access-kfht8\") pod \"isvc-raw-sklearn-batcher-91b4a-predictor-78f8d8cdc7-bv7nf\" (UID: \"151ad032-daaf-4774-b54b-c4f98d55891d\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-91b4a-predictor-78f8d8cdc7-bv7nf"
May 26 16:08:25.210013 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:08:25.209988    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/151ad032-daaf-4774-b54b-c4f98d55891d-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-91b4a-predictor-78f8d8cdc7-bv7nf\" (UID: \"151ad032-daaf-4774-b54b-c4f98d55891d\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-91b4a-predictor-78f8d8cdc7-bv7nf"
May 26 16:08:25.210309 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:08:25.210285    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-raw-sklearn-batcher-91b4a-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/151ad032-daaf-4774-b54b-c4f98d55891d-isvc-raw-sklearn-batcher-91b4a-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-91b4a-predictor-78f8d8cdc7-bv7nf\" (UID: \"151ad032-daaf-4774-b54b-c4f98d55891d\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-91b4a-predictor-78f8d8cdc7-bv7nf"
May 26 16:08:25.212080 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:08:25.212062    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/151ad032-daaf-4774-b54b-c4f98d55891d-proxy-tls\") pod \"isvc-raw-sklearn-batcher-91b4a-predictor-78f8d8cdc7-bv7nf\" (UID: \"151ad032-daaf-4774-b54b-c4f98d55891d\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-91b4a-predictor-78f8d8cdc7-bv7nf"
May 26 16:08:25.219537 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:08:25.219514    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfht8\" (UniqueName: \"kubernetes.io/projected/151ad032-daaf-4774-b54b-c4f98d55891d-kube-api-access-kfht8\") pod \"isvc-raw-sklearn-batcher-91b4a-predictor-78f8d8cdc7-bv7nf\" (UID: \"151ad032-daaf-4774-b54b-c4f98d55891d\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-91b4a-predictor-78f8d8cdc7-bv7nf"
May 26 16:08:25.264559 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:08:25.264534    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-91b4a-predictor-78f8d8cdc7-bv7nf"
May 26 16:08:25.396689 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:08:25.396651    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-91b4a-predictor-78f8d8cdc7-bv7nf"]
May 26 16:08:25.400026 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:08:25.399995    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod151ad032_daaf_4774_b54b_c4f98d55891d.slice/crio-705f612aa1a24a6747f5fe761d7bed20fa45f27ecabe83b30e68347620aef93b WatchSource:0}: Error finding container 705f612aa1a24a6747f5fe761d7bed20fa45f27ecabe83b30e68347620aef93b: Status 404 returned error can't find the container with id 705f612aa1a24a6747f5fe761d7bed20fa45f27ecabe83b30e68347620aef93b
May 26 16:08:26.153938 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:08:26.153904    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-91b4a-predictor-78f8d8cdc7-bv7nf" event={"ID":"151ad032-daaf-4774-b54b-c4f98d55891d","Type":"ContainerStarted","Data":"705f612aa1a24a6747f5fe761d7bed20fa45f27ecabe83b30e68347620aef93b"}
May 26 16:08:30.170359 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:08:30.170318    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-91b4a-predictor-78f8d8cdc7-bv7nf" event={"ID":"151ad032-daaf-4774-b54b-c4f98d55891d","Type":"ContainerStarted","Data":"05eb4d617cf87d8ae6887847cabaff1bf7f0f817545b6c01377b5c7db79ccc8c"}
May 26 16:08:33.179750 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:08:33.179718    2556 generic.go:358] "Generic (PLEG): container finished" podID="151ad032-daaf-4774-b54b-c4f98d55891d" containerID="05eb4d617cf87d8ae6887847cabaff1bf7f0f817545b6c01377b5c7db79ccc8c" exitCode=0
May 26 16:08:33.180125 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:08:33.179794    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-91b4a-predictor-78f8d8cdc7-bv7nf" event={"ID":"151ad032-daaf-4774-b54b-c4f98d55891d","Type":"ContainerDied","Data":"05eb4d617cf87d8ae6887847cabaff1bf7f0f817545b6c01377b5c7db79ccc8c"}
May 26 16:08:47.237773 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:08:47.237732    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-91b4a-predictor-78f8d8cdc7-bv7nf" event={"ID":"151ad032-daaf-4774-b54b-c4f98d55891d","Type":"ContainerStarted","Data":"7f1951c35911f108af969e4be4d6ee2947e26cef89d4ab9d40bf33c7d0fb3cf9"}
May 26 16:08:49.248791 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:08:49.248752    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-91b4a-predictor-78f8d8cdc7-bv7nf" event={"ID":"151ad032-daaf-4774-b54b-c4f98d55891d","Type":"ContainerStarted","Data":"a4d0fe0ac3115c2b7d7dbc7d02f75e8ef7779665d127ec1d4d822caa50e22ead"}
May 26 16:08:52.259952 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:08:52.259917    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-91b4a-predictor-78f8d8cdc7-bv7nf" event={"ID":"151ad032-daaf-4774-b54b-c4f98d55891d","Type":"ContainerStarted","Data":"3750e3080a85e564a95c16e2bd552d7e4e33b357bbe4c138550227a02588648f"}
May 26 16:08:52.260442 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:08:52.260113    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-91b4a-predictor-78f8d8cdc7-bv7nf"
May 26 16:08:52.260442 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:08:52.260208    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-91b4a-predictor-78f8d8cdc7-bv7nf"
May 26 16:08:52.261680 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:08:52.261647    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-91b4a-predictor-78f8d8cdc7-bv7nf" podUID="151ad032-daaf-4774-b54b-c4f98d55891d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
May 26 16:08:52.288641 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:08:52.288531    2556 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-91b4a-predictor-78f8d8cdc7-bv7nf" podStartSLOduration=2.178628081 podStartE2EDuration="28.288517346s" podCreationTimestamp="2026-05-26 16:08:24 +0000 UTC" firstStartedPulling="2026-05-26 16:08:25.401799954 +0000 UTC m=+467.246416924" lastFinishedPulling="2026-05-26 16:08:51.511689216 +0000 UTC m=+493.356306189" observedRunningTime="2026-05-26 16:08:52.288286423 +0000 UTC m=+494.132903410" watchObservedRunningTime="2026-05-26 16:08:52.288517346 +0000 UTC m=+494.133134342"
May 26 16:08:53.263729 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:08:53.263694    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-91b4a-predictor-78f8d8cdc7-bv7nf"
May 26 16:08:53.264148 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:08:53.263899    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-91b4a-predictor-78f8d8cdc7-bv7nf" podUID="151ad032-daaf-4774-b54b-c4f98d55891d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
May 26 16:08:53.264691 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:08:53.264659    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-91b4a-predictor-78f8d8cdc7-bv7nf" podUID="151ad032-daaf-4774-b54b-c4f98d55891d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 16:08:53.267569 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:08:53.267553    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-91b4a-predictor-78f8d8cdc7-bv7nf"
May 26 16:08:54.267304 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:08:54.267274    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-91b4a-predictor-78f8d8cdc7-bv7nf" podUID="151ad032-daaf-4774-b54b-c4f98d55891d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
May 26 16:08:54.267692 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:08:54.267608    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-91b4a-predictor-78f8d8cdc7-bv7nf" podUID="151ad032-daaf-4774-b54b-c4f98d55891d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 16:08:55.270657 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:08:55.270617    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-91b4a-predictor-78f8d8cdc7-bv7nf" podUID="151ad032-daaf-4774-b54b-c4f98d55891d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
May 26 16:08:55.271059 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:08:55.270890    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-91b4a-predictor-78f8d8cdc7-bv7nf" podUID="151ad032-daaf-4774-b54b-c4f98d55891d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 16:08:59.776143 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:08:59.776105    2556 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7645f78b96-bfcs8"]
May 26 16:09:05.271579 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:09:05.271529    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-91b4a-predictor-78f8d8cdc7-bv7nf" podUID="151ad032-daaf-4774-b54b-c4f98d55891d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
May 26 16:09:05.272057 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:09:05.271927    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-91b4a-predictor-78f8d8cdc7-bv7nf" podUID="151ad032-daaf-4774-b54b-c4f98d55891d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 16:09:15.271418 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:09:15.271370    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-91b4a-predictor-78f8d8cdc7-bv7nf" podUID="151ad032-daaf-4774-b54b-c4f98d55891d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
May 26 16:09:15.271915 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:09:15.271861    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-91b4a-predictor-78f8d8cdc7-bv7nf" podUID="151ad032-daaf-4774-b54b-c4f98d55891d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 16:09:24.794727 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:09:24.794678    2556 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-console/console-7645f78b96-bfcs8" podUID="1e228518-0a79-4c9c-bda1-79a5e2bd6688" containerName="console" containerID="cri-o://85ed881fef26e5e84248b0804992ac85675e7430ab07d97c1557276a6d9fcb42" gracePeriod=15
May 26 16:09:25.108765 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:09:25.108743    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-7645f78b96-bfcs8_1e228518-0a79-4c9c-bda1-79a5e2bd6688/console/0.log"
May 26 16:09:25.108867 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:09:25.108802    2556 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7645f78b96-bfcs8"
May 26 16:09:25.202380 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:09:25.202355    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1e228518-0a79-4c9c-bda1-79a5e2bd6688-console-serving-cert\") pod \"1e228518-0a79-4c9c-bda1-79a5e2bd6688\" (UID: \"1e228518-0a79-4c9c-bda1-79a5e2bd6688\") "
May 26 16:09:25.202491 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:09:25.202387    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1e228518-0a79-4c9c-bda1-79a5e2bd6688-service-ca\") pod \"1e228518-0a79-4c9c-bda1-79a5e2bd6688\" (UID: \"1e228518-0a79-4c9c-bda1-79a5e2bd6688\") "
May 26 16:09:25.202491 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:09:25.202404    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1e228518-0a79-4c9c-bda1-79a5e2bd6688-console-oauth-config\") pod \"1e228518-0a79-4c9c-bda1-79a5e2bd6688\" (UID: \"1e228518-0a79-4c9c-bda1-79a5e2bd6688\") "
May 26 16:09:25.202491 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:09:25.202428    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1e228518-0a79-4c9c-bda1-79a5e2bd6688-console-config\") pod \"1e228518-0a79-4c9c-bda1-79a5e2bd6688\" (UID: \"1e228518-0a79-4c9c-bda1-79a5e2bd6688\") "
May 26 16:09:25.202491 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:09:25.202482    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7hqqf\" (UniqueName: \"kubernetes.io/projected/1e228518-0a79-4c9c-bda1-79a5e2bd6688-kube-api-access-7hqqf\") pod \"1e228518-0a79-4c9c-bda1-79a5e2bd6688\" (UID: \"1e228518-0a79-4c9c-bda1-79a5e2bd6688\") "
May 26 16:09:25.202646 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:09:25.202512    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1e228518-0a79-4c9c-bda1-79a5e2bd6688-trusted-ca-bundle\") pod \"1e228518-0a79-4c9c-bda1-79a5e2bd6688\" (UID: \"1e228518-0a79-4c9c-bda1-79a5e2bd6688\") "
May 26 16:09:25.202646 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:09:25.202575    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1e228518-0a79-4c9c-bda1-79a5e2bd6688-oauth-serving-cert\") pod \"1e228518-0a79-4c9c-bda1-79a5e2bd6688\" (UID: \"1e228518-0a79-4c9c-bda1-79a5e2bd6688\") "
May 26 16:09:25.202846 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:09:25.202819    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e228518-0a79-4c9c-bda1-79a5e2bd6688-service-ca" (OuterVolumeSpecName: "service-ca") pod "1e228518-0a79-4c9c-bda1-79a5e2bd6688" (UID: "1e228518-0a79-4c9c-bda1-79a5e2bd6688"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 16:09:25.202972 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:09:25.202863    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e228518-0a79-4c9c-bda1-79a5e2bd6688-console-config" (OuterVolumeSpecName: "console-config") pod "1e228518-0a79-4c9c-bda1-79a5e2bd6688" (UID: "1e228518-0a79-4c9c-bda1-79a5e2bd6688"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 16:09:25.203070 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:09:25.203036    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e228518-0a79-4c9c-bda1-79a5e2bd6688-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "1e228518-0a79-4c9c-bda1-79a5e2bd6688" (UID: "1e228518-0a79-4c9c-bda1-79a5e2bd6688"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 16:09:25.203168 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:09:25.203143    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e228518-0a79-4c9c-bda1-79a5e2bd6688-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "1e228518-0a79-4c9c-bda1-79a5e2bd6688" (UID: "1e228518-0a79-4c9c-bda1-79a5e2bd6688"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 16:09:25.204589 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:09:25.204564    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e228518-0a79-4c9c-bda1-79a5e2bd6688-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "1e228518-0a79-4c9c-bda1-79a5e2bd6688" (UID: "1e228518-0a79-4c9c-bda1-79a5e2bd6688"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 16:09:25.204589 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:09:25.204563    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e228518-0a79-4c9c-bda1-79a5e2bd6688-kube-api-access-7hqqf" (OuterVolumeSpecName: "kube-api-access-7hqqf") pod "1e228518-0a79-4c9c-bda1-79a5e2bd6688" (UID: "1e228518-0a79-4c9c-bda1-79a5e2bd6688"). InnerVolumeSpecName "kube-api-access-7hqqf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 16:09:25.204698 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:09:25.204591    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e228518-0a79-4c9c-bda1-79a5e2bd6688-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "1e228518-0a79-4c9c-bda1-79a5e2bd6688" (UID: "1e228518-0a79-4c9c-bda1-79a5e2bd6688"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 16:09:25.271265 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:09:25.271231    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-91b4a-predictor-78f8d8cdc7-bv7nf" podUID="151ad032-daaf-4774-b54b-c4f98d55891d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
May 26 16:09:25.271604 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:09:25.271573    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-91b4a-predictor-78f8d8cdc7-bv7nf" podUID="151ad032-daaf-4774-b54b-c4f98d55891d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 16:09:25.303985 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:09:25.303963    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7hqqf\" (UniqueName: \"kubernetes.io/projected/1e228518-0a79-4c9c-bda1-79a5e2bd6688-kube-api-access-7hqqf\") on node \"ip-10-0-134-19.ec2.internal\" DevicePath \"\""
May 26 16:09:25.304065 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:09:25.303987    2556 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1e228518-0a79-4c9c-bda1-79a5e2bd6688-trusted-ca-bundle\") on node \"ip-10-0-134-19.ec2.internal\" DevicePath \"\""
May 26 16:09:25.304065 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:09:25.304000    2556 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1e228518-0a79-4c9c-bda1-79a5e2bd6688-oauth-serving-cert\") on node \"ip-10-0-134-19.ec2.internal\" DevicePath \"\""
May 26 16:09:25.304065 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:09:25.304013    2556 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1e228518-0a79-4c9c-bda1-79a5e2bd6688-console-serving-cert\") on node \"ip-10-0-134-19.ec2.internal\" DevicePath \"\""
May 26 16:09:25.304065 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:09:25.304026    2556 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1e228518-0a79-4c9c-bda1-79a5e2bd6688-service-ca\") on node \"ip-10-0-134-19.ec2.internal\" DevicePath \"\""
May 26 16:09:25.304065 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:09:25.304035    2556 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1e228518-0a79-4c9c-bda1-79a5e2bd6688-console-oauth-config\") on node \"ip-10-0-134-19.ec2.internal\" DevicePath \"\""
May 26 16:09:25.304065 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:09:25.304044    2556 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1e228518-0a79-4c9c-bda1-79a5e2bd6688-console-config\") on node \"ip-10-0-134-19.ec2.internal\" DevicePath \"\""
May 26 16:09:25.367990 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:09:25.367972    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-7645f78b96-bfcs8_1e228518-0a79-4c9c-bda1-79a5e2bd6688/console/0.log"
May 26 16:09:25.368105 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:09:25.368005    2556 generic.go:358] "Generic (PLEG): container finished" podID="1e228518-0a79-4c9c-bda1-79a5e2bd6688" containerID="85ed881fef26e5e84248b0804992ac85675e7430ab07d97c1557276a6d9fcb42" exitCode=2
May 26 16:09:25.368105 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:09:25.368081    2556 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7645f78b96-bfcs8"
May 26 16:09:25.368221 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:09:25.368107    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7645f78b96-bfcs8" event={"ID":"1e228518-0a79-4c9c-bda1-79a5e2bd6688","Type":"ContainerDied","Data":"85ed881fef26e5e84248b0804992ac85675e7430ab07d97c1557276a6d9fcb42"}
May 26 16:09:25.368221 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:09:25.368143    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7645f78b96-bfcs8" event={"ID":"1e228518-0a79-4c9c-bda1-79a5e2bd6688","Type":"ContainerDied","Data":"298dd8249d09dd3fe67f80f1ddadbb03afc157d40c4e8459f65d9cee3ab5f09b"}
May 26 16:09:25.368221 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:09:25.368158    2556 scope.go:117] "RemoveContainer" containerID="85ed881fef26e5e84248b0804992ac85675e7430ab07d97c1557276a6d9fcb42"
May 26 16:09:25.376271 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:09:25.376246    2556 scope.go:117] "RemoveContainer" containerID="85ed881fef26e5e84248b0804992ac85675e7430ab07d97c1557276a6d9fcb42"
May 26 16:09:25.376501 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:09:25.376483    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85ed881fef26e5e84248b0804992ac85675e7430ab07d97c1557276a6d9fcb42\": container with ID starting with 85ed881fef26e5e84248b0804992ac85675e7430ab07d97c1557276a6d9fcb42 not found: ID does not exist" containerID="85ed881fef26e5e84248b0804992ac85675e7430ab07d97c1557276a6d9fcb42"
May 26 16:09:25.376557 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:09:25.376506    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85ed881fef26e5e84248b0804992ac85675e7430ab07d97c1557276a6d9fcb42"} err="failed to get container status \"85ed881fef26e5e84248b0804992ac85675e7430ab07d97c1557276a6d9fcb42\": rpc error: code = NotFound desc = could not find container \"85ed881fef26e5e84248b0804992ac85675e7430ab07d97c1557276a6d9fcb42\": container with ID starting with 85ed881fef26e5e84248b0804992ac85675e7430ab07d97c1557276a6d9fcb42 not found: ID does not exist"
May 26 16:09:25.392719 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:09:25.392696    2556 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7645f78b96-bfcs8"]
May 26 16:09:25.398127 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:09:25.398104    2556 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7645f78b96-bfcs8"]
May 26 16:09:26.771135 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:09:26.771079    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1e228518-0a79-4c9c-bda1-79a5e2bd6688" path="/var/lib/kubelet/pods/1e228518-0a79-4c9c-bda1-79a5e2bd6688/volumes"
May 26 16:09:35.271384 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:09:35.271299    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-91b4a-predictor-78f8d8cdc7-bv7nf" podUID="151ad032-daaf-4774-b54b-c4f98d55891d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
May 26 16:09:35.271829 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:09:35.271715    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-91b4a-predictor-78f8d8cdc7-bv7nf" podUID="151ad032-daaf-4774-b54b-c4f98d55891d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 16:09:45.271046 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:09:45.271004    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-91b4a-predictor-78f8d8cdc7-bv7nf" podUID="151ad032-daaf-4774-b54b-c4f98d55891d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
May 26 16:09:45.271546 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:09:45.271470    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-91b4a-predictor-78f8d8cdc7-bv7nf" podUID="151ad032-daaf-4774-b54b-c4f98d55891d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 16:09:55.271916 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:09:55.271889    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-91b4a-predictor-78f8d8cdc7-bv7nf"
May 26 16:09:55.272532 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:09:55.272281    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-91b4a-predictor-78f8d8cdc7-bv7nf"
May 26 16:10:10.003473 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:10:10.003435    2556 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-91b4a-predictor-78f8d8cdc7-bv7nf"]
May 26 16:10:10.003969 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:10:10.003881    2556 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-91b4a-predictor-78f8d8cdc7-bv7nf" podUID="151ad032-daaf-4774-b54b-c4f98d55891d" containerName="kserve-container" containerID="cri-o://7f1951c35911f108af969e4be4d6ee2947e26cef89d4ab9d40bf33c7d0fb3cf9" gracePeriod=30
May 26 16:10:10.004054 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:10:10.004033    2556 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-91b4a-predictor-78f8d8cdc7-bv7nf" podUID="151ad032-daaf-4774-b54b-c4f98d55891d" containerName="agent" containerID="cri-o://3750e3080a85e564a95c16e2bd552d7e4e33b357bbe4c138550227a02588648f" gracePeriod=30
May 26 16:10:10.004172 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:10:10.004123    2556 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-91b4a-predictor-78f8d8cdc7-bv7nf" podUID="151ad032-daaf-4774-b54b-c4f98d55891d" containerName="kube-rbac-proxy" containerID="cri-o://a4d0fe0ac3115c2b7d7dbc7d02f75e8ef7779665d127ec1d4d822caa50e22ead" gracePeriod=30
May 26 16:10:10.157970 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:10:10.157938    2556 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-5c32b-predictor-86f64c4b8f-7mvh8"]
May 26 16:10:10.158276 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:10:10.158263    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e228518-0a79-4c9c-bda1-79a5e2bd6688" containerName="console"
May 26 16:10:10.158276 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:10:10.158277    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e228518-0a79-4c9c-bda1-79a5e2bd6688" containerName="console"
May 26 16:10:10.158369 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:10:10.158339    2556 memory_manager.go:356] "RemoveStaleState removing state" podUID="1e228518-0a79-4c9c-bda1-79a5e2bd6688" containerName="console"
May 26 16:10:10.161290 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:10:10.161271    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-5c32b-predictor-86f64c4b8f-7mvh8"
May 26 16:10:10.166339 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:10:10.166277    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-5c32b-predictor-serving-cert\""
May 26 16:10:10.166486 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:10:10.166452    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-5c32b-kube-rbac-proxy-sar-config\""
May 26 16:10:10.186640 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:10:10.186617    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-5c32b-predictor-86f64c4b8f-7mvh8"]
May 26 16:10:10.220776 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:10:10.220743    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c1dbdd50-e6c5-4ac1-b33e-56f276cac3f6-proxy-tls\") pod \"isvc-sklearn-graph-raw-5c32b-predictor-86f64c4b8f-7mvh8\" (UID: \"c1dbdd50-e6c5-4ac1-b33e-56f276cac3f6\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-5c32b-predictor-86f64c4b8f-7mvh8"
May 26 16:10:10.220877 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:10:10.220805    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-sklearn-graph-raw-5c32b-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c1dbdd50-e6c5-4ac1-b33e-56f276cac3f6-isvc-sklearn-graph-raw-5c32b-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-5c32b-predictor-86f64c4b8f-7mvh8\" (UID: \"c1dbdd50-e6c5-4ac1-b33e-56f276cac3f6\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-5c32b-predictor-86f64c4b8f-7mvh8"
May 26 16:10:10.220877 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:10:10.220828    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c1dbdd50-e6c5-4ac1-b33e-56f276cac3f6-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-5c32b-predictor-86f64c4b8f-7mvh8\" (UID: \"c1dbdd50-e6c5-4ac1-b33e-56f276cac3f6\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-5c32b-predictor-86f64c4b8f-7mvh8"
May 26 16:10:10.220877 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:10:10.220851    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmsrc\" (UniqueName: \"kubernetes.io/projected/c1dbdd50-e6c5-4ac1-b33e-56f276cac3f6-kube-api-access-gmsrc\") pod \"isvc-sklearn-graph-raw-5c32b-predictor-86f64c4b8f-7mvh8\" (UID: \"c1dbdd50-e6c5-4ac1-b33e-56f276cac3f6\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-5c32b-predictor-86f64c4b8f-7mvh8"
May 26 16:10:10.322244 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:10:10.322175    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-sklearn-graph-raw-5c32b-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c1dbdd50-e6c5-4ac1-b33e-56f276cac3f6-isvc-sklearn-graph-raw-5c32b-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-5c32b-predictor-86f64c4b8f-7mvh8\" (UID: \"c1dbdd50-e6c5-4ac1-b33e-56f276cac3f6\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-5c32b-predictor-86f64c4b8f-7mvh8"
May 26 16:10:10.322244 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:10:10.322210    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c1dbdd50-e6c5-4ac1-b33e-56f276cac3f6-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-5c32b-predictor-86f64c4b8f-7mvh8\" (UID: \"c1dbdd50-e6c5-4ac1-b33e-56f276cac3f6\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-5c32b-predictor-86f64c4b8f-7mvh8"
May 26 16:10:10.322244 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:10:10.322239    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-gmsrc\" (UniqueName: \"kubernetes.io/projected/c1dbdd50-e6c5-4ac1-b33e-56f276cac3f6-kube-api-access-gmsrc\") pod \"isvc-sklearn-graph-raw-5c32b-predictor-86f64c4b8f-7mvh8\" (UID: \"c1dbdd50-e6c5-4ac1-b33e-56f276cac3f6\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-5c32b-predictor-86f64c4b8f-7mvh8"
May 26 16:10:10.322459 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:10:10.322283    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c1dbdd50-e6c5-4ac1-b33e-56f276cac3f6-proxy-tls\") pod \"isvc-sklearn-graph-raw-5c32b-predictor-86f64c4b8f-7mvh8\" (UID: \"c1dbdd50-e6c5-4ac1-b33e-56f276cac3f6\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-5c32b-predictor-86f64c4b8f-7mvh8"
May 26 16:10:10.322505 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:10:10.322474    2556 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-graph-raw-5c32b-predictor-serving-cert: secret "isvc-sklearn-graph-raw-5c32b-predictor-serving-cert" not found
May 26 16:10:10.322547 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:10:10.322537    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c1dbdd50-e6c5-4ac1-b33e-56f276cac3f6-proxy-tls podName:c1dbdd50-e6c5-4ac1-b33e-56f276cac3f6 nodeName:}" failed. No retries permitted until 2026-05-26 16:10:10.822517412 +0000 UTC m=+572.667134383 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/c1dbdd50-e6c5-4ac1-b33e-56f276cac3f6-proxy-tls") pod "isvc-sklearn-graph-raw-5c32b-predictor-86f64c4b8f-7mvh8" (UID: "c1dbdd50-e6c5-4ac1-b33e-56f276cac3f6") : secret "isvc-sklearn-graph-raw-5c32b-predictor-serving-cert" not found
May 26 16:10:10.322672 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:10:10.322654    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c1dbdd50-e6c5-4ac1-b33e-56f276cac3f6-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-5c32b-predictor-86f64c4b8f-7mvh8\" (UID: \"c1dbdd50-e6c5-4ac1-b33e-56f276cac3f6\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-5c32b-predictor-86f64c4b8f-7mvh8"
May 26 16:10:10.322965 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:10:10.322942    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-sklearn-graph-raw-5c32b-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c1dbdd50-e6c5-4ac1-b33e-56f276cac3f6-isvc-sklearn-graph-raw-5c32b-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-5c32b-predictor-86f64c4b8f-7mvh8\" (UID: \"c1dbdd50-e6c5-4ac1-b33e-56f276cac3f6\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-5c32b-predictor-86f64c4b8f-7mvh8"
May 26 16:10:10.341858 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:10:10.341831    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmsrc\" (UniqueName: \"kubernetes.io/projected/c1dbdd50-e6c5-4ac1-b33e-56f276cac3f6-kube-api-access-gmsrc\") pod \"isvc-sklearn-graph-raw-5c32b-predictor-86f64c4b8f-7mvh8\" (UID: \"c1dbdd50-e6c5-4ac1-b33e-56f276cac3f6\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-5c32b-predictor-86f64c4b8f-7mvh8"
May 26 16:10:10.512441 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:10:10.512408    2556 generic.go:358] "Generic (PLEG): container finished" podID="151ad032-daaf-4774-b54b-c4f98d55891d" containerID="a4d0fe0ac3115c2b7d7dbc7d02f75e8ef7779665d127ec1d4d822caa50e22ead" exitCode=2
May 26 16:10:10.512577 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:10:10.512462    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-91b4a-predictor-78f8d8cdc7-bv7nf" event={"ID":"151ad032-daaf-4774-b54b-c4f98d55891d","Type":"ContainerDied","Data":"a4d0fe0ac3115c2b7d7dbc7d02f75e8ef7779665d127ec1d4d822caa50e22ead"}
May 26 16:10:10.827198 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:10:10.827163    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c1dbdd50-e6c5-4ac1-b33e-56f276cac3f6-proxy-tls\") pod \"isvc-sklearn-graph-raw-5c32b-predictor-86f64c4b8f-7mvh8\" (UID: \"c1dbdd50-e6c5-4ac1-b33e-56f276cac3f6\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-5c32b-predictor-86f64c4b8f-7mvh8"
May 26 16:10:10.829609 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:10:10.829578    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c1dbdd50-e6c5-4ac1-b33e-56f276cac3f6-proxy-tls\") pod \"isvc-sklearn-graph-raw-5c32b-predictor-86f64c4b8f-7mvh8\" (UID: \"c1dbdd50-e6c5-4ac1-b33e-56f276cac3f6\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-5c32b-predictor-86f64c4b8f-7mvh8"
May 26 16:10:11.073500 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:10:11.073470    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-5c32b-predictor-86f64c4b8f-7mvh8"
May 26 16:10:11.204887 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:10:11.204860    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-5c32b-predictor-86f64c4b8f-7mvh8"]
May 26 16:10:11.207189 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:10:11.207158    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1dbdd50_e6c5_4ac1_b33e_56f276cac3f6.slice/crio-6adbcd5bc50d69f21ed61f554fcfe0382efe10a46dcee743fdaaf37d897d9ece WatchSource:0}: Error finding container 6adbcd5bc50d69f21ed61f554fcfe0382efe10a46dcee743fdaaf37d897d9ece: Status 404 returned error can't find the container with id 6adbcd5bc50d69f21ed61f554fcfe0382efe10a46dcee743fdaaf37d897d9ece
May 26 16:10:11.516925 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:10:11.516888    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-5c32b-predictor-86f64c4b8f-7mvh8" event={"ID":"c1dbdd50-e6c5-4ac1-b33e-56f276cac3f6","Type":"ContainerStarted","Data":"b7ddd4ffdd91cc866f5b9cae692ab228cc2dccd5748038a734134c879576870d"}
May 26 16:10:11.516925 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:10:11.516925    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-5c32b-predictor-86f64c4b8f-7mvh8" event={"ID":"c1dbdd50-e6c5-4ac1-b33e-56f276cac3f6","Type":"ContainerStarted","Data":"6adbcd5bc50d69f21ed61f554fcfe0382efe10a46dcee743fdaaf37d897d9ece"}
May 26 16:10:13.264472 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:10:13.264433    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-91b4a-predictor-78f8d8cdc7-bv7nf" podUID="151ad032-daaf-4774-b54b-c4f98d55891d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.27:8643/healthz\": dial tcp 10.133.0.27:8643: connect: connection refused"
May 26 16:10:14.528576 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:10:14.528539    2556 generic.go:358] "Generic (PLEG): container finished" podID="151ad032-daaf-4774-b54b-c4f98d55891d" containerID="7f1951c35911f108af969e4be4d6ee2947e26cef89d4ab9d40bf33c7d0fb3cf9" exitCode=0
May 26 16:10:14.528945 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:10:14.528609    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-91b4a-predictor-78f8d8cdc7-bv7nf" event={"ID":"151ad032-daaf-4774-b54b-c4f98d55891d","Type":"ContainerDied","Data":"7f1951c35911f108af969e4be4d6ee2947e26cef89d4ab9d40bf33c7d0fb3cf9"}
May 26 16:10:15.271080 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:10:15.271003    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-91b4a-predictor-78f8d8cdc7-bv7nf" podUID="151ad032-daaf-4774-b54b-c4f98d55891d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
May 26 16:10:15.271341 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:10:15.271314    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-91b4a-predictor-78f8d8cdc7-bv7nf" podUID="151ad032-daaf-4774-b54b-c4f98d55891d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 16:10:15.533660 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:10:15.533591    2556 generic.go:358] "Generic (PLEG): container finished" podID="c1dbdd50-e6c5-4ac1-b33e-56f276cac3f6" containerID="b7ddd4ffdd91cc866f5b9cae692ab228cc2dccd5748038a734134c879576870d" exitCode=0
May 26 16:10:15.533660 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:10:15.533625    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-5c32b-predictor-86f64c4b8f-7mvh8" event={"ID":"c1dbdd50-e6c5-4ac1-b33e-56f276cac3f6","Type":"ContainerDied","Data":"b7ddd4ffdd91cc866f5b9cae692ab228cc2dccd5748038a734134c879576870d"}
May 26 16:10:16.538198 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:10:16.538160    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-5c32b-predictor-86f64c4b8f-7mvh8" event={"ID":"c1dbdd50-e6c5-4ac1-b33e-56f276cac3f6","Type":"ContainerStarted","Data":"43fcd1acdd049e5c9343b274f3bd5efea93c36fd17495467016d21a84bc95dd3"}
May 26 16:10:16.538198 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:10:16.538196    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-5c32b-predictor-86f64c4b8f-7mvh8" event={"ID":"c1dbdd50-e6c5-4ac1-b33e-56f276cac3f6","Type":"ContainerStarted","Data":"f358beb3ca5cb1c1129d1e80fbe4272bc19831d877e2d3d347eea046e4845fc2"}
May 26 16:10:16.538603 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:10:16.538478    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-5c32b-predictor-86f64c4b8f-7mvh8"
May 26 16:10:16.538643 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:10:16.538608    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-5c32b-predictor-86f64c4b8f-7mvh8"
May 26 16:10:16.539979 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:10:16.539950    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-5c32b-predictor-86f64c4b8f-7mvh8" podUID="c1dbdd50-e6c5-4ac1-b33e-56f276cac3f6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
May 26 16:10:16.560262 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:10:16.560218    2556 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-5c32b-predictor-86f64c4b8f-7mvh8" podStartSLOduration=6.560202836 podStartE2EDuration="6.560202836s" podCreationTimestamp="2026-05-26 16:10:10 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 16:10:16.558520266 +0000 UTC m=+578.403137257" watchObservedRunningTime="2026-05-26 16:10:16.560202836 +0000 UTC m=+578.404819831"
May 26 16:10:17.541624 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:10:17.541589    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-5c32b-predictor-86f64c4b8f-7mvh8" podUID="c1dbdd50-e6c5-4ac1-b33e-56f276cac3f6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
May 26 16:10:18.264808 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:10:18.264770    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-91b4a-predictor-78f8d8cdc7-bv7nf" podUID="151ad032-daaf-4774-b54b-c4f98d55891d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.27:8643/healthz\": dial tcp 10.133.0.27:8643: connect: connection refused"
May 26 16:10:22.546486 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:10:22.546458    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-5c32b-predictor-86f64c4b8f-7mvh8"
May 26 16:10:22.547068 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:10:22.547042    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-5c32b-predictor-86f64c4b8f-7mvh8" podUID="c1dbdd50-e6c5-4ac1-b33e-56f276cac3f6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
May 26 16:10:23.264171 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:10:23.264130    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-91b4a-predictor-78f8d8cdc7-bv7nf" podUID="151ad032-daaf-4774-b54b-c4f98d55891d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.27:8643/healthz\": dial tcp 10.133.0.27:8643: connect: connection refused"
May 26 16:10:23.264339 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:10:23.264251    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-91b4a-predictor-78f8d8cdc7-bv7nf"
May 26 16:10:25.270582 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:10:25.270538    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-91b4a-predictor-78f8d8cdc7-bv7nf" podUID="151ad032-daaf-4774-b54b-c4f98d55891d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
May 26 16:10:25.270955 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:10:25.270807    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-91b4a-predictor-78f8d8cdc7-bv7nf" podUID="151ad032-daaf-4774-b54b-c4f98d55891d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 16:10:28.264441 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:10:28.264399    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-91b4a-predictor-78f8d8cdc7-bv7nf" podUID="151ad032-daaf-4774-b54b-c4f98d55891d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.27:8643/healthz\": dial tcp 10.133.0.27:8643: connect: connection refused"
May 26 16:10:32.547901 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:10:32.547864    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-5c32b-predictor-86f64c4b8f-7mvh8" podUID="c1dbdd50-e6c5-4ac1-b33e-56f276cac3f6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
May 26 16:10:33.264450 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:10:33.264411    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-91b4a-predictor-78f8d8cdc7-bv7nf" podUID="151ad032-daaf-4774-b54b-c4f98d55891d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.27:8643/healthz\": dial tcp 10.133.0.27:8643: connect: connection refused"
May 26 16:10:35.271084 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:10:35.271035    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-91b4a-predictor-78f8d8cdc7-bv7nf" podUID="151ad032-daaf-4774-b54b-c4f98d55891d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
May 26 16:10:35.271478 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:10:35.271212    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-91b4a-predictor-78f8d8cdc7-bv7nf"
May 26 16:10:35.271478 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:10:35.271320    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-91b4a-predictor-78f8d8cdc7-bv7nf" podUID="151ad032-daaf-4774-b54b-c4f98d55891d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 16:10:35.271478 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:10:35.271408    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-91b4a-predictor-78f8d8cdc7-bv7nf"
May 26 16:10:38.264583 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:10:38.264537    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-91b4a-predictor-78f8d8cdc7-bv7nf" podUID="151ad032-daaf-4774-b54b-c4f98d55891d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.27:8643/healthz\": dial tcp 10.133.0.27:8643: connect: connection refused"
May 26 16:10:38.667288 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:10:38.667260    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sk7ns_4887c0d3-11fb-4287-bb94-a876a6c38803/ovn-acl-logging/0.log"
May 26 16:10:38.667288 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:10:38.667283    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sk7ns_4887c0d3-11fb-4287-bb94-a876a6c38803/ovn-acl-logging/0.log"
May 26 16:10:40.157911 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:10:40.157887    2556 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-91b4a-predictor-78f8d8cdc7-bv7nf"
May 26 16:10:40.256639 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:10:40.256613    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-raw-sklearn-batcher-91b4a-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/151ad032-daaf-4774-b54b-c4f98d55891d-isvc-raw-sklearn-batcher-91b4a-kube-rbac-proxy-sar-config\") pod \"151ad032-daaf-4774-b54b-c4f98d55891d\" (UID: \"151ad032-daaf-4774-b54b-c4f98d55891d\") "
May 26 16:10:40.256795 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:10:40.256668    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kfht8\" (UniqueName: \"kubernetes.io/projected/151ad032-daaf-4774-b54b-c4f98d55891d-kube-api-access-kfht8\") pod \"151ad032-daaf-4774-b54b-c4f98d55891d\" (UID: \"151ad032-daaf-4774-b54b-c4f98d55891d\") "
May 26 16:10:40.256795 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:10:40.256698    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/151ad032-daaf-4774-b54b-c4f98d55891d-proxy-tls\") pod \"151ad032-daaf-4774-b54b-c4f98d55891d\" (UID: \"151ad032-daaf-4774-b54b-c4f98d55891d\") "
May 26 16:10:40.256795 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:10:40.256730    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/151ad032-daaf-4774-b54b-c4f98d55891d-kserve-provision-location\") pod \"151ad032-daaf-4774-b54b-c4f98d55891d\" (UID: \"151ad032-daaf-4774-b54b-c4f98d55891d\") "
May 26 16:10:40.257035 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:10:40.257013    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/151ad032-daaf-4774-b54b-c4f98d55891d-isvc-raw-sklearn-batcher-91b4a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-raw-sklearn-batcher-91b4a-kube-rbac-proxy-sar-config") pod "151ad032-daaf-4774-b54b-c4f98d55891d" (UID: "151ad032-daaf-4774-b54b-c4f98d55891d"). InnerVolumeSpecName "isvc-raw-sklearn-batcher-91b4a-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 16:10:40.257161 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:10:40.257140    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/151ad032-daaf-4774-b54b-c4f98d55891d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "151ad032-daaf-4774-b54b-c4f98d55891d" (UID: "151ad032-daaf-4774-b54b-c4f98d55891d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 16:10:40.258816 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:10:40.258762    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/151ad032-daaf-4774-b54b-c4f98d55891d-kube-api-access-kfht8" (OuterVolumeSpecName: "kube-api-access-kfht8") pod "151ad032-daaf-4774-b54b-c4f98d55891d" (UID: "151ad032-daaf-4774-b54b-c4f98d55891d"). InnerVolumeSpecName "kube-api-access-kfht8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 16:10:40.258816 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:10:40.258788    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/151ad032-daaf-4774-b54b-c4f98d55891d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "151ad032-daaf-4774-b54b-c4f98d55891d" (UID: "151ad032-daaf-4774-b54b-c4f98d55891d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 16:10:40.357281 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:10:40.357251    2556 reconciler_common.go:299] "Volume detached for volume \"isvc-raw-sklearn-batcher-91b4a-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/151ad032-daaf-4774-b54b-c4f98d55891d-isvc-raw-sklearn-batcher-91b4a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-19.ec2.internal\" DevicePath \"\""
May 26 16:10:40.357281 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:10:40.357279    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kfht8\" (UniqueName: \"kubernetes.io/projected/151ad032-daaf-4774-b54b-c4f98d55891d-kube-api-access-kfht8\") on node \"ip-10-0-134-19.ec2.internal\" DevicePath \"\""
May 26 16:10:40.357441 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:10:40.357293    2556 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/151ad032-daaf-4774-b54b-c4f98d55891d-proxy-tls\") on node \"ip-10-0-134-19.ec2.internal\" DevicePath \"\""
May 26 16:10:40.357441 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:10:40.357309    2556 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/151ad032-daaf-4774-b54b-c4f98d55891d-kserve-provision-location\") on node \"ip-10-0-134-19.ec2.internal\" DevicePath \"\""
May 26 16:10:40.612698 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:10:40.612613    2556 generic.go:358] "Generic (PLEG): container finished" podID="151ad032-daaf-4774-b54b-c4f98d55891d" containerID="3750e3080a85e564a95c16e2bd552d7e4e33b357bbe4c138550227a02588648f" exitCode=0
May 26 16:10:40.612832 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:10:40.612700    2556 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-91b4a-predictor-78f8d8cdc7-bv7nf"
May 26 16:10:40.612832 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:10:40.612700    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-91b4a-predictor-78f8d8cdc7-bv7nf" event={"ID":"151ad032-daaf-4774-b54b-c4f98d55891d","Type":"ContainerDied","Data":"3750e3080a85e564a95c16e2bd552d7e4e33b357bbe4c138550227a02588648f"}
May 26 16:10:40.612832 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:10:40.612743    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-91b4a-predictor-78f8d8cdc7-bv7nf" event={"ID":"151ad032-daaf-4774-b54b-c4f98d55891d","Type":"ContainerDied","Data":"705f612aa1a24a6747f5fe761d7bed20fa45f27ecabe83b30e68347620aef93b"}
May 26 16:10:40.612832 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:10:40.612759    2556 scope.go:117] "RemoveContainer" containerID="3750e3080a85e564a95c16e2bd552d7e4e33b357bbe4c138550227a02588648f"
May 26 16:10:40.620700 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:10:40.620680    2556 scope.go:117] "RemoveContainer" containerID="a4d0fe0ac3115c2b7d7dbc7d02f75e8ef7779665d127ec1d4d822caa50e22ead"
May 26 16:10:40.627586 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:10:40.627571    2556 scope.go:117] "RemoveContainer" containerID="7f1951c35911f108af969e4be4d6ee2947e26cef89d4ab9d40bf33c7d0fb3cf9"
May 26 16:10:40.633872 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:10:40.633852    2556 scope.go:117] "RemoveContainer" containerID="05eb4d617cf87d8ae6887847cabaff1bf7f0f817545b6c01377b5c7db79ccc8c"
May 26 16:10:40.638564 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:10:40.638544    2556 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-91b4a-predictor-78f8d8cdc7-bv7nf"]
May 26 16:10:40.640752 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:10:40.640738    2556 scope.go:117] "RemoveContainer" containerID="3750e3080a85e564a95c16e2bd552d7e4e33b357bbe4c138550227a02588648f"
May 26 16:10:40.641001 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:10:40.640983    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3750e3080a85e564a95c16e2bd552d7e4e33b357bbe4c138550227a02588648f\": container with ID starting with 3750e3080a85e564a95c16e2bd552d7e4e33b357bbe4c138550227a02588648f not found: ID does not exist" containerID="3750e3080a85e564a95c16e2bd552d7e4e33b357bbe4c138550227a02588648f"
May 26 16:10:40.641050 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:10:40.641008    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3750e3080a85e564a95c16e2bd552d7e4e33b357bbe4c138550227a02588648f"} err="failed to get container status \"3750e3080a85e564a95c16e2bd552d7e4e33b357bbe4c138550227a02588648f\": rpc error: code = NotFound desc = could not find container \"3750e3080a85e564a95c16e2bd552d7e4e33b357bbe4c138550227a02588648f\": container with ID starting with 3750e3080a85e564a95c16e2bd552d7e4e33b357bbe4c138550227a02588648f not found: ID does not exist"
May 26 16:10:40.641050 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:10:40.641025    2556 scope.go:117] "RemoveContainer" containerID="a4d0fe0ac3115c2b7d7dbc7d02f75e8ef7779665d127ec1d4d822caa50e22ead"
May 26 16:10:40.641261 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:10:40.641240    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4d0fe0ac3115c2b7d7dbc7d02f75e8ef7779665d127ec1d4d822caa50e22ead\": container with ID starting with a4d0fe0ac3115c2b7d7dbc7d02f75e8ef7779665d127ec1d4d822caa50e22ead not found: ID does not exist" containerID="a4d0fe0ac3115c2b7d7dbc7d02f75e8ef7779665d127ec1d4d822caa50e22ead"
May 26 16:10:40.641305 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:10:40.641277    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4d0fe0ac3115c2b7d7dbc7d02f75e8ef7779665d127ec1d4d822caa50e22ead"} err="failed to get container status \"a4d0fe0ac3115c2b7d7dbc7d02f75e8ef7779665d127ec1d4d822caa50e22ead\": rpc error: code = NotFound desc = could not find container \"a4d0fe0ac3115c2b7d7dbc7d02f75e8ef7779665d127ec1d4d822caa50e22ead\": container with ID starting with a4d0fe0ac3115c2b7d7dbc7d02f75e8ef7779665d127ec1d4d822caa50e22ead not found: ID does not exist"
May 26 16:10:40.641305 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:10:40.641299    2556 scope.go:117] "RemoveContainer" containerID="7f1951c35911f108af969e4be4d6ee2947e26cef89d4ab9d40bf33c7d0fb3cf9"
May 26 16:10:40.641522 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:10:40.641501    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f1951c35911f108af969e4be4d6ee2947e26cef89d4ab9d40bf33c7d0fb3cf9\": container with ID starting with 7f1951c35911f108af969e4be4d6ee2947e26cef89d4ab9d40bf33c7d0fb3cf9 not found: ID does not exist" containerID="7f1951c35911f108af969e4be4d6ee2947e26cef89d4ab9d40bf33c7d0fb3cf9"
May 26 16:10:40.641646 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:10:40.641522    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f1951c35911f108af969e4be4d6ee2947e26cef89d4ab9d40bf33c7d0fb3cf9"} err="failed to get container status \"7f1951c35911f108af969e4be4d6ee2947e26cef89d4ab9d40bf33c7d0fb3cf9\": rpc error: code = NotFound desc = could not find container \"7f1951c35911f108af969e4be4d6ee2947e26cef89d4ab9d40bf33c7d0fb3cf9\": container with ID starting with 7f1951c35911f108af969e4be4d6ee2947e26cef89d4ab9d40bf33c7d0fb3cf9 not found: ID does not exist"
May 26 16:10:40.641646 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:10:40.641539    2556 scope.go:117] "RemoveContainer" containerID="05eb4d617cf87d8ae6887847cabaff1bf7f0f817545b6c01377b5c7db79ccc8c"
May 26 16:10:40.642028 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:10:40.641956    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05eb4d617cf87d8ae6887847cabaff1bf7f0f817545b6c01377b5c7db79ccc8c\": container with ID starting with 05eb4d617cf87d8ae6887847cabaff1bf7f0f817545b6c01377b5c7db79ccc8c not found: ID does not exist" containerID="05eb4d617cf87d8ae6887847cabaff1bf7f0f817545b6c01377b5c7db79ccc8c"
May 26 16:10:40.642217 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:10:40.642022    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05eb4d617cf87d8ae6887847cabaff1bf7f0f817545b6c01377b5c7db79ccc8c"} err="failed to get container status \"05eb4d617cf87d8ae6887847cabaff1bf7f0f817545b6c01377b5c7db79ccc8c\": rpc error: code = NotFound desc = could not find container \"05eb4d617cf87d8ae6887847cabaff1bf7f0f817545b6c01377b5c7db79ccc8c\": container with ID starting with 05eb4d617cf87d8ae6887847cabaff1bf7f0f817545b6c01377b5c7db79ccc8c not found: ID does not exist"
May 26 16:10:40.643431 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:10:40.643409    2556 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-91b4a-predictor-78f8d8cdc7-bv7nf"]
May 26 16:10:40.770681 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:10:40.770657    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="151ad032-daaf-4774-b54b-c4f98d55891d" path="/var/lib/kubelet/pods/151ad032-daaf-4774-b54b-c4f98d55891d/volumes"
May 26 16:10:42.547806 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:10:42.547766    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-5c32b-predictor-86f64c4b8f-7mvh8" podUID="c1dbdd50-e6c5-4ac1-b33e-56f276cac3f6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
May 26 16:10:52.547520 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:10:52.547475    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-5c32b-predictor-86f64c4b8f-7mvh8" podUID="c1dbdd50-e6c5-4ac1-b33e-56f276cac3f6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
May 26 16:11:02.547316 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:11:02.547276    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-5c32b-predictor-86f64c4b8f-7mvh8" podUID="c1dbdd50-e6c5-4ac1-b33e-56f276cac3f6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
May 26 16:11:12.547647 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:11:12.547608    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-5c32b-predictor-86f64c4b8f-7mvh8" podUID="c1dbdd50-e6c5-4ac1-b33e-56f276cac3f6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
May 26 16:11:22.547487 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:11:22.547456    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-5c32b-predictor-86f64c4b8f-7mvh8"
May 26 16:11:50.541906 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:11:50.541865    2556 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-5c32b-predictor-86f64c4b8f-7mvh8"]
May 26 16:11:50.542476 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:11:50.542173    2556 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-5c32b-predictor-86f64c4b8f-7mvh8" podUID="c1dbdd50-e6c5-4ac1-b33e-56f276cac3f6" containerName="kserve-container" containerID="cri-o://f358beb3ca5cb1c1129d1e80fbe4272bc19831d877e2d3d347eea046e4845fc2" gracePeriod=30
May 26 16:11:50.542476 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:11:50.542254    2556 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-5c32b-predictor-86f64c4b8f-7mvh8" podUID="c1dbdd50-e6c5-4ac1-b33e-56f276cac3f6" containerName="kube-rbac-proxy" containerID="cri-o://43fcd1acdd049e5c9343b274f3bd5efea93c36fd17495467016d21a84bc95dd3" gracePeriod=30
May 26 16:11:50.588619 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:11:50.588586    2556 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-89f63-predictor-564cbb96cc-ct4hd"]
May 26 16:11:50.588944 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:11:50.588928    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="151ad032-daaf-4774-b54b-c4f98d55891d" containerName="storage-initializer"
May 26 16:11:50.589024 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:11:50.588947    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="151ad032-daaf-4774-b54b-c4f98d55891d" containerName="storage-initializer"
May 26 16:11:50.589024 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:11:50.588963    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="151ad032-daaf-4774-b54b-c4f98d55891d" containerName="kube-rbac-proxy"
May 26 16:11:50.589024 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:11:50.588971    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="151ad032-daaf-4774-b54b-c4f98d55891d" containerName="kube-rbac-proxy"
May 26 16:11:50.589024 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:11:50.588983    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="151ad032-daaf-4774-b54b-c4f98d55891d" containerName="kserve-container"
May 26 16:11:50.589024 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:11:50.588992    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="151ad032-daaf-4774-b54b-c4f98d55891d" containerName="kserve-container"
May 26 16:11:50.589024 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:11:50.589013    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="151ad032-daaf-4774-b54b-c4f98d55891d" containerName="agent"
May 26 16:11:50.589024 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:11:50.589021    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="151ad032-daaf-4774-b54b-c4f98d55891d" containerName="agent"
May 26 16:11:50.589397 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:11:50.589120    2556 memory_manager.go:356] "RemoveStaleState removing state" podUID="151ad032-daaf-4774-b54b-c4f98d55891d" containerName="agent"
May 26 16:11:50.589397 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:11:50.589134    2556 memory_manager.go:356] "RemoveStaleState removing state" podUID="151ad032-daaf-4774-b54b-c4f98d55891d" containerName="kserve-container"
May 26 16:11:50.589397 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:11:50.589147    2556 memory_manager.go:356] "RemoveStaleState removing state" podUID="151ad032-daaf-4774-b54b-c4f98d55891d" containerName="kube-rbac-proxy"
May 26 16:11:50.591769 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:11:50.591750    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-89f63-predictor-564cbb96cc-ct4hd"
May 26 16:11:50.594842 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:11:50.594826    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-hpa-89f63-predictor-serving-cert\""
May 26 16:11:50.595175 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:11:50.595154    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-hpa-89f63-kube-rbac-proxy-sar-config\""
May 26 16:11:50.607598 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:11:50.607575    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-89f63-predictor-564cbb96cc-ct4hd"]
May 26 16:11:50.669080 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:11:50.669049    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tw84s\" (UniqueName: \"kubernetes.io/projected/43de9a6e-da72-41b0-97ca-0e4a7dadb2f8-kube-api-access-tw84s\") pod \"isvc-sklearn-graph-raw-hpa-89f63-predictor-564cbb96cc-ct4hd\" (UID: \"43de9a6e-da72-41b0-97ca-0e4a7dadb2f8\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-89f63-predictor-564cbb96cc-ct4hd"
May 26 16:11:50.669274 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:11:50.669178    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/43de9a6e-da72-41b0-97ca-0e4a7dadb2f8-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-89f63-predictor-564cbb96cc-ct4hd\" (UID: \"43de9a6e-da72-41b0-97ca-0e4a7dadb2f8\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-89f63-predictor-564cbb96cc-ct4hd"
May 26 16:11:50.669274 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:11:50.669238    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-sklearn-graph-raw-hpa-89f63-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/43de9a6e-da72-41b0-97ca-0e4a7dadb2f8-isvc-sklearn-graph-raw-hpa-89f63-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-89f63-predictor-564cbb96cc-ct4hd\" (UID: \"43de9a6e-da72-41b0-97ca-0e4a7dadb2f8\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-89f63-predictor-564cbb96cc-ct4hd"
May 26 16:11:50.669274 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:11:50.669266    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/43de9a6e-da72-41b0-97ca-0e4a7dadb2f8-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-89f63-predictor-564cbb96cc-ct4hd\" (UID: \"43de9a6e-da72-41b0-97ca-0e4a7dadb2f8\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-89f63-predictor-564cbb96cc-ct4hd"
May 26 16:11:50.769936 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:11:50.769908    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-sklearn-graph-raw-hpa-89f63-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/43de9a6e-da72-41b0-97ca-0e4a7dadb2f8-isvc-sklearn-graph-raw-hpa-89f63-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-89f63-predictor-564cbb96cc-ct4hd\" (UID: \"43de9a6e-da72-41b0-97ca-0e4a7dadb2f8\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-89f63-predictor-564cbb96cc-ct4hd"
May 26 16:11:50.770116 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:11:50.769946    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/43de9a6e-da72-41b0-97ca-0e4a7dadb2f8-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-89f63-predictor-564cbb96cc-ct4hd\" (UID: \"43de9a6e-da72-41b0-97ca-0e4a7dadb2f8\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-89f63-predictor-564cbb96cc-ct4hd"
May 26 16:11:50.770116 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:11:50.770083    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-tw84s\" (UniqueName: \"kubernetes.io/projected/43de9a6e-da72-41b0-97ca-0e4a7dadb2f8-kube-api-access-tw84s\") pod \"isvc-sklearn-graph-raw-hpa-89f63-predictor-564cbb96cc-ct4hd\" (UID: \"43de9a6e-da72-41b0-97ca-0e4a7dadb2f8\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-89f63-predictor-564cbb96cc-ct4hd"
May 26 16:11:50.770251 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:11:50.770210    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/43de9a6e-da72-41b0-97ca-0e4a7dadb2f8-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-89f63-predictor-564cbb96cc-ct4hd\" (UID: \"43de9a6e-da72-41b0-97ca-0e4a7dadb2f8\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-89f63-predictor-564cbb96cc-ct4hd"
May 26 16:11:50.770373 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:11:50.770351    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/43de9a6e-da72-41b0-97ca-0e4a7dadb2f8-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-89f63-predictor-564cbb96cc-ct4hd\" (UID: \"43de9a6e-da72-41b0-97ca-0e4a7dadb2f8\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-89f63-predictor-564cbb96cc-ct4hd"
May 26 16:11:50.770740 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:11:50.770711    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-sklearn-graph-raw-hpa-89f63-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/43de9a6e-da72-41b0-97ca-0e4a7dadb2f8-isvc-sklearn-graph-raw-hpa-89f63-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-89f63-predictor-564cbb96cc-ct4hd\" (UID: \"43de9a6e-da72-41b0-97ca-0e4a7dadb2f8\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-89f63-predictor-564cbb96cc-ct4hd"
May 26 16:11:50.772600 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:11:50.772575    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/43de9a6e-da72-41b0-97ca-0e4a7dadb2f8-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-89f63-predictor-564cbb96cc-ct4hd\" (UID: \"43de9a6e-da72-41b0-97ca-0e4a7dadb2f8\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-89f63-predictor-564cbb96cc-ct4hd"
May 26 16:11:50.780082 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:11:50.780049    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tw84s\" (UniqueName: \"kubernetes.io/projected/43de9a6e-da72-41b0-97ca-0e4a7dadb2f8-kube-api-access-tw84s\") pod \"isvc-sklearn-graph-raw-hpa-89f63-predictor-564cbb96cc-ct4hd\" (UID: \"43de9a6e-da72-41b0-97ca-0e4a7dadb2f8\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-89f63-predictor-564cbb96cc-ct4hd"
May 26 16:11:50.829172 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:11:50.829076    2556 generic.go:358] "Generic (PLEG): container finished" podID="c1dbdd50-e6c5-4ac1-b33e-56f276cac3f6" containerID="43fcd1acdd049e5c9343b274f3bd5efea93c36fd17495467016d21a84bc95dd3" exitCode=2
May 26 16:11:50.829285 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:11:50.829166    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-5c32b-predictor-86f64c4b8f-7mvh8" event={"ID":"c1dbdd50-e6c5-4ac1-b33e-56f276cac3f6","Type":"ContainerDied","Data":"43fcd1acdd049e5c9343b274f3bd5efea93c36fd17495467016d21a84bc95dd3"}
May 26 16:11:50.901744 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:11:50.901716    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-89f63-predictor-564cbb96cc-ct4hd"
May 26 16:11:51.038024 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:11:51.037992    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-89f63-predictor-564cbb96cc-ct4hd"]
May 26 16:11:51.040725 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:11:51.040696    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43de9a6e_da72_41b0_97ca_0e4a7dadb2f8.slice/crio-cfef28f054c087f9d1c018ba3b951fdd85ccdc024006ce5d930f17c68784b0b6 WatchSource:0}: Error finding container cfef28f054c087f9d1c018ba3b951fdd85ccdc024006ce5d930f17c68784b0b6: Status 404 returned error can't find the container with id cfef28f054c087f9d1c018ba3b951fdd85ccdc024006ce5d930f17c68784b0b6
May 26 16:11:51.042489 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:11:51.042473    2556 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 26 16:11:51.836350 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:11:51.836315    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-89f63-predictor-564cbb96cc-ct4hd" event={"ID":"43de9a6e-da72-41b0-97ca-0e4a7dadb2f8","Type":"ContainerStarted","Data":"28f75233d4b531822faa49fc0e96cf1e4dee917817915638df74ee68b3bcd085"}
May 26 16:11:51.836350 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:11:51.836353    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-89f63-predictor-564cbb96cc-ct4hd" event={"ID":"43de9a6e-da72-41b0-97ca-0e4a7dadb2f8","Type":"ContainerStarted","Data":"cfef28f054c087f9d1c018ba3b951fdd85ccdc024006ce5d930f17c68784b0b6"}
May 26 16:11:52.542615 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:11:52.542567    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-5c32b-predictor-86f64c4b8f-7mvh8" podUID="c1dbdd50-e6c5-4ac1-b33e-56f276cac3f6" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.28:8643/healthz\": dial tcp 10.133.0.28:8643: connect: connection refused"
May 26 16:11:52.546998 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:11:52.546972    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-5c32b-predictor-86f64c4b8f-7mvh8" podUID="c1dbdd50-e6c5-4ac1-b33e-56f276cac3f6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
May 26 16:11:54.697767 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:11:54.697744    2556 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-5c32b-predictor-86f64c4b8f-7mvh8"
May 26 16:11:54.803409 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:11:54.803343    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gmsrc\" (UniqueName: \"kubernetes.io/projected/c1dbdd50-e6c5-4ac1-b33e-56f276cac3f6-kube-api-access-gmsrc\") pod \"c1dbdd50-e6c5-4ac1-b33e-56f276cac3f6\" (UID: \"c1dbdd50-e6c5-4ac1-b33e-56f276cac3f6\") "
May 26 16:11:54.803530 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:11:54.803416    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c1dbdd50-e6c5-4ac1-b33e-56f276cac3f6-kserve-provision-location\") pod \"c1dbdd50-e6c5-4ac1-b33e-56f276cac3f6\" (UID: \"c1dbdd50-e6c5-4ac1-b33e-56f276cac3f6\") "
May 26 16:11:54.803530 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:11:54.803488    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c1dbdd50-e6c5-4ac1-b33e-56f276cac3f6-proxy-tls\") pod \"c1dbdd50-e6c5-4ac1-b33e-56f276cac3f6\" (UID: \"c1dbdd50-e6c5-4ac1-b33e-56f276cac3f6\") "
May 26 16:11:54.803530 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:11:54.803515    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-sklearn-graph-raw-5c32b-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c1dbdd50-e6c5-4ac1-b33e-56f276cac3f6-isvc-sklearn-graph-raw-5c32b-kube-rbac-proxy-sar-config\") pod \"c1dbdd50-e6c5-4ac1-b33e-56f276cac3f6\" (UID: \"c1dbdd50-e6c5-4ac1-b33e-56f276cac3f6\") "
May 26 16:11:54.803827 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:11:54.803751    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1dbdd50-e6c5-4ac1-b33e-56f276cac3f6-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c1dbdd50-e6c5-4ac1-b33e-56f276cac3f6" (UID: "c1dbdd50-e6c5-4ac1-b33e-56f276cac3f6"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 16:11:54.803892 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:11:54.803875    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1dbdd50-e6c5-4ac1-b33e-56f276cac3f6-isvc-sklearn-graph-raw-5c32b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-sklearn-graph-raw-5c32b-kube-rbac-proxy-sar-config") pod "c1dbdd50-e6c5-4ac1-b33e-56f276cac3f6" (UID: "c1dbdd50-e6c5-4ac1-b33e-56f276cac3f6"). InnerVolumeSpecName "isvc-sklearn-graph-raw-5c32b-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 16:11:54.805497 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:11:54.805472    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1dbdd50-e6c5-4ac1-b33e-56f276cac3f6-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c1dbdd50-e6c5-4ac1-b33e-56f276cac3f6" (UID: "c1dbdd50-e6c5-4ac1-b33e-56f276cac3f6"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 16:11:54.805551 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:11:54.805497    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1dbdd50-e6c5-4ac1-b33e-56f276cac3f6-kube-api-access-gmsrc" (OuterVolumeSpecName: "kube-api-access-gmsrc") pod "c1dbdd50-e6c5-4ac1-b33e-56f276cac3f6" (UID: "c1dbdd50-e6c5-4ac1-b33e-56f276cac3f6"). InnerVolumeSpecName "kube-api-access-gmsrc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 16:11:54.846676 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:11:54.846649    2556 generic.go:358] "Generic (PLEG): container finished" podID="c1dbdd50-e6c5-4ac1-b33e-56f276cac3f6" containerID="f358beb3ca5cb1c1129d1e80fbe4272bc19831d877e2d3d347eea046e4845fc2" exitCode=0
May 26 16:11:54.846784 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:11:54.846728    2556 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-5c32b-predictor-86f64c4b8f-7mvh8"
May 26 16:11:54.846784 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:11:54.846728    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-5c32b-predictor-86f64c4b8f-7mvh8" event={"ID":"c1dbdd50-e6c5-4ac1-b33e-56f276cac3f6","Type":"ContainerDied","Data":"f358beb3ca5cb1c1129d1e80fbe4272bc19831d877e2d3d347eea046e4845fc2"}
May 26 16:11:54.846784 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:11:54.846766    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-5c32b-predictor-86f64c4b8f-7mvh8" event={"ID":"c1dbdd50-e6c5-4ac1-b33e-56f276cac3f6","Type":"ContainerDied","Data":"6adbcd5bc50d69f21ed61f554fcfe0382efe10a46dcee743fdaaf37d897d9ece"}
May 26 16:11:54.846923 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:11:54.846786    2556 scope.go:117] "RemoveContainer" containerID="43fcd1acdd049e5c9343b274f3bd5efea93c36fd17495467016d21a84bc95dd3"
May 26 16:11:54.848141 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:11:54.848119    2556 generic.go:358] "Generic (PLEG): container finished" podID="43de9a6e-da72-41b0-97ca-0e4a7dadb2f8" containerID="28f75233d4b531822faa49fc0e96cf1e4dee917817915638df74ee68b3bcd085" exitCode=0
May 26 16:11:54.848245 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:11:54.848177    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-89f63-predictor-564cbb96cc-ct4hd" event={"ID":"43de9a6e-da72-41b0-97ca-0e4a7dadb2f8","Type":"ContainerDied","Data":"28f75233d4b531822faa49fc0e96cf1e4dee917817915638df74ee68b3bcd085"}
May 26 16:11:54.855305 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:11:54.855167    2556 scope.go:117] "RemoveContainer" containerID="f358beb3ca5cb1c1129d1e80fbe4272bc19831d877e2d3d347eea046e4845fc2"
May 26 16:11:54.862997 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:11:54.862917    2556 scope.go:117] "RemoveContainer" containerID="b7ddd4ffdd91cc866f5b9cae692ab228cc2dccd5748038a734134c879576870d"
May 26 16:11:54.869739 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:11:54.869717    2556 scope.go:117] "RemoveContainer" containerID="43fcd1acdd049e5c9343b274f3bd5efea93c36fd17495467016d21a84bc95dd3"
May 26 16:11:54.869987 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:11:54.869966    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43fcd1acdd049e5c9343b274f3bd5efea93c36fd17495467016d21a84bc95dd3\": container with ID starting with 43fcd1acdd049e5c9343b274f3bd5efea93c36fd17495467016d21a84bc95dd3 not found: ID does not exist" containerID="43fcd1acdd049e5c9343b274f3bd5efea93c36fd17495467016d21a84bc95dd3"
May 26 16:11:54.870070 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:11:54.869994    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43fcd1acdd049e5c9343b274f3bd5efea93c36fd17495467016d21a84bc95dd3"} err="failed to get container status \"43fcd1acdd049e5c9343b274f3bd5efea93c36fd17495467016d21a84bc95dd3\": rpc error: code = NotFound desc = could not find container \"43fcd1acdd049e5c9343b274f3bd5efea93c36fd17495467016d21a84bc95dd3\": container with ID starting with 43fcd1acdd049e5c9343b274f3bd5efea93c36fd17495467016d21a84bc95dd3 not found: ID does not exist"
May 26 16:11:54.870070 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:11:54.870010    2556 scope.go:117] "RemoveContainer" containerID="f358beb3ca5cb1c1129d1e80fbe4272bc19831d877e2d3d347eea046e4845fc2"
May 26 16:11:54.870270 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:11:54.870253    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f358beb3ca5cb1c1129d1e80fbe4272bc19831d877e2d3d347eea046e4845fc2\": container with ID starting with f358beb3ca5cb1c1129d1e80fbe4272bc19831d877e2d3d347eea046e4845fc2 not found: ID does not exist" containerID="f358beb3ca5cb1c1129d1e80fbe4272bc19831d877e2d3d347eea046e4845fc2"
May 26 16:11:54.870317 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:11:54.870276    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f358beb3ca5cb1c1129d1e80fbe4272bc19831d877e2d3d347eea046e4845fc2"} err="failed to get container status \"f358beb3ca5cb1c1129d1e80fbe4272bc19831d877e2d3d347eea046e4845fc2\": rpc error: code = NotFound desc = could not find container \"f358beb3ca5cb1c1129d1e80fbe4272bc19831d877e2d3d347eea046e4845fc2\": container with ID starting with f358beb3ca5cb1c1129d1e80fbe4272bc19831d877e2d3d347eea046e4845fc2 not found: ID does not exist"
May 26 16:11:54.870317 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:11:54.870294    2556 scope.go:117] "RemoveContainer" containerID="b7ddd4ffdd91cc866f5b9cae692ab228cc2dccd5748038a734134c879576870d"
May 26 16:11:54.870527 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:11:54.870512    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7ddd4ffdd91cc866f5b9cae692ab228cc2dccd5748038a734134c879576870d\": container with ID starting with b7ddd4ffdd91cc866f5b9cae692ab228cc2dccd5748038a734134c879576870d not found: ID does not exist" containerID="b7ddd4ffdd91cc866f5b9cae692ab228cc2dccd5748038a734134c879576870d"
May 26 16:11:54.870567 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:11:54.870530    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7ddd4ffdd91cc866f5b9cae692ab228cc2dccd5748038a734134c879576870d"} err="failed to get container status \"b7ddd4ffdd91cc866f5b9cae692ab228cc2dccd5748038a734134c879576870d\": rpc error: code = NotFound desc = could not find container \"b7ddd4ffdd91cc866f5b9cae692ab228cc2dccd5748038a734134c879576870d\": container with ID starting with b7ddd4ffdd91cc866f5b9cae692ab228cc2dccd5748038a734134c879576870d not found: ID does not exist"
May 26 16:11:54.899611 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:11:54.899589    2556 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-5c32b-predictor-86f64c4b8f-7mvh8"]
May 26 16:11:54.904338 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:11:54.904319    2556 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c1dbdd50-e6c5-4ac1-b33e-56f276cac3f6-proxy-tls\") on node \"ip-10-0-134-19.ec2.internal\" DevicePath \"\""
May 26 16:11:54.904416 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:11:54.904340    2556 reconciler_common.go:299] "Volume detached for volume \"isvc-sklearn-graph-raw-5c32b-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c1dbdd50-e6c5-4ac1-b33e-56f276cac3f6-isvc-sklearn-graph-raw-5c32b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-19.ec2.internal\" DevicePath \"\""
May 26 16:11:54.904416 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:11:54.904342    2556 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-5c32b-predictor-86f64c4b8f-7mvh8"]
May 26 16:11:54.904416 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:11:54.904352    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gmsrc\" (UniqueName: \"kubernetes.io/projected/c1dbdd50-e6c5-4ac1-b33e-56f276cac3f6-kube-api-access-gmsrc\") on node \"ip-10-0-134-19.ec2.internal\" DevicePath \"\""
May 26 16:11:54.904416 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:11:54.904363    2556 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c1dbdd50-e6c5-4ac1-b33e-56f276cac3f6-kserve-provision-location\") on node \"ip-10-0-134-19.ec2.internal\" DevicePath \"\""
May 26 16:11:55.853305 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:11:55.853271    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-89f63-predictor-564cbb96cc-ct4hd" event={"ID":"43de9a6e-da72-41b0-97ca-0e4a7dadb2f8","Type":"ContainerStarted","Data":"1a1428052135111fa97b751ee173646857abf027cc07f11973056b052d11b93c"}
May 26 16:11:55.853782 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:11:55.853312    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-89f63-predictor-564cbb96cc-ct4hd" event={"ID":"43de9a6e-da72-41b0-97ca-0e4a7dadb2f8","Type":"ContainerStarted","Data":"75f107f9b674235ece1f082976ae5b0f5de746b96f76cc032e11df747f140127"}
May 26 16:11:55.853782 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:11:55.853638    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-89f63-predictor-564cbb96cc-ct4hd"
May 26 16:11:55.853782 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:11:55.853775    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-89f63-predictor-564cbb96cc-ct4hd"
May 26 16:11:55.855076 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:11:55.855047    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-89f63-predictor-564cbb96cc-ct4hd" podUID="43de9a6e-da72-41b0-97ca-0e4a7dadb2f8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
May 26 16:11:55.880617 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:11:55.880567    2556 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-89f63-predictor-564cbb96cc-ct4hd" podStartSLOduration=5.880553918 podStartE2EDuration="5.880553918s" podCreationTimestamp="2026-05-26 16:11:50 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 16:11:55.879611576 +0000 UTC m=+677.724228579" watchObservedRunningTime="2026-05-26 16:11:55.880553918 +0000 UTC m=+677.725170907"
May 26 16:11:56.771471 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:11:56.771439    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c1dbdd50-e6c5-4ac1-b33e-56f276cac3f6" path="/var/lib/kubelet/pods/c1dbdd50-e6c5-4ac1-b33e-56f276cac3f6/volumes"
May 26 16:11:56.857267 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:11:56.857224    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-89f63-predictor-564cbb96cc-ct4hd" podUID="43de9a6e-da72-41b0-97ca-0e4a7dadb2f8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
May 26 16:12:01.861847 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:12:01.861814    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-89f63-predictor-564cbb96cc-ct4hd"
May 26 16:12:01.862464 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:12:01.862432    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-89f63-predictor-564cbb96cc-ct4hd" podUID="43de9a6e-da72-41b0-97ca-0e4a7dadb2f8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
May 26 16:12:11.863169 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:12:11.863133    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-89f63-predictor-564cbb96cc-ct4hd" podUID="43de9a6e-da72-41b0-97ca-0e4a7dadb2f8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
May 26 16:12:21.862667 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:12:21.862628    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-89f63-predictor-564cbb96cc-ct4hd" podUID="43de9a6e-da72-41b0-97ca-0e4a7dadb2f8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
May 26 16:12:31.862390 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:12:31.862304    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-89f63-predictor-564cbb96cc-ct4hd" podUID="43de9a6e-da72-41b0-97ca-0e4a7dadb2f8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
May 26 16:12:41.862461 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:12:41.862422    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-89f63-predictor-564cbb96cc-ct4hd" podUID="43de9a6e-da72-41b0-97ca-0e4a7dadb2f8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
May 26 16:12:51.863059 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:12:51.863016    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-89f63-predictor-564cbb96cc-ct4hd" podUID="43de9a6e-da72-41b0-97ca-0e4a7dadb2f8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
May 26 16:13:01.862988 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:13:01.862952    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-89f63-predictor-564cbb96cc-ct4hd"
May 26 16:13:30.788389 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:13:30.788359    2556 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-89f63-predictor-564cbb96cc-ct4hd"]
May 26 16:13:30.788906 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:13:30.788653    2556 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-89f63-predictor-564cbb96cc-ct4hd" podUID="43de9a6e-da72-41b0-97ca-0e4a7dadb2f8" containerName="kserve-container" containerID="cri-o://75f107f9b674235ece1f082976ae5b0f5de746b96f76cc032e11df747f140127" gracePeriod=30
May 26 16:13:30.788906 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:13:30.788673    2556 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-89f63-predictor-564cbb96cc-ct4hd" podUID="43de9a6e-da72-41b0-97ca-0e4a7dadb2f8" containerName="kube-rbac-proxy" containerID="cri-o://1a1428052135111fa97b751ee173646857abf027cc07f11973056b052d11b93c" gracePeriod=30
May 26 16:13:30.861331 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:13:30.861304    2556 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-4b426-predictor-75c485b4b4-zwcxz"]
May 26 16:13:30.861603 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:13:30.861592    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1dbdd50-e6c5-4ac1-b33e-56f276cac3f6" containerName="kserve-container"
May 26 16:13:30.861646 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:13:30.861605    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1dbdd50-e6c5-4ac1-b33e-56f276cac3f6" containerName="kserve-container"
May 26 16:13:30.861646 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:13:30.861621    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1dbdd50-e6c5-4ac1-b33e-56f276cac3f6" containerName="storage-initializer"
May 26 16:13:30.861646 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:13:30.861626    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1dbdd50-e6c5-4ac1-b33e-56f276cac3f6" containerName="storage-initializer"
May 26 16:13:30.861646 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:13:30.861636    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1dbdd50-e6c5-4ac1-b33e-56f276cac3f6" containerName="kube-rbac-proxy"
May 26 16:13:30.861646 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:13:30.861642    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1dbdd50-e6c5-4ac1-b33e-56f276cac3f6" containerName="kube-rbac-proxy"
May 26 16:13:30.861794 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:13:30.861690    2556 memory_manager.go:356] "RemoveStaleState removing state" podUID="c1dbdd50-e6c5-4ac1-b33e-56f276cac3f6" containerName="kube-rbac-proxy"
May 26 16:13:30.861794 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:13:30.861697    2556 memory_manager.go:356] "RemoveStaleState removing state" podUID="c1dbdd50-e6c5-4ac1-b33e-56f276cac3f6" containerName="kserve-container"
May 26 16:13:30.864501 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:13:30.864485    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-4b426-predictor-75c485b4b4-zwcxz"
May 26 16:13:30.867259 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:13:30.867235    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-4b426-predictor-serving-cert\""
May 26 16:13:30.867355 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:13:30.867261    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-4b426-kube-rbac-proxy-sar-config\""
May 26 16:13:30.875767 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:13:30.875743    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-4b426-predictor-75c485b4b4-zwcxz"]
May 26 16:13:31.026774 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:13:31.026749    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/95eb1efb-8eec-4333-b8b3-0dd730ea8532-proxy-tls\") pod \"message-dumper-raw-4b426-predictor-75c485b4b4-zwcxz\" (UID: \"95eb1efb-8eec-4333-b8b3-0dd730ea8532\") " pod="kserve-ci-e2e-test/message-dumper-raw-4b426-predictor-75c485b4b4-zwcxz"
May 26 16:13:31.026877 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:13:31.026778    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64hhv\" (UniqueName: \"kubernetes.io/projected/95eb1efb-8eec-4333-b8b3-0dd730ea8532-kube-api-access-64hhv\") pod \"message-dumper-raw-4b426-predictor-75c485b4b4-zwcxz\" (UID: \"95eb1efb-8eec-4333-b8b3-0dd730ea8532\") " pod="kserve-ci-e2e-test/message-dumper-raw-4b426-predictor-75c485b4b4-zwcxz"
May 26 16:13:31.026877 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:13:31.026817    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"message-dumper-raw-4b426-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/95eb1efb-8eec-4333-b8b3-0dd730ea8532-message-dumper-raw-4b426-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-4b426-predictor-75c485b4b4-zwcxz\" (UID: \"95eb1efb-8eec-4333-b8b3-0dd730ea8532\") " pod="kserve-ci-e2e-test/message-dumper-raw-4b426-predictor-75c485b4b4-zwcxz"
May 26 16:13:31.127580 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:13:31.127556    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/95eb1efb-8eec-4333-b8b3-0dd730ea8532-proxy-tls\") pod \"message-dumper-raw-4b426-predictor-75c485b4b4-zwcxz\" (UID: \"95eb1efb-8eec-4333-b8b3-0dd730ea8532\") " pod="kserve-ci-e2e-test/message-dumper-raw-4b426-predictor-75c485b4b4-zwcxz"
May 26 16:13:31.127710 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:13:31.127587    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-64hhv\" (UniqueName: \"kubernetes.io/projected/95eb1efb-8eec-4333-b8b3-0dd730ea8532-kube-api-access-64hhv\") pod \"message-dumper-raw-4b426-predictor-75c485b4b4-zwcxz\" (UID: \"95eb1efb-8eec-4333-b8b3-0dd730ea8532\") " pod="kserve-ci-e2e-test/message-dumper-raw-4b426-predictor-75c485b4b4-zwcxz"
May 26 16:13:31.127710 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:13:31.127619    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"message-dumper-raw-4b426-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/95eb1efb-8eec-4333-b8b3-0dd730ea8532-message-dumper-raw-4b426-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-4b426-predictor-75c485b4b4-zwcxz\" (UID: \"95eb1efb-8eec-4333-b8b3-0dd730ea8532\") " pod="kserve-ci-e2e-test/message-dumper-raw-4b426-predictor-75c485b4b4-zwcxz"
May 26 16:13:31.128238 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:13:31.128219    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"message-dumper-raw-4b426-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/95eb1efb-8eec-4333-b8b3-0dd730ea8532-message-dumper-raw-4b426-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-4b426-predictor-75c485b4b4-zwcxz\" (UID: \"95eb1efb-8eec-4333-b8b3-0dd730ea8532\") " pod="kserve-ci-e2e-test/message-dumper-raw-4b426-predictor-75c485b4b4-zwcxz"
May 26 16:13:31.129982 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:13:31.129959    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/95eb1efb-8eec-4333-b8b3-0dd730ea8532-proxy-tls\") pod \"message-dumper-raw-4b426-predictor-75c485b4b4-zwcxz\" (UID: \"95eb1efb-8eec-4333-b8b3-0dd730ea8532\") " pod="kserve-ci-e2e-test/message-dumper-raw-4b426-predictor-75c485b4b4-zwcxz"
May 26 16:13:31.138404 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:13:31.138381    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64hhv\" (UniqueName: \"kubernetes.io/projected/95eb1efb-8eec-4333-b8b3-0dd730ea8532-kube-api-access-64hhv\") pod \"message-dumper-raw-4b426-predictor-75c485b4b4-zwcxz\" (UID: \"95eb1efb-8eec-4333-b8b3-0dd730ea8532\") " pod="kserve-ci-e2e-test/message-dumper-raw-4b426-predictor-75c485b4b4-zwcxz"
May 26 16:13:31.149269 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:13:31.149245    2556 generic.go:358] "Generic (PLEG): container finished" podID="43de9a6e-da72-41b0-97ca-0e4a7dadb2f8" containerID="1a1428052135111fa97b751ee173646857abf027cc07f11973056b052d11b93c" exitCode=2
May 26 16:13:31.149353 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:13:31.149292    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-89f63-predictor-564cbb96cc-ct4hd" event={"ID":"43de9a6e-da72-41b0-97ca-0e4a7dadb2f8","Type":"ContainerDied","Data":"1a1428052135111fa97b751ee173646857abf027cc07f11973056b052d11b93c"}
May 26 16:13:31.175440 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:13:31.175418    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-4b426-predictor-75c485b4b4-zwcxz"
May 26 16:13:31.301189 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:13:31.301124    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-4b426-predictor-75c485b4b4-zwcxz"]
May 26 16:13:31.303135 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:13:31.303108    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95eb1efb_8eec_4333_b8b3_0dd730ea8532.slice/crio-b8cc8da261db6c8aa128ee3956b16a15be8ae2483bd7ccb2eb8e2a943563bc34 WatchSource:0}: Error finding container b8cc8da261db6c8aa128ee3956b16a15be8ae2483bd7ccb2eb8e2a943563bc34: Status 404 returned error can't find the container with id b8cc8da261db6c8aa128ee3956b16a15be8ae2483bd7ccb2eb8e2a943563bc34
May 26 16:13:31.857679 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:13:31.857642    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-89f63-predictor-564cbb96cc-ct4hd" podUID="43de9a6e-da72-41b0-97ca-0e4a7dadb2f8" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.29:8643/healthz\": dial tcp 10.133.0.29:8643: connect: connection refused"
May 26 16:13:31.862734 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:13:31.862709    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-89f63-predictor-564cbb96cc-ct4hd" podUID="43de9a6e-da72-41b0-97ca-0e4a7dadb2f8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
May 26 16:13:32.154417 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:13:32.154386    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-4b426-predictor-75c485b4b4-zwcxz" event={"ID":"95eb1efb-8eec-4333-b8b3-0dd730ea8532","Type":"ContainerStarted","Data":"b8cc8da261db6c8aa128ee3956b16a15be8ae2483bd7ccb2eb8e2a943563bc34"}
May 26 16:13:33.159331 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:13:33.159298    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-4b426-predictor-75c485b4b4-zwcxz" event={"ID":"95eb1efb-8eec-4333-b8b3-0dd730ea8532","Type":"ContainerStarted","Data":"4395b85a95c4537fd40f825af83ba07dd85cbf690ff5d17de989c7b5d5919712"}
May 26 16:13:33.159331 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:13:33.159332    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-4b426-predictor-75c485b4b4-zwcxz" event={"ID":"95eb1efb-8eec-4333-b8b3-0dd730ea8532","Type":"ContainerStarted","Data":"8be754488cc528859b681547305adb20397850d310ed5358988808fb364013fc"}
May 26 16:13:33.159736 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:13:33.159418    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-4b426-predictor-75c485b4b4-zwcxz"
May 26 16:13:33.182468 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:13:33.182419    2556 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/message-dumper-raw-4b426-predictor-75c485b4b4-zwcxz" podStartSLOduration=2.190112672 podStartE2EDuration="3.1824021s" podCreationTimestamp="2026-05-26 16:13:30 +0000 UTC" firstStartedPulling="2026-05-26 16:13:31.304824871 +0000 UTC m=+773.149441842" lastFinishedPulling="2026-05-26 16:13:32.297114297 +0000 UTC m=+774.141731270" observedRunningTime="2026-05-26 16:13:33.180210942 +0000 UTC m=+775.024827934" watchObservedRunningTime="2026-05-26 16:13:33.1824021 +0000 UTC m=+775.027019092"
May 26 16:13:34.163207 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:13:34.163177    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-4b426-predictor-75c485b4b4-zwcxz"
May 26 16:13:34.164775 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:13:34.164752    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-4b426-predictor-75c485b4b4-zwcxz"
May 26 16:13:34.630940 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:13:34.630919    2556 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-89f63-predictor-564cbb96cc-ct4hd"
May 26 16:13:34.755380 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:13:34.755309    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tw84s\" (UniqueName: \"kubernetes.io/projected/43de9a6e-da72-41b0-97ca-0e4a7dadb2f8-kube-api-access-tw84s\") pod \"43de9a6e-da72-41b0-97ca-0e4a7dadb2f8\" (UID: \"43de9a6e-da72-41b0-97ca-0e4a7dadb2f8\") "
May 26 16:13:34.755380 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:13:34.755346    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/43de9a6e-da72-41b0-97ca-0e4a7dadb2f8-proxy-tls\") pod \"43de9a6e-da72-41b0-97ca-0e4a7dadb2f8\" (UID: \"43de9a6e-da72-41b0-97ca-0e4a7dadb2f8\") "
May 26 16:13:34.755380 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:13:34.755375    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/43de9a6e-da72-41b0-97ca-0e4a7dadb2f8-kserve-provision-location\") pod \"43de9a6e-da72-41b0-97ca-0e4a7dadb2f8\" (UID: \"43de9a6e-da72-41b0-97ca-0e4a7dadb2f8\") "
May 26 16:13:34.755636 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:13:34.755434    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-sklearn-graph-raw-hpa-89f63-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/43de9a6e-da72-41b0-97ca-0e4a7dadb2f8-isvc-sklearn-graph-raw-hpa-89f63-kube-rbac-proxy-sar-config\") pod \"43de9a6e-da72-41b0-97ca-0e4a7dadb2f8\" (UID: \"43de9a6e-da72-41b0-97ca-0e4a7dadb2f8\") "
May 26 16:13:34.755745 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:13:34.755717    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43de9a6e-da72-41b0-97ca-0e4a7dadb2f8-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "43de9a6e-da72-41b0-97ca-0e4a7dadb2f8" (UID: "43de9a6e-da72-41b0-97ca-0e4a7dadb2f8"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 16:13:34.755791 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:13:34.755769    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43de9a6e-da72-41b0-97ca-0e4a7dadb2f8-isvc-sklearn-graph-raw-hpa-89f63-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-sklearn-graph-raw-hpa-89f63-kube-rbac-proxy-sar-config") pod "43de9a6e-da72-41b0-97ca-0e4a7dadb2f8" (UID: "43de9a6e-da72-41b0-97ca-0e4a7dadb2f8"). InnerVolumeSpecName "isvc-sklearn-graph-raw-hpa-89f63-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 16:13:34.757536 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:13:34.757502    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43de9a6e-da72-41b0-97ca-0e4a7dadb2f8-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "43de9a6e-da72-41b0-97ca-0e4a7dadb2f8" (UID: "43de9a6e-da72-41b0-97ca-0e4a7dadb2f8"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 16:13:34.757536 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:13:34.757517    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43de9a6e-da72-41b0-97ca-0e4a7dadb2f8-kube-api-access-tw84s" (OuterVolumeSpecName: "kube-api-access-tw84s") pod "43de9a6e-da72-41b0-97ca-0e4a7dadb2f8" (UID: "43de9a6e-da72-41b0-97ca-0e4a7dadb2f8"). InnerVolumeSpecName "kube-api-access-tw84s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 16:13:34.856406 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:13:34.856381    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tw84s\" (UniqueName: \"kubernetes.io/projected/43de9a6e-da72-41b0-97ca-0e4a7dadb2f8-kube-api-access-tw84s\") on node \"ip-10-0-134-19.ec2.internal\" DevicePath \"\""
May 26 16:13:34.856406 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:13:34.856405    2556 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/43de9a6e-da72-41b0-97ca-0e4a7dadb2f8-proxy-tls\") on node \"ip-10-0-134-19.ec2.internal\" DevicePath \"\""
May 26 16:13:34.856531 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:13:34.856415    2556 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/43de9a6e-da72-41b0-97ca-0e4a7dadb2f8-kserve-provision-location\") on node \"ip-10-0-134-19.ec2.internal\" DevicePath \"\""
May 26 16:13:34.856531 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:13:34.856425    2556 reconciler_common.go:299] "Volume detached for volume \"isvc-sklearn-graph-raw-hpa-89f63-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/43de9a6e-da72-41b0-97ca-0e4a7dadb2f8-isvc-sklearn-graph-raw-hpa-89f63-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-19.ec2.internal\" DevicePath \"\""
May 26 16:13:35.167140 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:13:35.167109    2556 generic.go:358] "Generic (PLEG): container finished" podID="43de9a6e-da72-41b0-97ca-0e4a7dadb2f8" containerID="75f107f9b674235ece1f082976ae5b0f5de746b96f76cc032e11df747f140127" exitCode=0
May 26 16:13:35.167480 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:13:35.167191    2556 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-89f63-predictor-564cbb96cc-ct4hd"
May 26 16:13:35.167480 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:13:35.167183    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-89f63-predictor-564cbb96cc-ct4hd" event={"ID":"43de9a6e-da72-41b0-97ca-0e4a7dadb2f8","Type":"ContainerDied","Data":"75f107f9b674235ece1f082976ae5b0f5de746b96f76cc032e11df747f140127"}
May 26 16:13:35.167480 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:13:35.167301    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-89f63-predictor-564cbb96cc-ct4hd" event={"ID":"43de9a6e-da72-41b0-97ca-0e4a7dadb2f8","Type":"ContainerDied","Data":"cfef28f054c087f9d1c018ba3b951fdd85ccdc024006ce5d930f17c68784b0b6"}
May 26 16:13:35.167480 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:13:35.167323    2556 scope.go:117] "RemoveContainer" containerID="1a1428052135111fa97b751ee173646857abf027cc07f11973056b052d11b93c"
May 26 16:13:35.174974 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:13:35.174958    2556 scope.go:117] "RemoveContainer" containerID="75f107f9b674235ece1f082976ae5b0f5de746b96f76cc032e11df747f140127"
May 26 16:13:35.182157 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:13:35.182141    2556 scope.go:117] "RemoveContainer" containerID="28f75233d4b531822faa49fc0e96cf1e4dee917817915638df74ee68b3bcd085"
May 26 16:13:35.188640 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:13:35.188626    2556 scope.go:117] "RemoveContainer" containerID="1a1428052135111fa97b751ee173646857abf027cc07f11973056b052d11b93c"
May 26 16:13:35.188913 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:13:35.188896    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a1428052135111fa97b751ee173646857abf027cc07f11973056b052d11b93c\": container with ID starting with 1a1428052135111fa97b751ee173646857abf027cc07f11973056b052d11b93c not found: ID does not exist" containerID="1a1428052135111fa97b751ee173646857abf027cc07f11973056b052d11b93c"
May 26 16:13:35.188960 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:13:35.188921    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a1428052135111fa97b751ee173646857abf027cc07f11973056b052d11b93c"} err="failed to get container status \"1a1428052135111fa97b751ee173646857abf027cc07f11973056b052d11b93c\": rpc error: code = NotFound desc = could not find container \"1a1428052135111fa97b751ee173646857abf027cc07f11973056b052d11b93c\": container with ID starting with 1a1428052135111fa97b751ee173646857abf027cc07f11973056b052d11b93c not found: ID does not exist"
May 26 16:13:35.188960 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:13:35.188937    2556 scope.go:117] "RemoveContainer" containerID="75f107f9b674235ece1f082976ae5b0f5de746b96f76cc032e11df747f140127"
May 26 16:13:35.189156 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:13:35.189135    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75f107f9b674235ece1f082976ae5b0f5de746b96f76cc032e11df747f140127\": container with ID starting with 75f107f9b674235ece1f082976ae5b0f5de746b96f76cc032e11df747f140127 not found: ID does not exist" containerID="75f107f9b674235ece1f082976ae5b0f5de746b96f76cc032e11df747f140127"
May 26 16:13:35.189208 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:13:35.189163    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75f107f9b674235ece1f082976ae5b0f5de746b96f76cc032e11df747f140127"} err="failed to get container status \"75f107f9b674235ece1f082976ae5b0f5de746b96f76cc032e11df747f140127\": rpc error: code = NotFound desc = could not find container \"75f107f9b674235ece1f082976ae5b0f5de746b96f76cc032e11df747f140127\": container with ID starting with 75f107f9b674235ece1f082976ae5b0f5de746b96f76cc032e11df747f140127 not found: ID does not exist"
May 26 16:13:35.189208 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:13:35.189181    2556 scope.go:117] "RemoveContainer" containerID="28f75233d4b531822faa49fc0e96cf1e4dee917817915638df74ee68b3bcd085"
May 26 16:13:35.189380 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:13:35.189361    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28f75233d4b531822faa49fc0e96cf1e4dee917817915638df74ee68b3bcd085\": container with ID starting with 28f75233d4b531822faa49fc0e96cf1e4dee917817915638df74ee68b3bcd085 not found: ID does not exist" containerID="28f75233d4b531822faa49fc0e96cf1e4dee917817915638df74ee68b3bcd085"
May 26 16:13:35.189421 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:13:35.189384    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28f75233d4b531822faa49fc0e96cf1e4dee917817915638df74ee68b3bcd085"} err="failed to get container status \"28f75233d4b531822faa49fc0e96cf1e4dee917817915638df74ee68b3bcd085\": rpc error: code = NotFound desc = could not find container \"28f75233d4b531822faa49fc0e96cf1e4dee917817915638df74ee68b3bcd085\": container with ID starting with 28f75233d4b531822faa49fc0e96cf1e4dee917817915638df74ee68b3bcd085 not found: ID does not exist"
May 26 16:13:35.192587 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:13:35.192566    2556 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-89f63-predictor-564cbb96cc-ct4hd"]
May 26 16:13:35.196453 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:13:35.196432    2556 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-89f63-predictor-564cbb96cc-ct4hd"]
May 26 16:13:36.772703 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:13:36.772660    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="43de9a6e-da72-41b0-97ca-0e4a7dadb2f8" path="/var/lib/kubelet/pods/43de9a6e-da72-41b0-97ca-0e4a7dadb2f8/volumes"
May 26 16:13:41.176384 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:13:41.176356    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-4b426-predictor-75c485b4b4-zwcxz"
May 26 16:13:50.938654 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:13:50.938619    2556 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-4b426-predictor-759bcfd9b-5qmg7"]
May 26 16:13:50.939136 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:13:50.939042    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43de9a6e-da72-41b0-97ca-0e4a7dadb2f8" containerName="kube-rbac-proxy"
May 26 16:13:50.939136 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:13:50.939055    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="43de9a6e-da72-41b0-97ca-0e4a7dadb2f8" containerName="kube-rbac-proxy"
May 26 16:13:50.939136 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:13:50.939077    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43de9a6e-da72-41b0-97ca-0e4a7dadb2f8" containerName="storage-initializer"
May 26 16:13:50.939136 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:13:50.939086    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="43de9a6e-da72-41b0-97ca-0e4a7dadb2f8" containerName="storage-initializer"
May 26 16:13:50.939136 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:13:50.939114    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43de9a6e-da72-41b0-97ca-0e4a7dadb2f8" containerName="kserve-container"
May 26 16:13:50.939136 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:13:50.939122    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="43de9a6e-da72-41b0-97ca-0e4a7dadb2f8" containerName="kserve-container"
May 26 16:13:50.939327 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:13:50.939197    2556 memory_manager.go:356] "RemoveStaleState removing state" podUID="43de9a6e-da72-41b0-97ca-0e4a7dadb2f8" containerName="kserve-container"
May 26 16:13:50.939327 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:13:50.939208    2556 memory_manager.go:356] "RemoveStaleState removing state" podUID="43de9a6e-da72-41b0-97ca-0e4a7dadb2f8" containerName="kube-rbac-proxy"
May 26 16:13:50.944162 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:13:50.944143    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-4b426-predictor-759bcfd9b-5qmg7"
May 26 16:13:50.946811 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:13:50.946788    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-4b426-predictor-serving-cert\""
May 26 16:13:50.946939 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:13:50.946794    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-4b426-kube-rbac-proxy-sar-config\""
May 26 16:13:50.959764 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:13:50.959722    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-4b426-predictor-759bcfd9b-5qmg7"]
May 26 16:13:50.963459 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:13:50.963421    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqbhn\" (UniqueName: \"kubernetes.io/projected/a4bf19f8-f39c-4e8b-b38e-f26a45667f3c-kube-api-access-qqbhn\") pod \"isvc-logger-raw-4b426-predictor-759bcfd9b-5qmg7\" (UID: \"a4bf19f8-f39c-4e8b-b38e-f26a45667f3c\") " pod="kserve-ci-e2e-test/isvc-logger-raw-4b426-predictor-759bcfd9b-5qmg7"
May 26 16:13:50.963576 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:13:50.963475    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-logger-raw-4b426-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a4bf19f8-f39c-4e8b-b38e-f26a45667f3c-isvc-logger-raw-4b426-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-4b426-predictor-759bcfd9b-5qmg7\" (UID: \"a4bf19f8-f39c-4e8b-b38e-f26a45667f3c\") " pod="kserve-ci-e2e-test/isvc-logger-raw-4b426-predictor-759bcfd9b-5qmg7"
May 26 16:13:50.963576 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:13:50.963520    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a4bf19f8-f39c-4e8b-b38e-f26a45667f3c-proxy-tls\") pod \"isvc-logger-raw-4b426-predictor-759bcfd9b-5qmg7\" (UID: \"a4bf19f8-f39c-4e8b-b38e-f26a45667f3c\") " pod="kserve-ci-e2e-test/isvc-logger-raw-4b426-predictor-759bcfd9b-5qmg7"
May 26 16:13:50.963697 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:13:50.963598    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a4bf19f8-f39c-4e8b-b38e-f26a45667f3c-kserve-provision-location\") pod \"isvc-logger-raw-4b426-predictor-759bcfd9b-5qmg7\" (UID: \"a4bf19f8-f39c-4e8b-b38e-f26a45667f3c\") " pod="kserve-ci-e2e-test/isvc-logger-raw-4b426-predictor-759bcfd9b-5qmg7"
May 26 16:13:51.064136 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:13:51.064086    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-logger-raw-4b426-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a4bf19f8-f39c-4e8b-b38e-f26a45667f3c-isvc-logger-raw-4b426-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-4b426-predictor-759bcfd9b-5qmg7\" (UID: \"a4bf19f8-f39c-4e8b-b38e-f26a45667f3c\") " pod="kserve-ci-e2e-test/isvc-logger-raw-4b426-predictor-759bcfd9b-5qmg7"
May 26 16:13:51.064245 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:13:51.064166    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a4bf19f8-f39c-4e8b-b38e-f26a45667f3c-proxy-tls\") pod \"isvc-logger-raw-4b426-predictor-759bcfd9b-5qmg7\" (UID: \"a4bf19f8-f39c-4e8b-b38e-f26a45667f3c\") " pod="kserve-ci-e2e-test/isvc-logger-raw-4b426-predictor-759bcfd9b-5qmg7"
May 26 16:13:51.064245 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:13:51.064218    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a4bf19f8-f39c-4e8b-b38e-f26a45667f3c-kserve-provision-location\") pod \"isvc-logger-raw-4b426-predictor-759bcfd9b-5qmg7\" (UID: \"a4bf19f8-f39c-4e8b-b38e-f26a45667f3c\") " pod="kserve-ci-e2e-test/isvc-logger-raw-4b426-predictor-759bcfd9b-5qmg7"
May 26 16:13:51.064360 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:13:51.064272    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-qqbhn\" (UniqueName: \"kubernetes.io/projected/a4bf19f8-f39c-4e8b-b38e-f26a45667f3c-kube-api-access-qqbhn\") pod \"isvc-logger-raw-4b426-predictor-759bcfd9b-5qmg7\" (UID: \"a4bf19f8-f39c-4e8b-b38e-f26a45667f3c\") " pod="kserve-ci-e2e-test/isvc-logger-raw-4b426-predictor-759bcfd9b-5qmg7"
May 26 16:13:51.064360 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:13:51.064316    2556 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-logger-raw-4b426-predictor-serving-cert: secret "isvc-logger-raw-4b426-predictor-serving-cert" not found
May 26 16:13:51.064465 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:13:51.064383    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a4bf19f8-f39c-4e8b-b38e-f26a45667f3c-proxy-tls podName:a4bf19f8-f39c-4e8b-b38e-f26a45667f3c nodeName:}" failed. No retries permitted until 2026-05-26 16:13:51.564363411 +0000 UTC m=+793.408980386 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/a4bf19f8-f39c-4e8b-b38e-f26a45667f3c-proxy-tls") pod "isvc-logger-raw-4b426-predictor-759bcfd9b-5qmg7" (UID: "a4bf19f8-f39c-4e8b-b38e-f26a45667f3c") : secret "isvc-logger-raw-4b426-predictor-serving-cert" not found
May 26 16:13:51.064592 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:13:51.064571    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a4bf19f8-f39c-4e8b-b38e-f26a45667f3c-kserve-provision-location\") pod \"isvc-logger-raw-4b426-predictor-759bcfd9b-5qmg7\" (UID: \"a4bf19f8-f39c-4e8b-b38e-f26a45667f3c\") " pod="kserve-ci-e2e-test/isvc-logger-raw-4b426-predictor-759bcfd9b-5qmg7"
May 26 16:13:51.064705 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:13:51.064686    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-logger-raw-4b426-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a4bf19f8-f39c-4e8b-b38e-f26a45667f3c-isvc-logger-raw-4b426-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-4b426-predictor-759bcfd9b-5qmg7\" (UID: \"a4bf19f8-f39c-4e8b-b38e-f26a45667f3c\") " pod="kserve-ci-e2e-test/isvc-logger-raw-4b426-predictor-759bcfd9b-5qmg7"
May 26 16:13:51.073874 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:13:51.073854    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqbhn\" (UniqueName: \"kubernetes.io/projected/a4bf19f8-f39c-4e8b-b38e-f26a45667f3c-kube-api-access-qqbhn\") pod \"isvc-logger-raw-4b426-predictor-759bcfd9b-5qmg7\" (UID: \"a4bf19f8-f39c-4e8b-b38e-f26a45667f3c\") " pod="kserve-ci-e2e-test/isvc-logger-raw-4b426-predictor-759bcfd9b-5qmg7"
May 26 16:13:51.569430 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:13:51.569394    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a4bf19f8-f39c-4e8b-b38e-f26a45667f3c-proxy-tls\") pod \"isvc-logger-raw-4b426-predictor-759bcfd9b-5qmg7\" (UID: \"a4bf19f8-f39c-4e8b-b38e-f26a45667f3c\") " pod="kserve-ci-e2e-test/isvc-logger-raw-4b426-predictor-759bcfd9b-5qmg7"
May 26 16:13:51.571738 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:13:51.571720    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a4bf19f8-f39c-4e8b-b38e-f26a45667f3c-proxy-tls\") pod \"isvc-logger-raw-4b426-predictor-759bcfd9b-5qmg7\" (UID: \"a4bf19f8-f39c-4e8b-b38e-f26a45667f3c\") " pod="kserve-ci-e2e-test/isvc-logger-raw-4b426-predictor-759bcfd9b-5qmg7"
May 26 16:13:51.861174 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:13:51.861077    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-4b426-predictor-759bcfd9b-5qmg7"
May 26 16:13:51.995429 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:13:51.995392    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-4b426-predictor-759bcfd9b-5qmg7"]
May 26 16:13:51.998642 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:13:51.998608    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4bf19f8_f39c_4e8b_b38e_f26a45667f3c.slice/crio-4d03db817484cc4e874c974029da5270ccb2bd18d5fbcea1c4b06643c1f61f0b WatchSource:0}: Error finding container 4d03db817484cc4e874c974029da5270ccb2bd18d5fbcea1c4b06643c1f61f0b: Status 404 returned error can't find the container with id 4d03db817484cc4e874c974029da5270ccb2bd18d5fbcea1c4b06643c1f61f0b
May 26 16:13:52.220141 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:13:52.220106    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-4b426-predictor-759bcfd9b-5qmg7" event={"ID":"a4bf19f8-f39c-4e8b-b38e-f26a45667f3c","Type":"ContainerStarted","Data":"33c977834aa0f3b54d750a18960727111ce7f72649d6b5702bcf527d0a40a165"}
May 26 16:13:52.220141 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:13:52.220145    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-4b426-predictor-759bcfd9b-5qmg7" event={"ID":"a4bf19f8-f39c-4e8b-b38e-f26a45667f3c","Type":"ContainerStarted","Data":"4d03db817484cc4e874c974029da5270ccb2bd18d5fbcea1c4b06643c1f61f0b"}
May 26 16:13:56.234455 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:13:56.234420    2556 generic.go:358] "Generic (PLEG): container finished" podID="a4bf19f8-f39c-4e8b-b38e-f26a45667f3c" containerID="33c977834aa0f3b54d750a18960727111ce7f72649d6b5702bcf527d0a40a165" exitCode=0
May 26 16:13:56.234935 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:13:56.234475    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-4b426-predictor-759bcfd9b-5qmg7" event={"ID":"a4bf19f8-f39c-4e8b-b38e-f26a45667f3c","Type":"ContainerDied","Data":"33c977834aa0f3b54d750a18960727111ce7f72649d6b5702bcf527d0a40a165"}
May 26 16:13:57.241039 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:13:57.240957    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-4b426-predictor-759bcfd9b-5qmg7" event={"ID":"a4bf19f8-f39c-4e8b-b38e-f26a45667f3c","Type":"ContainerStarted","Data":"4df65fcfb8f95d504424624941975c1d26154e4bc1b083ca52a79fb239022724"}
May 26 16:13:57.241039 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:13:57.240997    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-4b426-predictor-759bcfd9b-5qmg7" event={"ID":"a4bf19f8-f39c-4e8b-b38e-f26a45667f3c","Type":"ContainerStarted","Data":"b61d6b2f10c5d6c4de5138b8fd16b162d79836c902fac1451ee403a09a6a0903"}
May 26 16:13:57.241039 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:13:57.241010    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-4b426-predictor-759bcfd9b-5qmg7" event={"ID":"a4bf19f8-f39c-4e8b-b38e-f26a45667f3c","Type":"ContainerStarted","Data":"372bd8c2c0ed97785623cb3c90a20592a151ad18f749288b62639f5f4464ad09"}
May 26 16:13:57.241558 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:13:57.241345    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-4b426-predictor-759bcfd9b-5qmg7"
May 26 16:13:57.241558 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:13:57.241372    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-4b426-predictor-759bcfd9b-5qmg7"
May 26 16:13:57.241558 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:13:57.241383    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-4b426-predictor-759bcfd9b-5qmg7"
May 26 16:13:57.242760 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:13:57.242738    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-4b426-predictor-759bcfd9b-5qmg7" podUID="a4bf19f8-f39c-4e8b-b38e-f26a45667f3c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
May 26 16:13:57.243380 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:13:57.243359    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-4b426-predictor-759bcfd9b-5qmg7" podUID="a4bf19f8-f39c-4e8b-b38e-f26a45667f3c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 16:13:57.267895 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:13:57.267850    2556 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-logger-raw-4b426-predictor-759bcfd9b-5qmg7" podStartSLOduration=7.267837098 podStartE2EDuration="7.267837098s" podCreationTimestamp="2026-05-26 16:13:50 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 16:13:57.266200967 +0000 UTC m=+799.110817959" watchObservedRunningTime="2026-05-26 16:13:57.267837098 +0000 UTC m=+799.112454089"
May 26 16:13:58.244501 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:13:58.244465    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-4b426-predictor-759bcfd9b-5qmg7" podUID="a4bf19f8-f39c-4e8b-b38e-f26a45667f3c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
May 26 16:13:58.244883 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:13:58.244816    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-4b426-predictor-759bcfd9b-5qmg7" podUID="a4bf19f8-f39c-4e8b-b38e-f26a45667f3c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 16:14:03.248562 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:14:03.248533    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-4b426-predictor-759bcfd9b-5qmg7"
May 26 16:14:03.249196 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:14:03.249151    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-4b426-predictor-759bcfd9b-5qmg7" podUID="a4bf19f8-f39c-4e8b-b38e-f26a45667f3c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
May 26 16:14:03.249562 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:14:03.249539    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-4b426-predictor-759bcfd9b-5qmg7" podUID="a4bf19f8-f39c-4e8b-b38e-f26a45667f3c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 16:14:13.249113 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:14:13.249050    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-4b426-predictor-759bcfd9b-5qmg7" podUID="a4bf19f8-f39c-4e8b-b38e-f26a45667f3c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
May 26 16:14:13.249490 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:14:13.249428    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-4b426-predictor-759bcfd9b-5qmg7" podUID="a4bf19f8-f39c-4e8b-b38e-f26a45667f3c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 16:14:23.249850 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:14:23.249809    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-4b426-predictor-759bcfd9b-5qmg7" podUID="a4bf19f8-f39c-4e8b-b38e-f26a45667f3c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
May 26 16:14:23.250292 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:14:23.250203    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-4b426-predictor-759bcfd9b-5qmg7" podUID="a4bf19f8-f39c-4e8b-b38e-f26a45667f3c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 16:14:33.249154 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:14:33.249109    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-4b426-predictor-759bcfd9b-5qmg7" podUID="a4bf19f8-f39c-4e8b-b38e-f26a45667f3c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
May 26 16:14:33.249675 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:14:33.249514    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-4b426-predictor-759bcfd9b-5qmg7" podUID="a4bf19f8-f39c-4e8b-b38e-f26a45667f3c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 16:14:43.249578 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:14:43.249530    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-4b426-predictor-759bcfd9b-5qmg7" podUID="a4bf19f8-f39c-4e8b-b38e-f26a45667f3c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
May 26 16:14:43.250005 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:14:43.249972    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-4b426-predictor-759bcfd9b-5qmg7" podUID="a4bf19f8-f39c-4e8b-b38e-f26a45667f3c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 16:14:53.249343 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:14:53.249297    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-4b426-predictor-759bcfd9b-5qmg7" podUID="a4bf19f8-f39c-4e8b-b38e-f26a45667f3c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
May 26 16:14:53.249811 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:14:53.249787    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-4b426-predictor-759bcfd9b-5qmg7" podUID="a4bf19f8-f39c-4e8b-b38e-f26a45667f3c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 16:15:03.250280 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:15:03.250249    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-4b426-predictor-759bcfd9b-5qmg7"
May 26 16:15:03.250737 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:15:03.250312    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-4b426-predictor-759bcfd9b-5qmg7"
May 26 16:15:15.917929 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:15:15.917890    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_message-dumper-raw-4b426-predictor-75c485b4b4-zwcxz_95eb1efb-8eec-4333-b8b3-0dd730ea8532/kserve-container/0.log"
May 26 16:15:16.151478 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:15:16.151441    2556 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-4b426-predictor-759bcfd9b-5qmg7"]
May 26 16:15:16.151908 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:15:16.151878    2556 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-4b426-predictor-759bcfd9b-5qmg7" podUID="a4bf19f8-f39c-4e8b-b38e-f26a45667f3c" containerName="kserve-container" containerID="cri-o://372bd8c2c0ed97785623cb3c90a20592a151ad18f749288b62639f5f4464ad09" gracePeriod=30
May 26 16:15:16.152137 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:15:16.151896    2556 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-4b426-predictor-759bcfd9b-5qmg7" podUID="a4bf19f8-f39c-4e8b-b38e-f26a45667f3c" containerName="agent" containerID="cri-o://4df65fcfb8f95d504424624941975c1d26154e4bc1b083ca52a79fb239022724" gracePeriod=30
May 26 16:15:16.152263 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:15:16.151913    2556 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-4b426-predictor-759bcfd9b-5qmg7" podUID="a4bf19f8-f39c-4e8b-b38e-f26a45667f3c" containerName="kube-rbac-proxy" containerID="cri-o://b61d6b2f10c5d6c4de5138b8fd16b162d79836c902fac1451ee403a09a6a0903" gracePeriod=30
May 26 16:15:16.218297 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:15:16.218244    2556 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-c3450-predictor-5c6fc6c8c5-fwm8n"]
May 26 16:15:16.221598 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:15:16.221582    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-c3450-predictor-5c6fc6c8c5-fwm8n"
May 26 16:15:16.227489 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:15:16.227467    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-scale-raw-c3450-kube-rbac-proxy-sar-config\""
May 26 16:15:16.227687 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:15:16.227496    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-scale-raw-c3450-predictor-serving-cert\""
May 26 16:15:16.237653 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:15:16.237629    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-c3450-predictor-5c6fc6c8c5-fwm8n"]
May 26 16:15:16.312140 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:15:16.312087    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/55cb3473-0b1e-498a-bbcd-70b6660e763b-proxy-tls\") pod \"isvc-sklearn-scale-raw-c3450-predictor-5c6fc6c8c5-fwm8n\" (UID: \"55cb3473-0b1e-498a-bbcd-70b6660e763b\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-c3450-predictor-5c6fc6c8c5-fwm8n"
May 26 16:15:16.312277 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:15:16.312163    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-sklearn-scale-raw-c3450-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/55cb3473-0b1e-498a-bbcd-70b6660e763b-isvc-sklearn-scale-raw-c3450-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-c3450-predictor-5c6fc6c8c5-fwm8n\" (UID: \"55cb3473-0b1e-498a-bbcd-70b6660e763b\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-c3450-predictor-5c6fc6c8c5-fwm8n"
May 26 16:15:16.312277 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:15:16.312199    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/55cb3473-0b1e-498a-bbcd-70b6660e763b-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-c3450-predictor-5c6fc6c8c5-fwm8n\" (UID: \"55cb3473-0b1e-498a-bbcd-70b6660e763b\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-c3450-predictor-5c6fc6c8c5-fwm8n"
May 26 16:15:16.312277 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:15:16.312255    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66lnc\" (UniqueName: \"kubernetes.io/projected/55cb3473-0b1e-498a-bbcd-70b6660e763b-kube-api-access-66lnc\") pod \"isvc-sklearn-scale-raw-c3450-predictor-5c6fc6c8c5-fwm8n\" (UID: \"55cb3473-0b1e-498a-bbcd-70b6660e763b\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-c3450-predictor-5c6fc6c8c5-fwm8n"
May 26 16:15:16.323354 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:15:16.323323    2556 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-4b426-predictor-75c485b4b4-zwcxz"]
May 26 16:15:16.323691 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:15:16.323663    2556 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-4b426-predictor-75c485b4b4-zwcxz" podUID="95eb1efb-8eec-4333-b8b3-0dd730ea8532" containerName="kserve-container" containerID="cri-o://8be754488cc528859b681547305adb20397850d310ed5358988808fb364013fc" gracePeriod=30
May 26 16:15:16.323786 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:15:16.323731    2556 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-4b426-predictor-75c485b4b4-zwcxz" podUID="95eb1efb-8eec-4333-b8b3-0dd730ea8532" containerName="kube-rbac-proxy" containerID="cri-o://4395b85a95c4537fd40f825af83ba07dd85cbf690ff5d17de989c7b5d5919712" gracePeriod=30
May 26 16:15:16.412605 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:15:16.412577    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-66lnc\" (UniqueName: \"kubernetes.io/projected/55cb3473-0b1e-498a-bbcd-70b6660e763b-kube-api-access-66lnc\") pod \"isvc-sklearn-scale-raw-c3450-predictor-5c6fc6c8c5-fwm8n\" (UID: \"55cb3473-0b1e-498a-bbcd-70b6660e763b\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-c3450-predictor-5c6fc6c8c5-fwm8n"
May 26 16:15:16.412705 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:15:16.412628    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/55cb3473-0b1e-498a-bbcd-70b6660e763b-proxy-tls\") pod \"isvc-sklearn-scale-raw-c3450-predictor-5c6fc6c8c5-fwm8n\" (UID: \"55cb3473-0b1e-498a-bbcd-70b6660e763b\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-c3450-predictor-5c6fc6c8c5-fwm8n"
May 26 16:15:16.412705 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:15:16.412691    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-sklearn-scale-raw-c3450-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/55cb3473-0b1e-498a-bbcd-70b6660e763b-isvc-sklearn-scale-raw-c3450-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-c3450-predictor-5c6fc6c8c5-fwm8n\" (UID: \"55cb3473-0b1e-498a-bbcd-70b6660e763b\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-c3450-predictor-5c6fc6c8c5-fwm8n"
May 26 16:15:16.412781 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:15:16.412734    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/55cb3473-0b1e-498a-bbcd-70b6660e763b-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-c3450-predictor-5c6fc6c8c5-fwm8n\" (UID: \"55cb3473-0b1e-498a-bbcd-70b6660e763b\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-c3450-predictor-5c6fc6c8c5-fwm8n"
May 26 16:15:16.413064 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:15:16.413040    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/55cb3473-0b1e-498a-bbcd-70b6660e763b-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-c3450-predictor-5c6fc6c8c5-fwm8n\" (UID: \"55cb3473-0b1e-498a-bbcd-70b6660e763b\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-c3450-predictor-5c6fc6c8c5-fwm8n"
May 26 16:15:16.413338 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:15:16.413319    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-sklearn-scale-raw-c3450-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/55cb3473-0b1e-498a-bbcd-70b6660e763b-isvc-sklearn-scale-raw-c3450-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-c3450-predictor-5c6fc6c8c5-fwm8n\" (UID: \"55cb3473-0b1e-498a-bbcd-70b6660e763b\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-c3450-predictor-5c6fc6c8c5-fwm8n"
May 26 16:15:16.414812 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:15:16.414795    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/55cb3473-0b1e-498a-bbcd-70b6660e763b-proxy-tls\") pod \"isvc-sklearn-scale-raw-c3450-predictor-5c6fc6c8c5-fwm8n\" (UID: \"55cb3473-0b1e-498a-bbcd-70b6660e763b\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-c3450-predictor-5c6fc6c8c5-fwm8n"
May 26 16:15:16.427204 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:15:16.427180    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-66lnc\" (UniqueName: \"kubernetes.io/projected/55cb3473-0b1e-498a-bbcd-70b6660e763b-kube-api-access-66lnc\") pod \"isvc-sklearn-scale-raw-c3450-predictor-5c6fc6c8c5-fwm8n\" (UID: \"55cb3473-0b1e-498a-bbcd-70b6660e763b\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-c3450-predictor-5c6fc6c8c5-fwm8n"
May 26 16:15:16.494413 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:15:16.494386    2556 generic.go:358] "Generic (PLEG): container finished" podID="95eb1efb-8eec-4333-b8b3-0dd730ea8532" containerID="4395b85a95c4537fd40f825af83ba07dd85cbf690ff5d17de989c7b5d5919712" exitCode=2
May 26 16:15:16.494413 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:15:16.494409    2556 generic.go:358] "Generic (PLEG): container finished" podID="95eb1efb-8eec-4333-b8b3-0dd730ea8532" containerID="8be754488cc528859b681547305adb20397850d310ed5358988808fb364013fc" exitCode=2
May 26 16:15:16.494611 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:15:16.494457    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-4b426-predictor-75c485b4b4-zwcxz" event={"ID":"95eb1efb-8eec-4333-b8b3-0dd730ea8532","Type":"ContainerDied","Data":"4395b85a95c4537fd40f825af83ba07dd85cbf690ff5d17de989c7b5d5919712"}
May 26 16:15:16.494611 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:15:16.494489    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-4b426-predictor-75c485b4b4-zwcxz" event={"ID":"95eb1efb-8eec-4333-b8b3-0dd730ea8532","Type":"ContainerDied","Data":"8be754488cc528859b681547305adb20397850d310ed5358988808fb364013fc"}
May 26 16:15:16.496705 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:15:16.496680    2556 generic.go:358] "Generic (PLEG): container finished" podID="a4bf19f8-f39c-4e8b-b38e-f26a45667f3c" containerID="b61d6b2f10c5d6c4de5138b8fd16b162d79836c902fac1451ee403a09a6a0903" exitCode=2
May 26 16:15:16.496833 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:15:16.496747    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-4b426-predictor-759bcfd9b-5qmg7" event={"ID":"a4bf19f8-f39c-4e8b-b38e-f26a45667f3c","Type":"ContainerDied","Data":"b61d6b2f10c5d6c4de5138b8fd16b162d79836c902fac1451ee403a09a6a0903"}
May 26 16:15:16.530990 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:15:16.530965    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-c3450-predictor-5c6fc6c8c5-fwm8n"
May 26 16:15:16.562995 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:15:16.562973    2556 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-4b426-predictor-75c485b4b4-zwcxz"
May 26 16:15:16.614533 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:15:16.614504    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/95eb1efb-8eec-4333-b8b3-0dd730ea8532-proxy-tls\") pod \"95eb1efb-8eec-4333-b8b3-0dd730ea8532\" (UID: \"95eb1efb-8eec-4333-b8b3-0dd730ea8532\") "
May 26 16:15:16.614670 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:15:16.614602    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-64hhv\" (UniqueName: \"kubernetes.io/projected/95eb1efb-8eec-4333-b8b3-0dd730ea8532-kube-api-access-64hhv\") pod \"95eb1efb-8eec-4333-b8b3-0dd730ea8532\" (UID: \"95eb1efb-8eec-4333-b8b3-0dd730ea8532\") "
May 26 16:15:16.614670 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:15:16.614632    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"message-dumper-raw-4b426-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/95eb1efb-8eec-4333-b8b3-0dd730ea8532-message-dumper-raw-4b426-kube-rbac-proxy-sar-config\") pod \"95eb1efb-8eec-4333-b8b3-0dd730ea8532\" (UID: \"95eb1efb-8eec-4333-b8b3-0dd730ea8532\") "
May 26 16:15:16.615065 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:15:16.615040    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95eb1efb-8eec-4333-b8b3-0dd730ea8532-message-dumper-raw-4b426-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "message-dumper-raw-4b426-kube-rbac-proxy-sar-config") pod "95eb1efb-8eec-4333-b8b3-0dd730ea8532" (UID: "95eb1efb-8eec-4333-b8b3-0dd730ea8532"). InnerVolumeSpecName "message-dumper-raw-4b426-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 16:15:16.618574 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:15:16.618544    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95eb1efb-8eec-4333-b8b3-0dd730ea8532-kube-api-access-64hhv" (OuterVolumeSpecName: "kube-api-access-64hhv") pod "95eb1efb-8eec-4333-b8b3-0dd730ea8532" (UID: "95eb1efb-8eec-4333-b8b3-0dd730ea8532"). InnerVolumeSpecName "kube-api-access-64hhv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 16:15:16.618574 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:15:16.618565    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95eb1efb-8eec-4333-b8b3-0dd730ea8532-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "95eb1efb-8eec-4333-b8b3-0dd730ea8532" (UID: "95eb1efb-8eec-4333-b8b3-0dd730ea8532"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 16:15:16.660245 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:15:16.660221    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-c3450-predictor-5c6fc6c8c5-fwm8n"]
May 26 16:15:16.662574 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:15:16.662549    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55cb3473_0b1e_498a_bbcd_70b6660e763b.slice/crio-3103abe9e3f1dc728b0b298f168c420130bf3ed76b739204a4ea78be86e3bf72 WatchSource:0}: Error finding container 3103abe9e3f1dc728b0b298f168c420130bf3ed76b739204a4ea78be86e3bf72: Status 404 returned error can't find the container with id 3103abe9e3f1dc728b0b298f168c420130bf3ed76b739204a4ea78be86e3bf72
May 26 16:15:16.715652 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:15:16.715633    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-64hhv\" (UniqueName: \"kubernetes.io/projected/95eb1efb-8eec-4333-b8b3-0dd730ea8532-kube-api-access-64hhv\") on node \"ip-10-0-134-19.ec2.internal\" DevicePath \"\""
May 26 16:15:16.715742 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:15:16.715654    2556 reconciler_common.go:299] "Volume detached for volume \"message-dumper-raw-4b426-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/95eb1efb-8eec-4333-b8b3-0dd730ea8532-message-dumper-raw-4b426-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-19.ec2.internal\" DevicePath \"\""
May 26 16:15:16.715742 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:15:16.715665    2556 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/95eb1efb-8eec-4333-b8b3-0dd730ea8532-proxy-tls\") on node \"ip-10-0-134-19.ec2.internal\" DevicePath \"\""
May 26 16:15:17.501046 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:15:17.500960    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-4b426-predictor-75c485b4b4-zwcxz" event={"ID":"95eb1efb-8eec-4333-b8b3-0dd730ea8532","Type":"ContainerDied","Data":"b8cc8da261db6c8aa128ee3956b16a15be8ae2483bd7ccb2eb8e2a943563bc34"}
May 26 16:15:17.501046 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:15:17.500985    2556 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-4b426-predictor-75c485b4b4-zwcxz"
May 26 16:15:17.501046 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:15:17.501008    2556 scope.go:117] "RemoveContainer" containerID="4395b85a95c4537fd40f825af83ba07dd85cbf690ff5d17de989c7b5d5919712"
May 26 16:15:17.502439 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:15:17.502419    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-c3450-predictor-5c6fc6c8c5-fwm8n" event={"ID":"55cb3473-0b1e-498a-bbcd-70b6660e763b","Type":"ContainerStarted","Data":"6545e19f0ade09a5ce53fcaf15a2d6c7cb2a42e4cf56de87cb7d67781dbfb00c"}
May 26 16:15:17.502537 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:15:17.502444    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-c3450-predictor-5c6fc6c8c5-fwm8n" event={"ID":"55cb3473-0b1e-498a-bbcd-70b6660e763b","Type":"ContainerStarted","Data":"3103abe9e3f1dc728b0b298f168c420130bf3ed76b739204a4ea78be86e3bf72"}
May 26 16:15:17.508954 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:15:17.508935    2556 scope.go:117] "RemoveContainer" containerID="8be754488cc528859b681547305adb20397850d310ed5358988808fb364013fc"
May 26 16:15:17.530149 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:15:17.530127    2556 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-4b426-predictor-75c485b4b4-zwcxz"]
May 26 16:15:17.533460 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:15:17.533440    2556 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-4b426-predictor-75c485b4b4-zwcxz"]
May 26 16:15:18.244760 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:15:18.244722    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-4b426-predictor-759bcfd9b-5qmg7" podUID="a4bf19f8-f39c-4e8b-b38e-f26a45667f3c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.31:8643/healthz\": dial tcp 10.133.0.31:8643: connect: connection refused"
May 26 16:15:18.771410 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:15:18.771374    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="95eb1efb-8eec-4333-b8b3-0dd730ea8532" path="/var/lib/kubelet/pods/95eb1efb-8eec-4333-b8b3-0dd730ea8532/volumes"
May 26 16:15:20.515643 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:15:20.515611    2556 generic.go:358] "Generic (PLEG): container finished" podID="a4bf19f8-f39c-4e8b-b38e-f26a45667f3c" containerID="372bd8c2c0ed97785623cb3c90a20592a151ad18f749288b62639f5f4464ad09" exitCode=0
May 26 16:15:20.515968 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:15:20.515675    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-4b426-predictor-759bcfd9b-5qmg7" event={"ID":"a4bf19f8-f39c-4e8b-b38e-f26a45667f3c","Type":"ContainerDied","Data":"372bd8c2c0ed97785623cb3c90a20592a151ad18f749288b62639f5f4464ad09"}
May 26 16:15:20.517139 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:15:20.517119    2556 generic.go:358] "Generic (PLEG): container finished" podID="55cb3473-0b1e-498a-bbcd-70b6660e763b" containerID="6545e19f0ade09a5ce53fcaf15a2d6c7cb2a42e4cf56de87cb7d67781dbfb00c" exitCode=0
May 26 16:15:20.517226 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:15:20.517158    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-c3450-predictor-5c6fc6c8c5-fwm8n" event={"ID":"55cb3473-0b1e-498a-bbcd-70b6660e763b","Type":"ContainerDied","Data":"6545e19f0ade09a5ce53fcaf15a2d6c7cb2a42e4cf56de87cb7d67781dbfb00c"}
May 26 16:15:21.522511 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:15:21.522468    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-c3450-predictor-5c6fc6c8c5-fwm8n" event={"ID":"55cb3473-0b1e-498a-bbcd-70b6660e763b","Type":"ContainerStarted","Data":"560fa608bf89d80410d6df64fdb2f2a9512a0a0b6bf63a7bb6e5649ba748cdc7"}
May 26 16:15:21.522511 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:15:21.522507    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-c3450-predictor-5c6fc6c8c5-fwm8n" event={"ID":"55cb3473-0b1e-498a-bbcd-70b6660e763b","Type":"ContainerStarted","Data":"77b60e85ad6b046c827fcc39e2b396bf1d0057a9021964260d202bc5a52b8a08"}
May 26 16:15:21.522933 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:15:21.522728    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-c3450-predictor-5c6fc6c8c5-fwm8n"
May 26 16:15:21.548302 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:15:21.548262    2556 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-c3450-predictor-5c6fc6c8c5-fwm8n" podStartSLOduration=5.548249362 podStartE2EDuration="5.548249362s" podCreationTimestamp="2026-05-26 16:15:16 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 16:15:21.547452507 +0000 UTC m=+883.392069511" watchObservedRunningTime="2026-05-26 16:15:21.548249362 +0000 UTC m=+883.392866354"
May 26 16:15:22.525806 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:15:22.525775    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-c3450-predictor-5c6fc6c8c5-fwm8n"
May 26 16:15:22.526992 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:15:22.526962    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-c3450-predictor-5c6fc6c8c5-fwm8n" podUID="55cb3473-0b1e-498a-bbcd-70b6660e763b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
May 26 16:15:23.244702 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:15:23.244659    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-4b426-predictor-759bcfd9b-5qmg7" podUID="a4bf19f8-f39c-4e8b-b38e-f26a45667f3c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.31:8643/healthz\": dial tcp 10.133.0.31:8643: connect: connection refused"
May 26 16:15:23.250050 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:15:23.250012    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-4b426-predictor-759bcfd9b-5qmg7" podUID="a4bf19f8-f39c-4e8b-b38e-f26a45667f3c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
May 26 16:15:23.250362 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:15:23.250334    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-4b426-predictor-759bcfd9b-5qmg7" podUID="a4bf19f8-f39c-4e8b-b38e-f26a45667f3c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 16:15:23.529079 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:15:23.528987    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-c3450-predictor-5c6fc6c8c5-fwm8n" podUID="55cb3473-0b1e-498a-bbcd-70b6660e763b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
May 26 16:15:28.244766 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:15:28.244684    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-4b426-predictor-759bcfd9b-5qmg7" podUID="a4bf19f8-f39c-4e8b-b38e-f26a45667f3c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.31:8643/healthz\": dial tcp 10.133.0.31:8643: connect: connection refused"
May 26 16:15:28.245139 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:15:28.244811    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-4b426-predictor-759bcfd9b-5qmg7"
May 26 16:15:28.533548 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:15:28.533469    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-c3450-predictor-5c6fc6c8c5-fwm8n"
May 26 16:15:28.533956 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:15:28.533924    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-c3450-predictor-5c6fc6c8c5-fwm8n" podUID="55cb3473-0b1e-498a-bbcd-70b6660e763b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
May 26 16:15:33.244719 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:15:33.244679    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-4b426-predictor-759bcfd9b-5qmg7" podUID="a4bf19f8-f39c-4e8b-b38e-f26a45667f3c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.31:8643/healthz\": dial tcp 10.133.0.31:8643: connect: connection refused"
May 26 16:15:33.249099 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:15:33.249060    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-4b426-predictor-759bcfd9b-5qmg7" podUID="a4bf19f8-f39c-4e8b-b38e-f26a45667f3c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
May 26 16:15:33.249329 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:15:33.249315    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-4b426-predictor-759bcfd9b-5qmg7" podUID="a4bf19f8-f39c-4e8b-b38e-f26a45667f3c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 16:15:38.245276 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:15:38.245238    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-4b426-predictor-759bcfd9b-5qmg7" podUID="a4bf19f8-f39c-4e8b-b38e-f26a45667f3c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.31:8643/healthz\": dial tcp 10.133.0.31:8643: connect: connection refused"
May 26 16:15:38.534729 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:15:38.534645    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-c3450-predictor-5c6fc6c8c5-fwm8n" podUID="55cb3473-0b1e-498a-bbcd-70b6660e763b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
May 26 16:15:38.689157 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:15:38.689129    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sk7ns_4887c0d3-11fb-4287-bb94-a876a6c38803/ovn-acl-logging/0.log"
May 26 16:15:38.689884 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:15:38.689866    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sk7ns_4887c0d3-11fb-4287-bb94-a876a6c38803/ovn-acl-logging/0.log"
May 26 16:15:43.245072 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:15:43.245031    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-4b426-predictor-759bcfd9b-5qmg7" podUID="a4bf19f8-f39c-4e8b-b38e-f26a45667f3c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.31:8643/healthz\": dial tcp 10.133.0.31:8643: connect: connection refused"
May 26 16:15:43.249397 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:15:43.249358    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-4b426-predictor-759bcfd9b-5qmg7" podUID="a4bf19f8-f39c-4e8b-b38e-f26a45667f3c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
May 26 16:15:43.249536 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:15:43.249502    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-4b426-predictor-759bcfd9b-5qmg7"
May 26 16:15:43.249709 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:15:43.249683    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-4b426-predictor-759bcfd9b-5qmg7" podUID="a4bf19f8-f39c-4e8b-b38e-f26a45667f3c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 16:15:43.249814 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:15:43.249799    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-4b426-predictor-759bcfd9b-5qmg7"
May 26 16:15:46.604523 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:15:46.604443    2556 generic.go:358] "Generic (PLEG): container finished" podID="a4bf19f8-f39c-4e8b-b38e-f26a45667f3c" containerID="4df65fcfb8f95d504424624941975c1d26154e4bc1b083ca52a79fb239022724" exitCode=0
May 26 16:15:46.604855 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:15:46.604519    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-4b426-predictor-759bcfd9b-5qmg7" event={"ID":"a4bf19f8-f39c-4e8b-b38e-f26a45667f3c","Type":"ContainerDied","Data":"4df65fcfb8f95d504424624941975c1d26154e4bc1b083ca52a79fb239022724"}
May 26 16:15:46.799222 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:15:46.799201    2556 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-4b426-predictor-759bcfd9b-5qmg7"
May 26 16:15:46.933035 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:15:46.933007    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qqbhn\" (UniqueName: \"kubernetes.io/projected/a4bf19f8-f39c-4e8b-b38e-f26a45667f3c-kube-api-access-qqbhn\") pod \"a4bf19f8-f39c-4e8b-b38e-f26a45667f3c\" (UID: \"a4bf19f8-f39c-4e8b-b38e-f26a45667f3c\") "
May 26 16:15:46.933208 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:15:46.933060    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a4bf19f8-f39c-4e8b-b38e-f26a45667f3c-proxy-tls\") pod \"a4bf19f8-f39c-4e8b-b38e-f26a45667f3c\" (UID: \"a4bf19f8-f39c-4e8b-b38e-f26a45667f3c\") "
May 26 16:15:46.933208 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:15:46.933189    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a4bf19f8-f39c-4e8b-b38e-f26a45667f3c-kserve-provision-location\") pod \"a4bf19f8-f39c-4e8b-b38e-f26a45667f3c\" (UID: \"a4bf19f8-f39c-4e8b-b38e-f26a45667f3c\") "
May 26 16:15:46.933284 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:15:46.933246    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-logger-raw-4b426-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a4bf19f8-f39c-4e8b-b38e-f26a45667f3c-isvc-logger-raw-4b426-kube-rbac-proxy-sar-config\") pod \"a4bf19f8-f39c-4e8b-b38e-f26a45667f3c\" (UID: \"a4bf19f8-f39c-4e8b-b38e-f26a45667f3c\") "
May 26 16:15:46.933525 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:15:46.933495    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4bf19f8-f39c-4e8b-b38e-f26a45667f3c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a4bf19f8-f39c-4e8b-b38e-f26a45667f3c" (UID: "a4bf19f8-f39c-4e8b-b38e-f26a45667f3c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 16:15:46.933610 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:15:46.933559    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4bf19f8-f39c-4e8b-b38e-f26a45667f3c-isvc-logger-raw-4b426-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-logger-raw-4b426-kube-rbac-proxy-sar-config") pod "a4bf19f8-f39c-4e8b-b38e-f26a45667f3c" (UID: "a4bf19f8-f39c-4e8b-b38e-f26a45667f3c"). InnerVolumeSpecName "isvc-logger-raw-4b426-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 16:15:46.935065 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:15:46.935045    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4bf19f8-f39c-4e8b-b38e-f26a45667f3c-kube-api-access-qqbhn" (OuterVolumeSpecName: "kube-api-access-qqbhn") pod "a4bf19f8-f39c-4e8b-b38e-f26a45667f3c" (UID: "a4bf19f8-f39c-4e8b-b38e-f26a45667f3c"). InnerVolumeSpecName "kube-api-access-qqbhn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 16:15:46.935183 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:15:46.935143    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4bf19f8-f39c-4e8b-b38e-f26a45667f3c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a4bf19f8-f39c-4e8b-b38e-f26a45667f3c" (UID: "a4bf19f8-f39c-4e8b-b38e-f26a45667f3c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 16:15:47.034335 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:15:47.034305    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qqbhn\" (UniqueName: \"kubernetes.io/projected/a4bf19f8-f39c-4e8b-b38e-f26a45667f3c-kube-api-access-qqbhn\") on node \"ip-10-0-134-19.ec2.internal\" DevicePath \"\""
May 26 16:15:47.034335 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:15:47.034330    2556 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a4bf19f8-f39c-4e8b-b38e-f26a45667f3c-proxy-tls\") on node \"ip-10-0-134-19.ec2.internal\" DevicePath \"\""
May 26 16:15:47.034335 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:15:47.034341    2556 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a4bf19f8-f39c-4e8b-b38e-f26a45667f3c-kserve-provision-location\") on node \"ip-10-0-134-19.ec2.internal\" DevicePath \"\""
May 26 16:15:47.034579 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:15:47.034352    2556 reconciler_common.go:299] "Volume detached for volume \"isvc-logger-raw-4b426-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a4bf19f8-f39c-4e8b-b38e-f26a45667f3c-isvc-logger-raw-4b426-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-19.ec2.internal\" DevicePath \"\""
May 26 16:15:47.610267 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:15:47.610230    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-4b426-predictor-759bcfd9b-5qmg7" event={"ID":"a4bf19f8-f39c-4e8b-b38e-f26a45667f3c","Type":"ContainerDied","Data":"4d03db817484cc4e874c974029da5270ccb2bd18d5fbcea1c4b06643c1f61f0b"}
May 26 16:15:47.610267 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:15:47.610264    2556 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-4b426-predictor-759bcfd9b-5qmg7"
May 26 16:15:47.610705 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:15:47.610285    2556 scope.go:117] "RemoveContainer" containerID="4df65fcfb8f95d504424624941975c1d26154e4bc1b083ca52a79fb239022724"
May 26 16:15:47.619262 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:15:47.619240    2556 scope.go:117] "RemoveContainer" containerID="b61d6b2f10c5d6c4de5138b8fd16b162d79836c902fac1451ee403a09a6a0903"
May 26 16:15:47.627958 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:15:47.627941    2556 scope.go:117] "RemoveContainer" containerID="372bd8c2c0ed97785623cb3c90a20592a151ad18f749288b62639f5f4464ad09"
May 26 16:15:47.634539 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:15:47.634522    2556 scope.go:117] "RemoveContainer" containerID="33c977834aa0f3b54d750a18960727111ce7f72649d6b5702bcf527d0a40a165"
May 26 16:15:47.642804 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:15:47.642784    2556 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-4b426-predictor-759bcfd9b-5qmg7"]
May 26 16:15:47.649702 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:15:47.649681    2556 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-4b426-predictor-759bcfd9b-5qmg7"]
May 26 16:15:48.534750 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:15:48.534707    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-c3450-predictor-5c6fc6c8c5-fwm8n" podUID="55cb3473-0b1e-498a-bbcd-70b6660e763b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
May 26 16:15:48.771479 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:15:48.771447    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a4bf19f8-f39c-4e8b-b38e-f26a45667f3c" path="/var/lib/kubelet/pods/a4bf19f8-f39c-4e8b-b38e-f26a45667f3c/volumes"
May 26 16:15:58.534195 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:15:58.534150    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-c3450-predictor-5c6fc6c8c5-fwm8n" podUID="55cb3473-0b1e-498a-bbcd-70b6660e763b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
May 26 16:16:08.534037 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:16:08.533999    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-c3450-predictor-5c6fc6c8c5-fwm8n" podUID="55cb3473-0b1e-498a-bbcd-70b6660e763b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
May 26 16:16:18.534920 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:16:18.534880    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-c3450-predictor-5c6fc6c8c5-fwm8n" podUID="55cb3473-0b1e-498a-bbcd-70b6660e763b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
May 26 16:16:28.533975 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:16:28.533931    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-c3450-predictor-5c6fc6c8c5-fwm8n" podUID="55cb3473-0b1e-498a-bbcd-70b6660e763b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
May 26 16:16:38.534209 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:16:38.534169    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-c3450-predictor-5c6fc6c8c5-fwm8n" podUID="55cb3473-0b1e-498a-bbcd-70b6660e763b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
May 26 16:16:44.767818 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:16:44.767775    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-c3450-predictor-5c6fc6c8c5-fwm8n" podUID="55cb3473-0b1e-498a-bbcd-70b6660e763b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
May 26 16:16:54.767707 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:16:54.767661    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-c3450-predictor-5c6fc6c8c5-fwm8n" podUID="55cb3473-0b1e-498a-bbcd-70b6660e763b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
May 26 16:17:04.768479 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:17:04.768433    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-c3450-predictor-5c6fc6c8c5-fwm8n" podUID="55cb3473-0b1e-498a-bbcd-70b6660e763b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
May 26 16:17:14.767824 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:17:14.767771    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-c3450-predictor-5c6fc6c8c5-fwm8n" podUID="55cb3473-0b1e-498a-bbcd-70b6660e763b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
May 26 16:17:24.767647 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:17:24.767605    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-c3450-predictor-5c6fc6c8c5-fwm8n" podUID="55cb3473-0b1e-498a-bbcd-70b6660e763b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
May 26 16:17:34.771585 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:17:34.771555    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-c3450-predictor-5c6fc6c8c5-fwm8n"
May 26 16:17:36.431695 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:17:36.431662    2556 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-c3450-predictor-5c6fc6c8c5-fwm8n"]
May 26 16:17:36.432173 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:17:36.432000    2556 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-c3450-predictor-5c6fc6c8c5-fwm8n" podUID="55cb3473-0b1e-498a-bbcd-70b6660e763b" containerName="kserve-container" containerID="cri-o://77b60e85ad6b046c827fcc39e2b396bf1d0057a9021964260d202bc5a52b8a08" gracePeriod=30
May 26 16:17:36.432173 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:17:36.432037    2556 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-c3450-predictor-5c6fc6c8c5-fwm8n" podUID="55cb3473-0b1e-498a-bbcd-70b6660e763b" containerName="kube-rbac-proxy" containerID="cri-o://560fa608bf89d80410d6df64fdb2f2a9512a0a0b6bf63a7bb6e5649ba748cdc7" gracePeriod=30
May 26 16:17:36.561179 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:17:36.561150    2556 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-primary-ba2514-predictor-686dd7994-vcx9l"]
May 26 16:17:36.561519 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:17:36.561503    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4bf19f8-f39c-4e8b-b38e-f26a45667f3c" containerName="agent"
May 26 16:17:36.561597 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:17:36.561522    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4bf19f8-f39c-4e8b-b38e-f26a45667f3c" containerName="agent"
May 26 16:17:36.561597 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:17:36.561546    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4bf19f8-f39c-4e8b-b38e-f26a45667f3c" containerName="kube-rbac-proxy"
May 26 16:17:36.561597 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:17:36.561556    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4bf19f8-f39c-4e8b-b38e-f26a45667f3c" containerName="kube-rbac-proxy"
May 26 16:17:36.561597 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:17:36.561570    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4bf19f8-f39c-4e8b-b38e-f26a45667f3c" containerName="storage-initializer"
May 26 16:17:36.561597 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:17:36.561579    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4bf19f8-f39c-4e8b-b38e-f26a45667f3c" containerName="storage-initializer"
May 26 16:17:36.561597 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:17:36.561594    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="95eb1efb-8eec-4333-b8b3-0dd730ea8532" containerName="kserve-container"
May 26 16:17:36.561888 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:17:36.561603    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="95eb1efb-8eec-4333-b8b3-0dd730ea8532" containerName="kserve-container"
May 26 16:17:36.561888 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:17:36.561621    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="95eb1efb-8eec-4333-b8b3-0dd730ea8532" containerName="kube-rbac-proxy"
May 26 16:17:36.561888 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:17:36.561629    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="95eb1efb-8eec-4333-b8b3-0dd730ea8532" containerName="kube-rbac-proxy"
May 26 16:17:36.561888 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:17:36.561638    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4bf19f8-f39c-4e8b-b38e-f26a45667f3c" containerName="kserve-container"
May 26 16:17:36.561888 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:17:36.561646    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4bf19f8-f39c-4e8b-b38e-f26a45667f3c" containerName="kserve-container"
May 26 16:17:36.561888 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:17:36.561720    2556 memory_manager.go:356] "RemoveStaleState removing state" podUID="a4bf19f8-f39c-4e8b-b38e-f26a45667f3c" containerName="kube-rbac-proxy"
May 26 16:17:36.561888 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:17:36.561737    2556 memory_manager.go:356] "RemoveStaleState removing state" podUID="a4bf19f8-f39c-4e8b-b38e-f26a45667f3c" containerName="agent"
May 26 16:17:36.561888 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:17:36.561749    2556 memory_manager.go:356] "RemoveStaleState removing state" podUID="95eb1efb-8eec-4333-b8b3-0dd730ea8532" containerName="kube-rbac-proxy"
May 26 16:17:36.561888 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:17:36.561759    2556 memory_manager.go:356] "RemoveStaleState removing state" podUID="a4bf19f8-f39c-4e8b-b38e-f26a45667f3c" containerName="kserve-container"
May 26 16:17:36.561888 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:17:36.561771    2556 memory_manager.go:356] "RemoveStaleState removing state" podUID="95eb1efb-8eec-4333-b8b3-0dd730ea8532" containerName="kserve-container"
May 26 16:17:36.564964 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:17:36.564945    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-ba2514-predictor-686dd7994-vcx9l"
May 26 16:17:36.567708 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:17:36.567682    2556 status_manager.go:895] "Failed to get status for pod" podUID="59ce4742-c702-4783-aca6-e7b3ab74513f" pod="kserve-ci-e2e-test/isvc-primary-ba2514-predictor-686dd7994-vcx9l" err="pods \"isvc-primary-ba2514-predictor-686dd7994-vcx9l\" is forbidden: User \"system:node:ip-10-0-134-19.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kserve-ci-e2e-test\": no relationship found between node 'ip-10-0-134-19.ec2.internal' and this object"
May 26 16:17:36.567903 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:17:36.567882    2556 reflector.go:200] "Failed to watch" err="failed to list *v1.Secret: secrets \"isvc-primary-ba2514-predictor-serving-cert\" is forbidden: User \"system:node:ip-10-0-134-19.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"kserve-ci-e2e-test\": no relationship found between node 'ip-10-0-134-19.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-ba2514-predictor-serving-cert\"" type="*v1.Secret"
May 26 16:17:36.568318 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:17:36.568305    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-ba2514-kube-rbac-proxy-sar-config\""
May 26 16:17:36.584004 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:17:36.583985    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-ba2514-predictor-686dd7994-vcx9l"]
May 26 16:17:36.745796 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:17:36.745707    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-primary-ba2514-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/59ce4742-c702-4783-aca6-e7b3ab74513f-isvc-primary-ba2514-kube-rbac-proxy-sar-config\") pod \"isvc-primary-ba2514-predictor-686dd7994-vcx9l\" (UID: \"59ce4742-c702-4783-aca6-e7b3ab74513f\") " pod="kserve-ci-e2e-test/isvc-primary-ba2514-predictor-686dd7994-vcx9l"
May 26 16:17:36.745796 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:17:36.745755    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fsq8k\" (UniqueName: \"kubernetes.io/projected/59ce4742-c702-4783-aca6-e7b3ab74513f-kube-api-access-fsq8k\") pod \"isvc-primary-ba2514-predictor-686dd7994-vcx9l\" (UID: \"59ce4742-c702-4783-aca6-e7b3ab74513f\") " pod="kserve-ci-e2e-test/isvc-primary-ba2514-predictor-686dd7994-vcx9l"
May 26 16:17:36.745992 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:17:36.745810    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/59ce4742-c702-4783-aca6-e7b3ab74513f-kserve-provision-location\") pod \"isvc-primary-ba2514-predictor-686dd7994-vcx9l\" (UID: \"59ce4742-c702-4783-aca6-e7b3ab74513f\") " pod="kserve-ci-e2e-test/isvc-primary-ba2514-predictor-686dd7994-vcx9l"
May 26 16:17:36.745992 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:17:36.745855    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/59ce4742-c702-4783-aca6-e7b3ab74513f-proxy-tls\") pod \"isvc-primary-ba2514-predictor-686dd7994-vcx9l\" (UID: \"59ce4742-c702-4783-aca6-e7b3ab74513f\") " pod="kserve-ci-e2e-test/isvc-primary-ba2514-predictor-686dd7994-vcx9l"
May 26 16:17:36.846446 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:17:36.846401    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-primary-ba2514-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/59ce4742-c702-4783-aca6-e7b3ab74513f-isvc-primary-ba2514-kube-rbac-proxy-sar-config\") pod \"isvc-primary-ba2514-predictor-686dd7994-vcx9l\" (UID: \"59ce4742-c702-4783-aca6-e7b3ab74513f\") " pod="kserve-ci-e2e-test/isvc-primary-ba2514-predictor-686dd7994-vcx9l"
May 26 16:17:36.846614 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:17:36.846475    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-fsq8k\" (UniqueName: \"kubernetes.io/projected/59ce4742-c702-4783-aca6-e7b3ab74513f-kube-api-access-fsq8k\") pod \"isvc-primary-ba2514-predictor-686dd7994-vcx9l\" (UID: \"59ce4742-c702-4783-aca6-e7b3ab74513f\") " pod="kserve-ci-e2e-test/isvc-primary-ba2514-predictor-686dd7994-vcx9l"
May 26 16:17:36.846614 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:17:36.846516    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/59ce4742-c702-4783-aca6-e7b3ab74513f-kserve-provision-location\") pod \"isvc-primary-ba2514-predictor-686dd7994-vcx9l\" (UID: \"59ce4742-c702-4783-aca6-e7b3ab74513f\") " pod="kserve-ci-e2e-test/isvc-primary-ba2514-predictor-686dd7994-vcx9l"
May 26 16:17:36.846614 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:17:36.846550    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/59ce4742-c702-4783-aca6-e7b3ab74513f-proxy-tls\") pod \"isvc-primary-ba2514-predictor-686dd7994-vcx9l\" (UID: \"59ce4742-c702-4783-aca6-e7b3ab74513f\") " pod="kserve-ci-e2e-test/isvc-primary-ba2514-predictor-686dd7994-vcx9l"
May 26 16:17:36.847064 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:17:36.847034    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/59ce4742-c702-4783-aca6-e7b3ab74513f-kserve-provision-location\") pod \"isvc-primary-ba2514-predictor-686dd7994-vcx9l\" (UID: \"59ce4742-c702-4783-aca6-e7b3ab74513f\") " pod="kserve-ci-e2e-test/isvc-primary-ba2514-predictor-686dd7994-vcx9l"
May 26 16:17:36.847133 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:17:36.847114    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-primary-ba2514-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/59ce4742-c702-4783-aca6-e7b3ab74513f-isvc-primary-ba2514-kube-rbac-proxy-sar-config\") pod \"isvc-primary-ba2514-predictor-686dd7994-vcx9l\" (UID: \"59ce4742-c702-4783-aca6-e7b3ab74513f\") " pod="kserve-ci-e2e-test/isvc-primary-ba2514-predictor-686dd7994-vcx9l"
May 26 16:17:36.857484 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:17:36.857453    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fsq8k\" (UniqueName: \"kubernetes.io/projected/59ce4742-c702-4783-aca6-e7b3ab74513f-kube-api-access-fsq8k\") pod \"isvc-primary-ba2514-predictor-686dd7994-vcx9l\" (UID: \"59ce4742-c702-4783-aca6-e7b3ab74513f\") " pod="kserve-ci-e2e-test/isvc-primary-ba2514-predictor-686dd7994-vcx9l"
May 26 16:17:36.948602 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:17:36.948570    2556 generic.go:358] "Generic (PLEG): container finished" podID="55cb3473-0b1e-498a-bbcd-70b6660e763b" containerID="560fa608bf89d80410d6df64fdb2f2a9512a0a0b6bf63a7bb6e5649ba748cdc7" exitCode=2
May 26 16:17:36.948759 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:17:36.948633    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-c3450-predictor-5c6fc6c8c5-fwm8n" event={"ID":"55cb3473-0b1e-498a-bbcd-70b6660e763b","Type":"ContainerDied","Data":"560fa608bf89d80410d6df64fdb2f2a9512a0a0b6bf63a7bb6e5649ba748cdc7"}
May 26 16:17:37.750307 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:17:37.750275    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-ba2514-predictor-serving-cert\""
May 26 16:17:37.759268 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:17:37.759244    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/59ce4742-c702-4783-aca6-e7b3ab74513f-proxy-tls\") pod \"isvc-primary-ba2514-predictor-686dd7994-vcx9l\" (UID: \"59ce4742-c702-4783-aca6-e7b3ab74513f\") " pod="kserve-ci-e2e-test/isvc-primary-ba2514-predictor-686dd7994-vcx9l"
May 26 16:17:37.774452 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:17:37.774421    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-ba2514-predictor-686dd7994-vcx9l"
May 26 16:17:37.906120 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:17:37.906084    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-ba2514-predictor-686dd7994-vcx9l"]
May 26 16:17:37.908477 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:17:37.908454    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59ce4742_c702_4783_aca6_e7b3ab74513f.slice/crio-de2da45a6cc7346a72a59621ecbdc88c908967b21d9d0ba9b450921a790f1800 WatchSource:0}: Error finding container de2da45a6cc7346a72a59621ecbdc88c908967b21d9d0ba9b450921a790f1800: Status 404 returned error can't find the container with id de2da45a6cc7346a72a59621ecbdc88c908967b21d9d0ba9b450921a790f1800
May 26 16:17:37.910185 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:17:37.910167    2556 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 26 16:17:37.952826 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:17:37.952802    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-ba2514-predictor-686dd7994-vcx9l" event={"ID":"59ce4742-c702-4783-aca6-e7b3ab74513f","Type":"ContainerStarted","Data":"de2da45a6cc7346a72a59621ecbdc88c908967b21d9d0ba9b450921a790f1800"}
May 26 16:17:38.529567 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:17:38.529524    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-c3450-predictor-5c6fc6c8c5-fwm8n" podUID="55cb3473-0b1e-498a-bbcd-70b6660e763b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.32:8643/healthz\": dial tcp 10.133.0.32:8643: connect: connection refused"
May 26 16:17:38.957363 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:17:38.957328    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-ba2514-predictor-686dd7994-vcx9l" event={"ID":"59ce4742-c702-4783-aca6-e7b3ab74513f","Type":"ContainerStarted","Data":"f14e22618b28e0bde562653c1f30d858acdec72493344c8a7b4411be94d267a0"}
May 26 16:17:41.967979 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:17:41.967890    2556 generic.go:358] "Generic (PLEG): container finished" podID="59ce4742-c702-4783-aca6-e7b3ab74513f" containerID="f14e22618b28e0bde562653c1f30d858acdec72493344c8a7b4411be94d267a0" exitCode=0
May 26 16:17:41.968357 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:17:41.967968    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-ba2514-predictor-686dd7994-vcx9l" event={"ID":"59ce4742-c702-4783-aca6-e7b3ab74513f","Type":"ContainerDied","Data":"f14e22618b28e0bde562653c1f30d858acdec72493344c8a7b4411be94d267a0"}
May 26 16:17:42.972697 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:17:42.972658    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-ba2514-predictor-686dd7994-vcx9l" event={"ID":"59ce4742-c702-4783-aca6-e7b3ab74513f","Type":"ContainerStarted","Data":"58a91c81cd6f4d154871c9fea9b0f876ff59f487ccb0de2cfc49f44986155ed7"}
May 26 16:17:42.972697 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:17:42.972697    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-ba2514-predictor-686dd7994-vcx9l" event={"ID":"59ce4742-c702-4783-aca6-e7b3ab74513f","Type":"ContainerStarted","Data":"1194514042a2ea15d164f1519bfc187c98e55f77eb3c9971377f8d0d6cd968d6"}
May 26 16:17:42.973174 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:17:42.972994    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-ba2514-predictor-686dd7994-vcx9l"
May 26 16:17:42.973174 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:17:42.973122    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-ba2514-predictor-686dd7994-vcx9l"
May 26 16:17:42.974272 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:17:42.974244    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-ba2514-predictor-686dd7994-vcx9l" podUID="59ce4742-c702-4783-aca6-e7b3ab74513f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
May 26 16:17:42.995561 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:17:42.995524    2556 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-primary-ba2514-predictor-686dd7994-vcx9l" podStartSLOduration=6.995513472 podStartE2EDuration="6.995513472s" podCreationTimestamp="2026-05-26 16:17:36 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 16:17:42.994244658 +0000 UTC m=+1024.838861649" watchObservedRunningTime="2026-05-26 16:17:42.995513472 +0000 UTC m=+1024.840130456"
May 26 16:17:43.529661 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:17:43.529623    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-c3450-predictor-5c6fc6c8c5-fwm8n" podUID="55cb3473-0b1e-498a-bbcd-70b6660e763b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.32:8643/healthz\": dial tcp 10.133.0.32:8643: connect: connection refused"
May 26 16:17:43.975784 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:17:43.975745    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-ba2514-predictor-686dd7994-vcx9l" podUID="59ce4742-c702-4783-aca6-e7b3ab74513f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
May 26 16:17:44.767905 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:17:44.767861    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-c3450-predictor-5c6fc6c8c5-fwm8n" podUID="55cb3473-0b1e-498a-bbcd-70b6660e763b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
May 26 16:17:45.174108 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:17:45.174067    2556 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-c3450-predictor-5c6fc6c8c5-fwm8n"
May 26 16:17:45.201459 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:17:45.201433    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/55cb3473-0b1e-498a-bbcd-70b6660e763b-proxy-tls\") pod \"55cb3473-0b1e-498a-bbcd-70b6660e763b\" (UID: \"55cb3473-0b1e-498a-bbcd-70b6660e763b\") "
May 26 16:17:45.201575 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:17:45.201525    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/55cb3473-0b1e-498a-bbcd-70b6660e763b-kserve-provision-location\") pod \"55cb3473-0b1e-498a-bbcd-70b6660e763b\" (UID: \"55cb3473-0b1e-498a-bbcd-70b6660e763b\") "
May 26 16:17:45.201575 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:17:45.201554    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-66lnc\" (UniqueName: \"kubernetes.io/projected/55cb3473-0b1e-498a-bbcd-70b6660e763b-kube-api-access-66lnc\") pod \"55cb3473-0b1e-498a-bbcd-70b6660e763b\" (UID: \"55cb3473-0b1e-498a-bbcd-70b6660e763b\") "
May 26 16:17:45.201683 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:17:45.201611    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-sklearn-scale-raw-c3450-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/55cb3473-0b1e-498a-bbcd-70b6660e763b-isvc-sklearn-scale-raw-c3450-kube-rbac-proxy-sar-config\") pod \"55cb3473-0b1e-498a-bbcd-70b6660e763b\" (UID: \"55cb3473-0b1e-498a-bbcd-70b6660e763b\") "
May 26 16:17:45.201859 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:17:45.201830    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55cb3473-0b1e-498a-bbcd-70b6660e763b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "55cb3473-0b1e-498a-bbcd-70b6660e763b" (UID: "55cb3473-0b1e-498a-bbcd-70b6660e763b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 16:17:45.202028 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:17:45.201999    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55cb3473-0b1e-498a-bbcd-70b6660e763b-isvc-sklearn-scale-raw-c3450-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-sklearn-scale-raw-c3450-kube-rbac-proxy-sar-config") pod "55cb3473-0b1e-498a-bbcd-70b6660e763b" (UID: "55cb3473-0b1e-498a-bbcd-70b6660e763b"). InnerVolumeSpecName "isvc-sklearn-scale-raw-c3450-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 16:17:45.203484 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:17:45.203463    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55cb3473-0b1e-498a-bbcd-70b6660e763b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "55cb3473-0b1e-498a-bbcd-70b6660e763b" (UID: "55cb3473-0b1e-498a-bbcd-70b6660e763b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 16:17:45.203613 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:17:45.203598    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55cb3473-0b1e-498a-bbcd-70b6660e763b-kube-api-access-66lnc" (OuterVolumeSpecName: "kube-api-access-66lnc") pod "55cb3473-0b1e-498a-bbcd-70b6660e763b" (UID: "55cb3473-0b1e-498a-bbcd-70b6660e763b"). InnerVolumeSpecName "kube-api-access-66lnc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 16:17:45.302438 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:17:45.302379    2556 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/55cb3473-0b1e-498a-bbcd-70b6660e763b-kserve-provision-location\") on node \"ip-10-0-134-19.ec2.internal\" DevicePath \"\""
May 26 16:17:45.302438 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:17:45.302402    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-66lnc\" (UniqueName: \"kubernetes.io/projected/55cb3473-0b1e-498a-bbcd-70b6660e763b-kube-api-access-66lnc\") on node \"ip-10-0-134-19.ec2.internal\" DevicePath \"\""
May 26 16:17:45.302438 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:17:45.302412    2556 reconciler_common.go:299] "Volume detached for volume \"isvc-sklearn-scale-raw-c3450-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/55cb3473-0b1e-498a-bbcd-70b6660e763b-isvc-sklearn-scale-raw-c3450-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-19.ec2.internal\" DevicePath \"\""
May 26 16:17:45.302438 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:17:45.302423    2556 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/55cb3473-0b1e-498a-bbcd-70b6660e763b-proxy-tls\") on node \"ip-10-0-134-19.ec2.internal\" DevicePath \"\""
May 26 16:17:45.983350 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:17:45.983312    2556 generic.go:358] "Generic (PLEG): container finished" podID="55cb3473-0b1e-498a-bbcd-70b6660e763b" containerID="77b60e85ad6b046c827fcc39e2b396bf1d0057a9021964260d202bc5a52b8a08" exitCode=0
May 26 16:17:45.983519 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:17:45.983369    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-c3450-predictor-5c6fc6c8c5-fwm8n" event={"ID":"55cb3473-0b1e-498a-bbcd-70b6660e763b","Type":"ContainerDied","Data":"77b60e85ad6b046c827fcc39e2b396bf1d0057a9021964260d202bc5a52b8a08"}
May 26 16:17:45.983519 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:17:45.983393    2556 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-c3450-predictor-5c6fc6c8c5-fwm8n"
May 26 16:17:45.983519 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:17:45.983404    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-c3450-predictor-5c6fc6c8c5-fwm8n" event={"ID":"55cb3473-0b1e-498a-bbcd-70b6660e763b","Type":"ContainerDied","Data":"3103abe9e3f1dc728b0b298f168c420130bf3ed76b739204a4ea78be86e3bf72"}
May 26 16:17:45.983519 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:17:45.983426    2556 scope.go:117] "RemoveContainer" containerID="560fa608bf89d80410d6df64fdb2f2a9512a0a0b6bf63a7bb6e5649ba748cdc7"
May 26 16:17:45.991447 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:17:45.991298    2556 scope.go:117] "RemoveContainer" containerID="77b60e85ad6b046c827fcc39e2b396bf1d0057a9021964260d202bc5a52b8a08"
May 26 16:17:45.997969 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:17:45.997952    2556 scope.go:117] "RemoveContainer" containerID="6545e19f0ade09a5ce53fcaf15a2d6c7cb2a42e4cf56de87cb7d67781dbfb00c"
May 26 16:17:46.004134 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:17:46.004120    2556 scope.go:117] "RemoveContainer" containerID="560fa608bf89d80410d6df64fdb2f2a9512a0a0b6bf63a7bb6e5649ba748cdc7"
May 26 16:17:46.004372 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:17:46.004353    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"560fa608bf89d80410d6df64fdb2f2a9512a0a0b6bf63a7bb6e5649ba748cdc7\": container with ID starting with 560fa608bf89d80410d6df64fdb2f2a9512a0a0b6bf63a7bb6e5649ba748cdc7 not found: ID does not exist" containerID="560fa608bf89d80410d6df64fdb2f2a9512a0a0b6bf63a7bb6e5649ba748cdc7"
May 26 16:17:46.004435 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:17:46.004379    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"560fa608bf89d80410d6df64fdb2f2a9512a0a0b6bf63a7bb6e5649ba748cdc7"} err="failed to get container status \"560fa608bf89d80410d6df64fdb2f2a9512a0a0b6bf63a7bb6e5649ba748cdc7\": rpc error: code = NotFound desc = could not find container \"560fa608bf89d80410d6df64fdb2f2a9512a0a0b6bf63a7bb6e5649ba748cdc7\": container with ID starting with 560fa608bf89d80410d6df64fdb2f2a9512a0a0b6bf63a7bb6e5649ba748cdc7 not found: ID does not exist"
May 26 16:17:46.004435 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:17:46.004396    2556 scope.go:117] "RemoveContainer" containerID="77b60e85ad6b046c827fcc39e2b396bf1d0057a9021964260d202bc5a52b8a08"
May 26 16:17:46.004599 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:17:46.004583    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77b60e85ad6b046c827fcc39e2b396bf1d0057a9021964260d202bc5a52b8a08\": container with ID starting with 77b60e85ad6b046c827fcc39e2b396bf1d0057a9021964260d202bc5a52b8a08 not found: ID does not exist" containerID="77b60e85ad6b046c827fcc39e2b396bf1d0057a9021964260d202bc5a52b8a08"
May 26 16:17:46.004638 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:17:46.004605    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77b60e85ad6b046c827fcc39e2b396bf1d0057a9021964260d202bc5a52b8a08"} err="failed to get container status \"77b60e85ad6b046c827fcc39e2b396bf1d0057a9021964260d202bc5a52b8a08\": rpc error: code = NotFound desc = could not find container \"77b60e85ad6b046c827fcc39e2b396bf1d0057a9021964260d202bc5a52b8a08\": container with ID starting with 77b60e85ad6b046c827fcc39e2b396bf1d0057a9021964260d202bc5a52b8a08 not found: ID does not exist"
May 26 16:17:46.004638 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:17:46.004619    2556 scope.go:117] "RemoveContainer" containerID="6545e19f0ade09a5ce53fcaf15a2d6c7cb2a42e4cf56de87cb7d67781dbfb00c"
May 26 16:17:46.004827 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:17:46.004812    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6545e19f0ade09a5ce53fcaf15a2d6c7cb2a42e4cf56de87cb7d67781dbfb00c\": container with ID starting with 6545e19f0ade09a5ce53fcaf15a2d6c7cb2a42e4cf56de87cb7d67781dbfb00c not found: ID does not exist" containerID="6545e19f0ade09a5ce53fcaf15a2d6c7cb2a42e4cf56de87cb7d67781dbfb00c"
May 26 16:17:46.004871 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:17:46.004829    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6545e19f0ade09a5ce53fcaf15a2d6c7cb2a42e4cf56de87cb7d67781dbfb00c"} err="failed to get container status \"6545e19f0ade09a5ce53fcaf15a2d6c7cb2a42e4cf56de87cb7d67781dbfb00c\": rpc error: code = NotFound desc = could not find container \"6545e19f0ade09a5ce53fcaf15a2d6c7cb2a42e4cf56de87cb7d67781dbfb00c\": container with ID starting with 6545e19f0ade09a5ce53fcaf15a2d6c7cb2a42e4cf56de87cb7d67781dbfb00c not found: ID does not exist"
May 26 16:17:46.013228 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:17:46.013207    2556 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-c3450-predictor-5c6fc6c8c5-fwm8n"]
May 26 16:17:46.021251 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:17:46.021229    2556 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-c3450-predictor-5c6fc6c8c5-fwm8n"]
May 26 16:17:46.771215 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:17:46.771175    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="55cb3473-0b1e-498a-bbcd-70b6660e763b" path="/var/lib/kubelet/pods/55cb3473-0b1e-498a-bbcd-70b6660e763b/volumes"
May 26 16:17:48.980325 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:17:48.980298    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-ba2514-predictor-686dd7994-vcx9l"
May 26 16:17:48.980868 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:17:48.980842    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-ba2514-predictor-686dd7994-vcx9l" podUID="59ce4742-c702-4783-aca6-e7b3ab74513f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
May 26 16:17:58.981424 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:17:58.981388    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-ba2514-predictor-686dd7994-vcx9l" podUID="59ce4742-c702-4783-aca6-e7b3ab74513f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
May 26 16:18:08.981708 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:18:08.981664    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-ba2514-predictor-686dd7994-vcx9l" podUID="59ce4742-c702-4783-aca6-e7b3ab74513f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
May 26 16:18:18.980930 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:18:18.980892    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-ba2514-predictor-686dd7994-vcx9l" podUID="59ce4742-c702-4783-aca6-e7b3ab74513f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
May 26 16:18:28.981438 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:18:28.981357    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-ba2514-predictor-686dd7994-vcx9l" podUID="59ce4742-c702-4783-aca6-e7b3ab74513f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
May 26 16:18:38.981311 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:18:38.981272    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-ba2514-predictor-686dd7994-vcx9l" podUID="59ce4742-c702-4783-aca6-e7b3ab74513f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
May 26 16:18:48.981234 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:18:48.981206    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-ba2514-predictor-686dd7994-vcx9l"
May 26 16:18:56.699457 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:18:56.699420    2556 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-ba2514-predictor-6b8dc68545-bl4zz"]
May 26 16:18:56.699869 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:18:56.699855    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="55cb3473-0b1e-498a-bbcd-70b6660e763b" containerName="storage-initializer"
May 26 16:18:56.699916 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:18:56.699872    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="55cb3473-0b1e-498a-bbcd-70b6660e763b" containerName="storage-initializer"
May 26 16:18:56.699916 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:18:56.699887    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="55cb3473-0b1e-498a-bbcd-70b6660e763b" containerName="kserve-container"
May 26 16:18:56.699916 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:18:56.699896    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="55cb3473-0b1e-498a-bbcd-70b6660e763b" containerName="kserve-container"
May 26 16:18:56.700010 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:18:56.699923    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="55cb3473-0b1e-498a-bbcd-70b6660e763b" containerName="kube-rbac-proxy"
May 26 16:18:56.700010 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:18:56.699932    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="55cb3473-0b1e-498a-bbcd-70b6660e763b" containerName="kube-rbac-proxy"
May 26 16:18:56.700010 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:18:56.700001    2556 memory_manager.go:356] "RemoveStaleState removing state" podUID="55cb3473-0b1e-498a-bbcd-70b6660e763b" containerName="kube-rbac-proxy"
May 26 16:18:56.700126 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:18:56.700014    2556 memory_manager.go:356] "RemoveStaleState removing state" podUID="55cb3473-0b1e-498a-bbcd-70b6660e763b" containerName="kserve-container"
May 26 16:18:56.704461 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:18:56.704441    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-ba2514-predictor-6b8dc68545-bl4zz"
May 26 16:18:56.707415 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:18:56.707364    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-ba2514-kube-rbac-proxy-sar-config\""
May 26 16:18:56.707589 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:18:56.707533    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
May 26 16:18:56.707845 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:18:56.707828    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-ba2514-predictor-serving-cert\""
May 26 16:18:56.708698 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:18:56.708681    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-secret-ba2514\""
May 26 16:18:56.708790 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:18:56.708684    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-sa-ba2514-dockercfg-5sh4f\""
May 26 16:18:56.714622 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:18:56.714603    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-ba2514-predictor-6b8dc68545-bl4zz"]
May 26 16:18:56.807086 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:18:56.807060    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c8fe44d0-deee-4c97-943f-a7a0a51c500a-proxy-tls\") pod \"isvc-secondary-ba2514-predictor-6b8dc68545-bl4zz\" (UID: \"c8fe44d0-deee-4c97-943f-a7a0a51c500a\") " pod="kserve-ci-e2e-test/isvc-secondary-ba2514-predictor-6b8dc68545-bl4zz"
May 26 16:18:56.807244 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:18:56.807136    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7qzb\" (UniqueName: \"kubernetes.io/projected/c8fe44d0-deee-4c97-943f-a7a0a51c500a-kube-api-access-c7qzb\") pod \"isvc-secondary-ba2514-predictor-6b8dc68545-bl4zz\" (UID: \"c8fe44d0-deee-4c97-943f-a7a0a51c500a\") " pod="kserve-ci-e2e-test/isvc-secondary-ba2514-predictor-6b8dc68545-bl4zz"
May 26 16:18:56.807244 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:18:56.807162    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c8fe44d0-deee-4c97-943f-a7a0a51c500a-kserve-provision-location\") pod \"isvc-secondary-ba2514-predictor-6b8dc68545-bl4zz\" (UID: \"c8fe44d0-deee-4c97-943f-a7a0a51c500a\") " pod="kserve-ci-e2e-test/isvc-secondary-ba2514-predictor-6b8dc68545-bl4zz"
May 26 16:18:56.807244 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:18:56.807191    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c8fe44d0-deee-4c97-943f-a7a0a51c500a-cabundle-cert\") pod \"isvc-secondary-ba2514-predictor-6b8dc68545-bl4zz\" (UID: \"c8fe44d0-deee-4c97-943f-a7a0a51c500a\") " pod="kserve-ci-e2e-test/isvc-secondary-ba2514-predictor-6b8dc68545-bl4zz"
May 26 16:18:56.807354 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:18:56.807248    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-secondary-ba2514-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c8fe44d0-deee-4c97-943f-a7a0a51c500a-isvc-secondary-ba2514-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-ba2514-predictor-6b8dc68545-bl4zz\" (UID: \"c8fe44d0-deee-4c97-943f-a7a0a51c500a\") " pod="kserve-ci-e2e-test/isvc-secondary-ba2514-predictor-6b8dc68545-bl4zz"
May 26 16:18:56.908363 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:18:56.908330    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-c7qzb\" (UniqueName: \"kubernetes.io/projected/c8fe44d0-deee-4c97-943f-a7a0a51c500a-kube-api-access-c7qzb\") pod \"isvc-secondary-ba2514-predictor-6b8dc68545-bl4zz\" (UID: \"c8fe44d0-deee-4c97-943f-a7a0a51c500a\") " pod="kserve-ci-e2e-test/isvc-secondary-ba2514-predictor-6b8dc68545-bl4zz"
May 26 16:18:56.908363 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:18:56.908363    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c8fe44d0-deee-4c97-943f-a7a0a51c500a-kserve-provision-location\") pod \"isvc-secondary-ba2514-predictor-6b8dc68545-bl4zz\" (UID: \"c8fe44d0-deee-4c97-943f-a7a0a51c500a\") " pod="kserve-ci-e2e-test/isvc-secondary-ba2514-predictor-6b8dc68545-bl4zz"
May 26 16:18:56.908578 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:18:56.908407    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c8fe44d0-deee-4c97-943f-a7a0a51c500a-cabundle-cert\") pod \"isvc-secondary-ba2514-predictor-6b8dc68545-bl4zz\" (UID: \"c8fe44d0-deee-4c97-943f-a7a0a51c500a\") " pod="kserve-ci-e2e-test/isvc-secondary-ba2514-predictor-6b8dc68545-bl4zz"
May 26 16:18:56.908578 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:18:56.908438    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-secondary-ba2514-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c8fe44d0-deee-4c97-943f-a7a0a51c500a-isvc-secondary-ba2514-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-ba2514-predictor-6b8dc68545-bl4zz\" (UID: \"c8fe44d0-deee-4c97-943f-a7a0a51c500a\") " pod="kserve-ci-e2e-test/isvc-secondary-ba2514-predictor-6b8dc68545-bl4zz"
May 26 16:18:56.908578 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:18:56.908493    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c8fe44d0-deee-4c97-943f-a7a0a51c500a-proxy-tls\") pod \"isvc-secondary-ba2514-predictor-6b8dc68545-bl4zz\" (UID: \"c8fe44d0-deee-4c97-943f-a7a0a51c500a\") " pod="kserve-ci-e2e-test/isvc-secondary-ba2514-predictor-6b8dc68545-bl4zz"
May 26 16:18:56.908846 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:18:56.908822    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c8fe44d0-deee-4c97-943f-a7a0a51c500a-kserve-provision-location\") pod \"isvc-secondary-ba2514-predictor-6b8dc68545-bl4zz\" (UID: \"c8fe44d0-deee-4c97-943f-a7a0a51c500a\") " pod="kserve-ci-e2e-test/isvc-secondary-ba2514-predictor-6b8dc68545-bl4zz"
May 26 16:18:56.909113 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:18:56.909074    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c8fe44d0-deee-4c97-943f-a7a0a51c500a-cabundle-cert\") pod \"isvc-secondary-ba2514-predictor-6b8dc68545-bl4zz\" (UID: \"c8fe44d0-deee-4c97-943f-a7a0a51c500a\") " pod="kserve-ci-e2e-test/isvc-secondary-ba2514-predictor-6b8dc68545-bl4zz"
May 26 16:18:56.909180 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:18:56.909161    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-secondary-ba2514-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c8fe44d0-deee-4c97-943f-a7a0a51c500a-isvc-secondary-ba2514-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-ba2514-predictor-6b8dc68545-bl4zz\" (UID: \"c8fe44d0-deee-4c97-943f-a7a0a51c500a\") " pod="kserve-ci-e2e-test/isvc-secondary-ba2514-predictor-6b8dc68545-bl4zz"
May 26 16:18:56.910919 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:18:56.910898    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c8fe44d0-deee-4c97-943f-a7a0a51c500a-proxy-tls\") pod \"isvc-secondary-ba2514-predictor-6b8dc68545-bl4zz\" (UID: \"c8fe44d0-deee-4c97-943f-a7a0a51c500a\") " pod="kserve-ci-e2e-test/isvc-secondary-ba2514-predictor-6b8dc68545-bl4zz"
May 26 16:18:56.918936 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:18:56.918917    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7qzb\" (UniqueName: \"kubernetes.io/projected/c8fe44d0-deee-4c97-943f-a7a0a51c500a-kube-api-access-c7qzb\") pod \"isvc-secondary-ba2514-predictor-6b8dc68545-bl4zz\" (UID: \"c8fe44d0-deee-4c97-943f-a7a0a51c500a\") " pod="kserve-ci-e2e-test/isvc-secondary-ba2514-predictor-6b8dc68545-bl4zz"
May 26 16:18:57.016458 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:18:57.016399    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-ba2514-predictor-6b8dc68545-bl4zz"
May 26 16:18:57.140329 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:18:57.140299    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-ba2514-predictor-6b8dc68545-bl4zz"]
May 26 16:18:57.143546 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:18:57.143523    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8fe44d0_deee_4c97_943f_a7a0a51c500a.slice/crio-970eb5d4e33e2f5f9f155b061d3ca8c69512a05fd8aa63817a5cea633ea40dfc WatchSource:0}: Error finding container 970eb5d4e33e2f5f9f155b061d3ca8c69512a05fd8aa63817a5cea633ea40dfc: Status 404 returned error can't find the container with id 970eb5d4e33e2f5f9f155b061d3ca8c69512a05fd8aa63817a5cea633ea40dfc
May 26 16:18:57.196304 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:18:57.196278    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-ba2514-predictor-6b8dc68545-bl4zz" event={"ID":"c8fe44d0-deee-4c97-943f-a7a0a51c500a","Type":"ContainerStarted","Data":"970eb5d4e33e2f5f9f155b061d3ca8c69512a05fd8aa63817a5cea633ea40dfc"}
May 26 16:18:58.199910 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:18:58.199869    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-ba2514-predictor-6b8dc68545-bl4zz" event={"ID":"c8fe44d0-deee-4c97-943f-a7a0a51c500a","Type":"ContainerStarted","Data":"a981941668dc96871bd521035daa5f466c6014f2808eaf23d150e958c9302160"}
May 26 16:19:03.217221 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:19:03.217190    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-ba2514-predictor-6b8dc68545-bl4zz_c8fe44d0-deee-4c97-943f-a7a0a51c500a/storage-initializer/0.log"
May 26 16:19:03.217590 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:19:03.217230    2556 generic.go:358] "Generic (PLEG): container finished" podID="c8fe44d0-deee-4c97-943f-a7a0a51c500a" containerID="a981941668dc96871bd521035daa5f466c6014f2808eaf23d150e958c9302160" exitCode=1
May 26 16:19:03.217590 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:19:03.217274    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-ba2514-predictor-6b8dc68545-bl4zz" event={"ID":"c8fe44d0-deee-4c97-943f-a7a0a51c500a","Type":"ContainerDied","Data":"a981941668dc96871bd521035daa5f466c6014f2808eaf23d150e958c9302160"}
May 26 16:19:04.222331 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:19:04.222299    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-ba2514-predictor-6b8dc68545-bl4zz_c8fe44d0-deee-4c97-943f-a7a0a51c500a/storage-initializer/0.log"
May 26 16:19:04.222705 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:19:04.222417    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-ba2514-predictor-6b8dc68545-bl4zz" event={"ID":"c8fe44d0-deee-4c97-943f-a7a0a51c500a","Type":"ContainerStarted","Data":"f990244c3c51a778454f3101a69a5e23d1244bbc729590ce792c233fb5d43767"}
May 26 16:19:08.237877 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:19:08.237845    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-ba2514-predictor-6b8dc68545-bl4zz_c8fe44d0-deee-4c97-943f-a7a0a51c500a/storage-initializer/1.log"
May 26 16:19:08.238275 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:19:08.238214    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-ba2514-predictor-6b8dc68545-bl4zz_c8fe44d0-deee-4c97-943f-a7a0a51c500a/storage-initializer/0.log"
May 26 16:19:08.238275 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:19:08.238246    2556 generic.go:358] "Generic (PLEG): container finished" podID="c8fe44d0-deee-4c97-943f-a7a0a51c500a" containerID="f990244c3c51a778454f3101a69a5e23d1244bbc729590ce792c233fb5d43767" exitCode=1
May 26 16:19:08.238376 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:19:08.238283    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-ba2514-predictor-6b8dc68545-bl4zz" event={"ID":"c8fe44d0-deee-4c97-943f-a7a0a51c500a","Type":"ContainerDied","Data":"f990244c3c51a778454f3101a69a5e23d1244bbc729590ce792c233fb5d43767"}
May 26 16:19:08.238376 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:19:08.238311    2556 scope.go:117] "RemoveContainer" containerID="a981941668dc96871bd521035daa5f466c6014f2808eaf23d150e958c9302160"
May 26 16:19:08.238727 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:19:08.238708    2556 scope.go:117] "RemoveContainer" containerID="a981941668dc96871bd521035daa5f466c6014f2808eaf23d150e958c9302160"
May 26 16:19:08.248728 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:19:08.248701    2556 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-ba2514-predictor-6b8dc68545-bl4zz_kserve-ci-e2e-test_c8fe44d0-deee-4c97-943f-a7a0a51c500a_0 in pod sandbox 970eb5d4e33e2f5f9f155b061d3ca8c69512a05fd8aa63817a5cea633ea40dfc from index: no such id: 'a981941668dc96871bd521035daa5f466c6014f2808eaf23d150e958c9302160'" containerID="a981941668dc96871bd521035daa5f466c6014f2808eaf23d150e958c9302160"
May 26 16:19:08.248826 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:19:08.248738    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a981941668dc96871bd521035daa5f466c6014f2808eaf23d150e958c9302160"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-ba2514-predictor-6b8dc68545-bl4zz_kserve-ci-e2e-test_c8fe44d0-deee-4c97-943f-a7a0a51c500a_0 in pod sandbox 970eb5d4e33e2f5f9f155b061d3ca8c69512a05fd8aa63817a5cea633ea40dfc from index: no such id: 'a981941668dc96871bd521035daa5f466c6014f2808eaf23d150e958c9302160'"
May 26 16:19:08.248957 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:19:08.248929    2556 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-secondary-ba2514-predictor-6b8dc68545-bl4zz_kserve-ci-e2e-test(c8fe44d0-deee-4c97-943f-a7a0a51c500a)\"" pod="kserve-ci-e2e-test/isvc-secondary-ba2514-predictor-6b8dc68545-bl4zz" podUID="c8fe44d0-deee-4c97-943f-a7a0a51c500a"
May 26 16:19:09.242536 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:19:09.242508    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-ba2514-predictor-6b8dc68545-bl4zz_c8fe44d0-deee-4c97-943f-a7a0a51c500a/storage-initializer/1.log"
May 26 16:19:14.785348 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:19:14.785318    2556 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-ba2514-predictor-6b8dc68545-bl4zz"]
May 26 16:19:14.841156 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:19:14.841124    2556 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-ba2514-predictor-686dd7994-vcx9l"]
May 26 16:19:14.841671 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:19:14.841514    2556 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-ba2514-predictor-686dd7994-vcx9l" podUID="59ce4742-c702-4783-aca6-e7b3ab74513f" containerName="kserve-container" containerID="cri-o://1194514042a2ea15d164f1519bfc187c98e55f77eb3c9971377f8d0d6cd968d6" gracePeriod=30
May 26 16:19:14.841671 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:19:14.841577    2556 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-ba2514-predictor-686dd7994-vcx9l" podUID="59ce4742-c702-4783-aca6-e7b3ab74513f" containerName="kube-rbac-proxy" containerID="cri-o://58a91c81cd6f4d154871c9fea9b0f876ff59f487ccb0de2cfc49f44986155ed7" gracePeriod=30
May 26 16:19:14.888582 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:19:14.888552    2556 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-3dec36-predictor-6f8b45dcff-p5629"]
May 26 16:19:14.893104 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:19:14.893079    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-3dec36-predictor-6f8b45dcff-p5629"
May 26 16:19:14.896061 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:19:14.896037    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-3dec36-kube-rbac-proxy-sar-config\""
May 26 16:19:14.896197 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:19:14.896167    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-secret-3dec36\""
May 26 16:19:14.896258 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:19:14.896209    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-sa-3dec36-dockercfg-qtmpn\""
May 26 16:19:14.897380 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:19:14.897353    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-3dec36-predictor-serving-cert\""
May 26 16:19:14.902873 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:19:14.902848    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-3dec36-predictor-6f8b45dcff-p5629"]
May 26 16:19:14.958561 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:19:14.958544    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zk4v6\" (UniqueName: \"kubernetes.io/projected/81669d50-52be-4ca4-98e5-7d3de6d4e7cc-kube-api-access-zk4v6\") pod \"isvc-init-fail-3dec36-predictor-6f8b45dcff-p5629\" (UID: \"81669d50-52be-4ca4-98e5-7d3de6d4e7cc\") " pod="kserve-ci-e2e-test/isvc-init-fail-3dec36-predictor-6f8b45dcff-p5629"
May 26 16:19:14.958641 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:19:14.958583    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-init-fail-3dec36-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/81669d50-52be-4ca4-98e5-7d3de6d4e7cc-isvc-init-fail-3dec36-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-3dec36-predictor-6f8b45dcff-p5629\" (UID: \"81669d50-52be-4ca4-98e5-7d3de6d4e7cc\") " pod="kserve-ci-e2e-test/isvc-init-fail-3dec36-predictor-6f8b45dcff-p5629"
May 26 16:19:14.958641 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:19:14.958609    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/81669d50-52be-4ca4-98e5-7d3de6d4e7cc-kserve-provision-location\") pod \"isvc-init-fail-3dec36-predictor-6f8b45dcff-p5629\" (UID: \"81669d50-52be-4ca4-98e5-7d3de6d4e7cc\") " pod="kserve-ci-e2e-test/isvc-init-fail-3dec36-predictor-6f8b45dcff-p5629"
May 26 16:19:14.958743 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:19:14.958710    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/81669d50-52be-4ca4-98e5-7d3de6d4e7cc-proxy-tls\") pod \"isvc-init-fail-3dec36-predictor-6f8b45dcff-p5629\" (UID: \"81669d50-52be-4ca4-98e5-7d3de6d4e7cc\") " pod="kserve-ci-e2e-test/isvc-init-fail-3dec36-predictor-6f8b45dcff-p5629"
May 26 16:19:14.958798 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:19:14.958754    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/81669d50-52be-4ca4-98e5-7d3de6d4e7cc-cabundle-cert\") pod \"isvc-init-fail-3dec36-predictor-6f8b45dcff-p5629\" (UID: \"81669d50-52be-4ca4-98e5-7d3de6d4e7cc\") " pod="kserve-ci-e2e-test/isvc-init-fail-3dec36-predictor-6f8b45dcff-p5629"
May 26 16:19:14.961070 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:19:14.961055    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-ba2514-predictor-6b8dc68545-bl4zz_c8fe44d0-deee-4c97-943f-a7a0a51c500a/storage-initializer/1.log"
May 26 16:19:14.961220 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:19:14.961181    2556 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-ba2514-predictor-6b8dc68545-bl4zz"
May 26 16:19:15.059940 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:19:15.059872    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c7qzb\" (UniqueName: \"kubernetes.io/projected/c8fe44d0-deee-4c97-943f-a7a0a51c500a-kube-api-access-c7qzb\") pod \"c8fe44d0-deee-4c97-943f-a7a0a51c500a\" (UID: \"c8fe44d0-deee-4c97-943f-a7a0a51c500a\") "
May 26 16:19:15.059940 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:19:15.059921    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c8fe44d0-deee-4c97-943f-a7a0a51c500a-proxy-tls\") pod \"c8fe44d0-deee-4c97-943f-a7a0a51c500a\" (UID: \"c8fe44d0-deee-4c97-943f-a7a0a51c500a\") "
May 26 16:19:15.060148 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:19:15.059954    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c8fe44d0-deee-4c97-943f-a7a0a51c500a-kserve-provision-location\") pod \"c8fe44d0-deee-4c97-943f-a7a0a51c500a\" (UID: \"c8fe44d0-deee-4c97-943f-a7a0a51c500a\") "
May 26 16:19:15.060148 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:19:15.059992    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c8fe44d0-deee-4c97-943f-a7a0a51c500a-cabundle-cert\") pod \"c8fe44d0-deee-4c97-943f-a7a0a51c500a\" (UID: \"c8fe44d0-deee-4c97-943f-a7a0a51c500a\") "
May 26 16:19:15.060148 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:19:15.060021    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-secondary-ba2514-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c8fe44d0-deee-4c97-943f-a7a0a51c500a-isvc-secondary-ba2514-kube-rbac-proxy-sar-config\") pod \"c8fe44d0-deee-4c97-943f-a7a0a51c500a\" (UID: \"c8fe44d0-deee-4c97-943f-a7a0a51c500a\") "
May 26 16:19:15.060148 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:19:15.060136    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-zk4v6\" (UniqueName: \"kubernetes.io/projected/81669d50-52be-4ca4-98e5-7d3de6d4e7cc-kube-api-access-zk4v6\") pod \"isvc-init-fail-3dec36-predictor-6f8b45dcff-p5629\" (UID: \"81669d50-52be-4ca4-98e5-7d3de6d4e7cc\") " pod="kserve-ci-e2e-test/isvc-init-fail-3dec36-predictor-6f8b45dcff-p5629"
May 26 16:19:15.060612 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:19:15.060178    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-init-fail-3dec36-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/81669d50-52be-4ca4-98e5-7d3de6d4e7cc-isvc-init-fail-3dec36-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-3dec36-predictor-6f8b45dcff-p5629\" (UID: \"81669d50-52be-4ca4-98e5-7d3de6d4e7cc\") " pod="kserve-ci-e2e-test/isvc-init-fail-3dec36-predictor-6f8b45dcff-p5629"
May 26 16:19:15.060612 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:19:15.060217    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/81669d50-52be-4ca4-98e5-7d3de6d4e7cc-kserve-provision-location\") pod \"isvc-init-fail-3dec36-predictor-6f8b45dcff-p5629\" (UID: \"81669d50-52be-4ca4-98e5-7d3de6d4e7cc\") " pod="kserve-ci-e2e-test/isvc-init-fail-3dec36-predictor-6f8b45dcff-p5629"
May 26 16:19:15.060612 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:19:15.060259    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/81669d50-52be-4ca4-98e5-7d3de6d4e7cc-proxy-tls\") pod \"isvc-init-fail-3dec36-predictor-6f8b45dcff-p5629\" (UID: \"81669d50-52be-4ca4-98e5-7d3de6d4e7cc\") " pod="kserve-ci-e2e-test/isvc-init-fail-3dec36-predictor-6f8b45dcff-p5629"
May 26 16:19:15.060612 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:19:15.060302    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/81669d50-52be-4ca4-98e5-7d3de6d4e7cc-cabundle-cert\") pod \"isvc-init-fail-3dec36-predictor-6f8b45dcff-p5629\" (UID: \"81669d50-52be-4ca4-98e5-7d3de6d4e7cc\") " pod="kserve-ci-e2e-test/isvc-init-fail-3dec36-predictor-6f8b45dcff-p5629"
May 26 16:19:15.060612 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:19:15.060387    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8fe44d0-deee-4c97-943f-a7a0a51c500a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c8fe44d0-deee-4c97-943f-a7a0a51c500a" (UID: "c8fe44d0-deee-4c97-943f-a7a0a51c500a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 16:19:15.060612 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:19:15.060476    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8fe44d0-deee-4c97-943f-a7a0a51c500a-isvc-secondary-ba2514-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-secondary-ba2514-kube-rbac-proxy-sar-config") pod "c8fe44d0-deee-4c97-943f-a7a0a51c500a" (UID: "c8fe44d0-deee-4c97-943f-a7a0a51c500a"). InnerVolumeSpecName "isvc-secondary-ba2514-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 16:19:15.060875 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:19:15.060747    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8fe44d0-deee-4c97-943f-a7a0a51c500a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c8fe44d0-deee-4c97-943f-a7a0a51c500a" (UID: "c8fe44d0-deee-4c97-943f-a7a0a51c500a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 16:19:15.060962 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:19:15.060904    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/81669d50-52be-4ca4-98e5-7d3de6d4e7cc-kserve-provision-location\") pod \"isvc-init-fail-3dec36-predictor-6f8b45dcff-p5629\" (UID: \"81669d50-52be-4ca4-98e5-7d3de6d4e7cc\") " pod="kserve-ci-e2e-test/isvc-init-fail-3dec36-predictor-6f8b45dcff-p5629"
May 26 16:19:15.061017 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:19:15.060997    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/81669d50-52be-4ca4-98e5-7d3de6d4e7cc-cabundle-cert\") pod \"isvc-init-fail-3dec36-predictor-6f8b45dcff-p5629\" (UID: \"81669d50-52be-4ca4-98e5-7d3de6d4e7cc\") " pod="kserve-ci-e2e-test/isvc-init-fail-3dec36-predictor-6f8b45dcff-p5629"
May 26 16:19:15.061163 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:19:15.061136    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-init-fail-3dec36-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/81669d50-52be-4ca4-98e5-7d3de6d4e7cc-isvc-init-fail-3dec36-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-3dec36-predictor-6f8b45dcff-p5629\" (UID: \"81669d50-52be-4ca4-98e5-7d3de6d4e7cc\") " pod="kserve-ci-e2e-test/isvc-init-fail-3dec36-predictor-6f8b45dcff-p5629"
May 26 16:19:15.062188 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:19:15.062163    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8fe44d0-deee-4c97-943f-a7a0a51c500a-kube-api-access-c7qzb" (OuterVolumeSpecName: "kube-api-access-c7qzb") pod "c8fe44d0-deee-4c97-943f-a7a0a51c500a" (UID: "c8fe44d0-deee-4c97-943f-a7a0a51c500a"). InnerVolumeSpecName "kube-api-access-c7qzb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 16:19:15.062320 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:19:15.062305    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8fe44d0-deee-4c97-943f-a7a0a51c500a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c8fe44d0-deee-4c97-943f-a7a0a51c500a" (UID: "c8fe44d0-deee-4c97-943f-a7a0a51c500a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 16:19:15.062937 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:19:15.062918    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/81669d50-52be-4ca4-98e5-7d3de6d4e7cc-proxy-tls\") pod \"isvc-init-fail-3dec36-predictor-6f8b45dcff-p5629\" (UID: \"81669d50-52be-4ca4-98e5-7d3de6d4e7cc\") " pod="kserve-ci-e2e-test/isvc-init-fail-3dec36-predictor-6f8b45dcff-p5629"
May 26 16:19:15.071148 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:19:15.071130    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zk4v6\" (UniqueName: \"kubernetes.io/projected/81669d50-52be-4ca4-98e5-7d3de6d4e7cc-kube-api-access-zk4v6\") pod \"isvc-init-fail-3dec36-predictor-6f8b45dcff-p5629\" (UID: \"81669d50-52be-4ca4-98e5-7d3de6d4e7cc\") " pod="kserve-ci-e2e-test/isvc-init-fail-3dec36-predictor-6f8b45dcff-p5629"
May 26 16:19:15.161463 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:19:15.161439    2556 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c8fe44d0-deee-4c97-943f-a7a0a51c500a-cabundle-cert\") on node \"ip-10-0-134-19.ec2.internal\" DevicePath \"\""
May 26 16:19:15.161463 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:19:15.161462    2556 reconciler_common.go:299] "Volume detached for volume \"isvc-secondary-ba2514-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c8fe44d0-deee-4c97-943f-a7a0a51c500a-isvc-secondary-ba2514-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-19.ec2.internal\" DevicePath \"\""
May 26 16:19:15.161577 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:19:15.161478    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c7qzb\" (UniqueName: \"kubernetes.io/projected/c8fe44d0-deee-4c97-943f-a7a0a51c500a-kube-api-access-c7qzb\") on node \"ip-10-0-134-19.ec2.internal\" DevicePath \"\""
May 26 16:19:15.161577 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:19:15.161491    2556 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c8fe44d0-deee-4c97-943f-a7a0a51c500a-proxy-tls\") on node \"ip-10-0-134-19.ec2.internal\" DevicePath \"\""
May 26 16:19:15.161577 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:19:15.161506    2556 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c8fe44d0-deee-4c97-943f-a7a0a51c500a-kserve-provision-location\") on node \"ip-10-0-134-19.ec2.internal\" DevicePath \"\""
May 26 16:19:15.204458 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:19:15.204433    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-3dec36-predictor-6f8b45dcff-p5629"
May 26 16:19:15.262167 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:19:15.262146    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-ba2514-predictor-6b8dc68545-bl4zz_c8fe44d0-deee-4c97-943f-a7a0a51c500a/storage-initializer/1.log"
May 26 16:19:15.262474 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:19:15.262327    2556 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-ba2514-predictor-6b8dc68545-bl4zz"
May 26 16:19:15.262474 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:19:15.262335    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-ba2514-predictor-6b8dc68545-bl4zz" event={"ID":"c8fe44d0-deee-4c97-943f-a7a0a51c500a","Type":"ContainerDied","Data":"970eb5d4e33e2f5f9f155b061d3ca8c69512a05fd8aa63817a5cea633ea40dfc"}
May 26 16:19:15.262474 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:19:15.262375    2556 scope.go:117] "RemoveContainer" containerID="f990244c3c51a778454f3101a69a5e23d1244bbc729590ce792c233fb5d43767"
May 26 16:19:15.266656 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:19:15.266489    2556 generic.go:358] "Generic (PLEG): container finished" podID="59ce4742-c702-4783-aca6-e7b3ab74513f" containerID="58a91c81cd6f4d154871c9fea9b0f876ff59f487ccb0de2cfc49f44986155ed7" exitCode=2
May 26 16:19:15.266656 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:19:15.266613    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-ba2514-predictor-686dd7994-vcx9l" event={"ID":"59ce4742-c702-4783-aca6-e7b3ab74513f","Type":"ContainerDied","Data":"58a91c81cd6f4d154871c9fea9b0f876ff59f487ccb0de2cfc49f44986155ed7"}
May 26 16:19:15.318373 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:19:15.318310    2556 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-ba2514-predictor-6b8dc68545-bl4zz"]
May 26 16:19:15.327004 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:19:15.326979    2556 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-ba2514-predictor-6b8dc68545-bl4zz"]
May 26 16:19:15.348431 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:19:15.348407    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-3dec36-predictor-6f8b45dcff-p5629"]
May 26 16:19:15.350301 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:19:15.350278    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81669d50_52be_4ca4_98e5_7d3de6d4e7cc.slice/crio-c3838930622c69b4e3322e0bb1468843aedcbba280cf9894bce1c27fcc60de01 WatchSource:0}: Error finding container c3838930622c69b4e3322e0bb1468843aedcbba280cf9894bce1c27fcc60de01: Status 404 returned error can't find the container with id c3838930622c69b4e3322e0bb1468843aedcbba280cf9894bce1c27fcc60de01
May 26 16:19:16.271033 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:19:16.270990    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-3dec36-predictor-6f8b45dcff-p5629" event={"ID":"81669d50-52be-4ca4-98e5-7d3de6d4e7cc","Type":"ContainerStarted","Data":"c62d1f07a17d1f4cb1881badada5a8f4b687f723bed3777b26f4d6cef86d55f8"}
May 26 16:19:16.271033 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:19:16.271034    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-3dec36-predictor-6f8b45dcff-p5629" event={"ID":"81669d50-52be-4ca4-98e5-7d3de6d4e7cc","Type":"ContainerStarted","Data":"c3838930622c69b4e3322e0bb1468843aedcbba280cf9894bce1c27fcc60de01"}
May 26 16:19:16.772649 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:19:16.772616    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c8fe44d0-deee-4c97-943f-a7a0a51c500a" path="/var/lib/kubelet/pods/c8fe44d0-deee-4c97-943f-a7a0a51c500a/volumes"
May 26 16:19:18.788170 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:19:18.788147    2556 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-ba2514-predictor-686dd7994-vcx9l"
May 26 16:19:18.888402 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:19:18.888376    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fsq8k\" (UniqueName: \"kubernetes.io/projected/59ce4742-c702-4783-aca6-e7b3ab74513f-kube-api-access-fsq8k\") pod \"59ce4742-c702-4783-aca6-e7b3ab74513f\" (UID: \"59ce4742-c702-4783-aca6-e7b3ab74513f\") "
May 26 16:19:18.888560 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:19:18.888412    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/59ce4742-c702-4783-aca6-e7b3ab74513f-kserve-provision-location\") pod \"59ce4742-c702-4783-aca6-e7b3ab74513f\" (UID: \"59ce4742-c702-4783-aca6-e7b3ab74513f\") "
May 26 16:19:18.888560 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:19:18.888447    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/59ce4742-c702-4783-aca6-e7b3ab74513f-proxy-tls\") pod \"59ce4742-c702-4783-aca6-e7b3ab74513f\" (UID: \"59ce4742-c702-4783-aca6-e7b3ab74513f\") "
May 26 16:19:18.888560 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:19:18.888531    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-primary-ba2514-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/59ce4742-c702-4783-aca6-e7b3ab74513f-isvc-primary-ba2514-kube-rbac-proxy-sar-config\") pod \"59ce4742-c702-4783-aca6-e7b3ab74513f\" (UID: \"59ce4742-c702-4783-aca6-e7b3ab74513f\") "
May 26 16:19:18.888792 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:19:18.888708    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59ce4742-c702-4783-aca6-e7b3ab74513f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "59ce4742-c702-4783-aca6-e7b3ab74513f" (UID: "59ce4742-c702-4783-aca6-e7b3ab74513f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 16:19:18.888872 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:19:18.888853    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59ce4742-c702-4783-aca6-e7b3ab74513f-isvc-primary-ba2514-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-primary-ba2514-kube-rbac-proxy-sar-config") pod "59ce4742-c702-4783-aca6-e7b3ab74513f" (UID: "59ce4742-c702-4783-aca6-e7b3ab74513f"). InnerVolumeSpecName "isvc-primary-ba2514-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 16:19:18.890480 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:19:18.890456    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59ce4742-c702-4783-aca6-e7b3ab74513f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "59ce4742-c702-4783-aca6-e7b3ab74513f" (UID: "59ce4742-c702-4783-aca6-e7b3ab74513f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 16:19:18.890595 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:19:18.890554    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59ce4742-c702-4783-aca6-e7b3ab74513f-kube-api-access-fsq8k" (OuterVolumeSpecName: "kube-api-access-fsq8k") pod "59ce4742-c702-4783-aca6-e7b3ab74513f" (UID: "59ce4742-c702-4783-aca6-e7b3ab74513f"). InnerVolumeSpecName "kube-api-access-fsq8k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 16:19:18.989224 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:19:18.989195    2556 reconciler_common.go:299] "Volume detached for volume \"isvc-primary-ba2514-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/59ce4742-c702-4783-aca6-e7b3ab74513f-isvc-primary-ba2514-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-19.ec2.internal\" DevicePath \"\""
May 26 16:19:18.989224 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:19:18.989222    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fsq8k\" (UniqueName: \"kubernetes.io/projected/59ce4742-c702-4783-aca6-e7b3ab74513f-kube-api-access-fsq8k\") on node \"ip-10-0-134-19.ec2.internal\" DevicePath \"\""
May 26 16:19:18.989402 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:19:18.989238    2556 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/59ce4742-c702-4783-aca6-e7b3ab74513f-kserve-provision-location\") on node \"ip-10-0-134-19.ec2.internal\" DevicePath \"\""
May 26 16:19:18.989402 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:19:18.989251    2556 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/59ce4742-c702-4783-aca6-e7b3ab74513f-proxy-tls\") on node \"ip-10-0-134-19.ec2.internal\" DevicePath \"\""
May 26 16:19:19.285041 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:19:19.284949    2556 generic.go:358] "Generic (PLEG): container finished" podID="59ce4742-c702-4783-aca6-e7b3ab74513f" containerID="1194514042a2ea15d164f1519bfc187c98e55f77eb3c9971377f8d0d6cd968d6" exitCode=0
May 26 16:19:19.285041 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:19:19.285005    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-ba2514-predictor-686dd7994-vcx9l" event={"ID":"59ce4742-c702-4783-aca6-e7b3ab74513f","Type":"ContainerDied","Data":"1194514042a2ea15d164f1519bfc187c98e55f77eb3c9971377f8d0d6cd968d6"}
May 26 16:19:19.285241 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:19:19.285042    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-ba2514-predictor-686dd7994-vcx9l" event={"ID":"59ce4742-c702-4783-aca6-e7b3ab74513f","Type":"ContainerDied","Data":"de2da45a6cc7346a72a59621ecbdc88c908967b21d9d0ba9b450921a790f1800"}
May 26 16:19:19.285241 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:19:19.285046    2556 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-ba2514-predictor-686dd7994-vcx9l"
May 26 16:19:19.285241 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:19:19.285059    2556 scope.go:117] "RemoveContainer" containerID="58a91c81cd6f4d154871c9fea9b0f876ff59f487ccb0de2cfc49f44986155ed7"
May 26 16:19:19.292850 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:19:19.292702    2556 scope.go:117] "RemoveContainer" containerID="1194514042a2ea15d164f1519bfc187c98e55f77eb3c9971377f8d0d6cd968d6"
May 26 16:19:19.299782 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:19:19.299766    2556 scope.go:117] "RemoveContainer" containerID="f14e22618b28e0bde562653c1f30d858acdec72493344c8a7b4411be94d267a0"
May 26 16:19:19.306361 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:19:19.306339    2556 scope.go:117] "RemoveContainer" containerID="58a91c81cd6f4d154871c9fea9b0f876ff59f487ccb0de2cfc49f44986155ed7"
May 26 16:19:19.306622 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:19:19.306600    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58a91c81cd6f4d154871c9fea9b0f876ff59f487ccb0de2cfc49f44986155ed7\": container with ID starting with 58a91c81cd6f4d154871c9fea9b0f876ff59f487ccb0de2cfc49f44986155ed7 not found: ID does not exist" containerID="58a91c81cd6f4d154871c9fea9b0f876ff59f487ccb0de2cfc49f44986155ed7"
May 26 16:19:19.306701 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:19:19.306637    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58a91c81cd6f4d154871c9fea9b0f876ff59f487ccb0de2cfc49f44986155ed7"} err="failed to get container status \"58a91c81cd6f4d154871c9fea9b0f876ff59f487ccb0de2cfc49f44986155ed7\": rpc error: code = NotFound desc = could not find container \"58a91c81cd6f4d154871c9fea9b0f876ff59f487ccb0de2cfc49f44986155ed7\": container with ID starting with 58a91c81cd6f4d154871c9fea9b0f876ff59f487ccb0de2cfc49f44986155ed7 not found: ID does not exist"
May 26 16:19:19.306701 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:19:19.306664    2556 scope.go:117] "RemoveContainer" containerID="1194514042a2ea15d164f1519bfc187c98e55f77eb3c9971377f8d0d6cd968d6"
May 26 16:19:19.306883 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:19:19.306866    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1194514042a2ea15d164f1519bfc187c98e55f77eb3c9971377f8d0d6cd968d6\": container with ID starting with 1194514042a2ea15d164f1519bfc187c98e55f77eb3c9971377f8d0d6cd968d6 not found: ID does not exist" containerID="1194514042a2ea15d164f1519bfc187c98e55f77eb3c9971377f8d0d6cd968d6"
May 26 16:19:19.306927 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:19:19.306890    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1194514042a2ea15d164f1519bfc187c98e55f77eb3c9971377f8d0d6cd968d6"} err="failed to get container status \"1194514042a2ea15d164f1519bfc187c98e55f77eb3c9971377f8d0d6cd968d6\": rpc error: code = NotFound desc = could not find container \"1194514042a2ea15d164f1519bfc187c98e55f77eb3c9971377f8d0d6cd968d6\": container with ID starting with 1194514042a2ea15d164f1519bfc187c98e55f77eb3c9971377f8d0d6cd968d6 not found: ID does not exist"
May 26 16:19:19.306927 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:19:19.306906    2556 scope.go:117] "RemoveContainer" containerID="f14e22618b28e0bde562653c1f30d858acdec72493344c8a7b4411be94d267a0"
May 26 16:19:19.307157 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:19:19.307140    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f14e22618b28e0bde562653c1f30d858acdec72493344c8a7b4411be94d267a0\": container with ID starting with f14e22618b28e0bde562653c1f30d858acdec72493344c8a7b4411be94d267a0 not found: ID does not exist" containerID="f14e22618b28e0bde562653c1f30d858acdec72493344c8a7b4411be94d267a0"
May 26 16:19:19.307200 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:19:19.307159    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f14e22618b28e0bde562653c1f30d858acdec72493344c8a7b4411be94d267a0"} err="failed to get container status \"f14e22618b28e0bde562653c1f30d858acdec72493344c8a7b4411be94d267a0\": rpc error: code = NotFound desc = could not find container \"f14e22618b28e0bde562653c1f30d858acdec72493344c8a7b4411be94d267a0\": container with ID starting with f14e22618b28e0bde562653c1f30d858acdec72493344c8a7b4411be94d267a0 not found: ID does not exist"
May 26 16:19:19.327269 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:19:19.327245    2556 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-ba2514-predictor-686dd7994-vcx9l"]
May 26 16:19:19.330847 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:19:19.330826    2556 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-ba2514-predictor-686dd7994-vcx9l"]
May 26 16:19:20.771783 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:19:20.771751    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="59ce4742-c702-4783-aca6-e7b3ab74513f" path="/var/lib/kubelet/pods/59ce4742-c702-4783-aca6-e7b3ab74513f/volumes"
May 26 16:19:21.293379 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:19:21.293353    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-3dec36-predictor-6f8b45dcff-p5629_81669d50-52be-4ca4-98e5-7d3de6d4e7cc/storage-initializer/0.log"
May 26 16:19:21.293563 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:19:21.293389    2556 generic.go:358] "Generic (PLEG): container finished" podID="81669d50-52be-4ca4-98e5-7d3de6d4e7cc" containerID="c62d1f07a17d1f4cb1881badada5a8f4b687f723bed3777b26f4d6cef86d55f8" exitCode=1
May 26 16:19:21.293563 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:19:21.293455    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-3dec36-predictor-6f8b45dcff-p5629" event={"ID":"81669d50-52be-4ca4-98e5-7d3de6d4e7cc","Type":"ContainerDied","Data":"c62d1f07a17d1f4cb1881badada5a8f4b687f723bed3777b26f4d6cef86d55f8"}
May 26 16:19:22.300199 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:19:22.300168    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-3dec36-predictor-6f8b45dcff-p5629_81669d50-52be-4ca4-98e5-7d3de6d4e7cc/storage-initializer/0.log"
May 26 16:19:22.300576 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:19:22.300291    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-3dec36-predictor-6f8b45dcff-p5629" event={"ID":"81669d50-52be-4ca4-98e5-7d3de6d4e7cc","Type":"ContainerStarted","Data":"a663d17f95a5559756b575fd7b6a83e9ccaaf69adf0c6a8655f8719711344bea"}
May 26 16:19:24.309422 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:19:24.309392    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-3dec36-predictor-6f8b45dcff-p5629_81669d50-52be-4ca4-98e5-7d3de6d4e7cc/storage-initializer/1.log"
May 26 16:19:24.309825 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:19:24.309723    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-3dec36-predictor-6f8b45dcff-p5629_81669d50-52be-4ca4-98e5-7d3de6d4e7cc/storage-initializer/0.log"
May 26 16:19:24.309825 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:19:24.309755    2556 generic.go:358] "Generic (PLEG): container finished" podID="81669d50-52be-4ca4-98e5-7d3de6d4e7cc" containerID="a663d17f95a5559756b575fd7b6a83e9ccaaf69adf0c6a8655f8719711344bea" exitCode=1
May 26 16:19:24.309825 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:19:24.309817    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-3dec36-predictor-6f8b45dcff-p5629" event={"ID":"81669d50-52be-4ca4-98e5-7d3de6d4e7cc","Type":"ContainerDied","Data":"a663d17f95a5559756b575fd7b6a83e9ccaaf69adf0c6a8655f8719711344bea"}
May 26 16:19:24.309935 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:19:24.309846    2556 scope.go:117] "RemoveContainer" containerID="c62d1f07a17d1f4cb1881badada5a8f4b687f723bed3777b26f4d6cef86d55f8"
May 26 16:19:24.310248 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:19:24.310232    2556 scope.go:117] "RemoveContainer" containerID="c62d1f07a17d1f4cb1881badada5a8f4b687f723bed3777b26f4d6cef86d55f8"
May 26 16:19:24.320269 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:19:24.320235    2556 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-init-fail-3dec36-predictor-6f8b45dcff-p5629_kserve-ci-e2e-test_81669d50-52be-4ca4-98e5-7d3de6d4e7cc_0 in pod sandbox c3838930622c69b4e3322e0bb1468843aedcbba280cf9894bce1c27fcc60de01 from index: no such id: 'c62d1f07a17d1f4cb1881badada5a8f4b687f723bed3777b26f4d6cef86d55f8'" containerID="c62d1f07a17d1f4cb1881badada5a8f4b687f723bed3777b26f4d6cef86d55f8"
May 26 16:19:24.320344 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:19:24.320290    2556 kuberuntime_container.go:951] "Unhandled Error" err="failed to remove pod init container \"storage-initializer\": rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-init-fail-3dec36-predictor-6f8b45dcff-p5629_kserve-ci-e2e-test_81669d50-52be-4ca4-98e5-7d3de6d4e7cc_0 in pod sandbox c3838930622c69b4e3322e0bb1468843aedcbba280cf9894bce1c27fcc60de01 from index: no such id: 'c62d1f07a17d1f4cb1881badada5a8f4b687f723bed3777b26f4d6cef86d55f8'; Skipping pod \"isvc-init-fail-3dec36-predictor-6f8b45dcff-p5629_kserve-ci-e2e-test(81669d50-52be-4ca4-98e5-7d3de6d4e7cc)\"" logger="UnhandledError"
May 26 16:19:24.321881 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:19:24.321861    2556 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-init-fail-3dec36-predictor-6f8b45dcff-p5629_kserve-ci-e2e-test(81669d50-52be-4ca4-98e5-7d3de6d4e7cc)\"" pod="kserve-ci-e2e-test/isvc-init-fail-3dec36-predictor-6f8b45dcff-p5629" podUID="81669d50-52be-4ca4-98e5-7d3de6d4e7cc"
May 26 16:19:24.939198 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:19:24.939165    2556 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-3dec36-predictor-6f8b45dcff-p5629"]
May 26 16:19:25.163528 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:19:25.163494    2556 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-ea140-predictor-85487667c-ztpzz"]
May 26 16:19:25.163823 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:19:25.163811    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59ce4742-c702-4783-aca6-e7b3ab74513f" containerName="kube-rbac-proxy"
May 26 16:19:25.163881 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:19:25.163826    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="59ce4742-c702-4783-aca6-e7b3ab74513f" containerName="kube-rbac-proxy"
May 26 16:19:25.163881 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:19:25.163834    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59ce4742-c702-4783-aca6-e7b3ab74513f" containerName="kserve-container"
May 26 16:19:25.163881 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:19:25.163839    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="59ce4742-c702-4783-aca6-e7b3ab74513f" containerName="kserve-container"
May 26 16:19:25.163881 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:19:25.163849    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59ce4742-c702-4783-aca6-e7b3ab74513f" containerName="storage-initializer"
May 26 16:19:25.163881 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:19:25.163854    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="59ce4742-c702-4783-aca6-e7b3ab74513f" containerName="storage-initializer"
May 26 16:19:25.163881 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:19:25.163860    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c8fe44d0-deee-4c97-943f-a7a0a51c500a" containerName="storage-initializer"
May 26 16:19:25.163881 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:19:25.163865    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="c8fe44d0-deee-4c97-943f-a7a0a51c500a" containerName="storage-initializer"
May 26 16:19:25.164141 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:19:25.163916    2556 memory_manager.go:356] "RemoveStaleState removing state" podUID="c8fe44d0-deee-4c97-943f-a7a0a51c500a" containerName="storage-initializer"
May 26 16:19:25.164141 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:19:25.163924    2556 memory_manager.go:356] "RemoveStaleState removing state" podUID="59ce4742-c702-4783-aca6-e7b3ab74513f" containerName="kube-rbac-proxy"
May 26 16:19:25.164141 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:19:25.163931    2556 memory_manager.go:356] "RemoveStaleState removing state" podUID="c8fe44d0-deee-4c97-943f-a7a0a51c500a" containerName="storage-initializer"
May 26 16:19:25.164141 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:19:25.163939    2556 memory_manager.go:356] "RemoveStaleState removing state" podUID="59ce4742-c702-4783-aca6-e7b3ab74513f" containerName="kserve-container"
May 26 16:19:25.164141 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:19:25.163991    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c8fe44d0-deee-4c97-943f-a7a0a51c500a" containerName="storage-initializer"
May 26 16:19:25.164141 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:19:25.163997    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="c8fe44d0-deee-4c97-943f-a7a0a51c500a" containerName="storage-initializer"
May 26 16:19:25.167904 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:19:25.167885    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-ea140-predictor-85487667c-ztpzz"
May 26 16:19:25.174740 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:19:25.174719    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-ea140-kube-rbac-proxy-sar-config\""
May 26 16:19:25.174892 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:19:25.174830    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-ea140-predictor-serving-cert\""
May 26 16:19:25.175001 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:19:25.174979    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-llsfr\""
May 26 16:19:25.194503 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:19:25.194446    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-ea140-predictor-85487667c-ztpzz"]
May 26 16:19:25.234453 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:19:25.234425    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3b1ddd12-bc61-4aff-9907-b08cd577ee3a-kserve-provision-location\") pod \"raw-sklearn-ea140-predictor-85487667c-ztpzz\" (UID: \"3b1ddd12-bc61-4aff-9907-b08cd577ee3a\") " pod="kserve-ci-e2e-test/raw-sklearn-ea140-predictor-85487667c-ztpzz"
May 26 16:19:25.234531 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:19:25.234472    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"raw-sklearn-ea140-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3b1ddd12-bc61-4aff-9907-b08cd577ee3a-raw-sklearn-ea140-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-ea140-predictor-85487667c-ztpzz\" (UID: \"3b1ddd12-bc61-4aff-9907-b08cd577ee3a\") " pod="kserve-ci-e2e-test/raw-sklearn-ea140-predictor-85487667c-ztpzz"
May 26 16:19:25.234531 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:19:25.234515    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2hcd\" (UniqueName: \"kubernetes.io/projected/3b1ddd12-bc61-4aff-9907-b08cd577ee3a-kube-api-access-w2hcd\") pod \"raw-sklearn-ea140-predictor-85487667c-ztpzz\" (UID: \"3b1ddd12-bc61-4aff-9907-b08cd577ee3a\") " pod="kserve-ci-e2e-test/raw-sklearn-ea140-predictor-85487667c-ztpzz"
May 26 16:19:25.234611 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:19:25.234571    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3b1ddd12-bc61-4aff-9907-b08cd577ee3a-proxy-tls\") pod \"raw-sklearn-ea140-predictor-85487667c-ztpzz\" (UID: \"3b1ddd12-bc61-4aff-9907-b08cd577ee3a\") " pod="kserve-ci-e2e-test/raw-sklearn-ea140-predictor-85487667c-ztpzz"
May 26 16:19:25.315254 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:19:25.315232    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-3dec36-predictor-6f8b45dcff-p5629_81669d50-52be-4ca4-98e5-7d3de6d4e7cc/storage-initializer/1.log"
May 26 16:19:25.335859 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:19:25.335830    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3b1ddd12-bc61-4aff-9907-b08cd577ee3a-kserve-provision-location\") pod \"raw-sklearn-ea140-predictor-85487667c-ztpzz\" (UID: \"3b1ddd12-bc61-4aff-9907-b08cd577ee3a\") " pod="kserve-ci-e2e-test/raw-sklearn-ea140-predictor-85487667c-ztpzz"
May 26 16:19:25.335990 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:19:25.335882    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"raw-sklearn-ea140-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3b1ddd12-bc61-4aff-9907-b08cd577ee3a-raw-sklearn-ea140-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-ea140-predictor-85487667c-ztpzz\" (UID: \"3b1ddd12-bc61-4aff-9907-b08cd577ee3a\") " pod="kserve-ci-e2e-test/raw-sklearn-ea140-predictor-85487667c-ztpzz"
May 26 16:19:25.336048 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:19:25.336031    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-w2hcd\" (UniqueName: \"kubernetes.io/projected/3b1ddd12-bc61-4aff-9907-b08cd577ee3a-kube-api-access-w2hcd\") pod \"raw-sklearn-ea140-predictor-85487667c-ztpzz\" (UID: \"3b1ddd12-bc61-4aff-9907-b08cd577ee3a\") " pod="kserve-ci-e2e-test/raw-sklearn-ea140-predictor-85487667c-ztpzz"
May 26 16:19:25.336128 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:19:25.336068    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3b1ddd12-bc61-4aff-9907-b08cd577ee3a-proxy-tls\") pod \"raw-sklearn-ea140-predictor-85487667c-ztpzz\" (UID: \"3b1ddd12-bc61-4aff-9907-b08cd577ee3a\") " pod="kserve-ci-e2e-test/raw-sklearn-ea140-predictor-85487667c-ztpzz"
May 26 16:19:25.336369 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:19:25.336250    2556 secret.go:189] Couldn't get secret kserve-ci-e2e-test/raw-sklearn-ea140-predictor-serving-cert: secret "raw-sklearn-ea140-predictor-serving-cert" not found
May 26 16:19:25.336369 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:19:25.336283    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3b1ddd12-bc61-4aff-9907-b08cd577ee3a-kserve-provision-location\") pod \"raw-sklearn-ea140-predictor-85487667c-ztpzz\" (UID: \"3b1ddd12-bc61-4aff-9907-b08cd577ee3a\") " pod="kserve-ci-e2e-test/raw-sklearn-ea140-predictor-85487667c-ztpzz"
May 26 16:19:25.336369 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:19:25.336317    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b1ddd12-bc61-4aff-9907-b08cd577ee3a-proxy-tls podName:3b1ddd12-bc61-4aff-9907-b08cd577ee3a nodeName:}" failed. No retries permitted until 2026-05-26 16:19:25.836297737 +0000 UTC m=+1127.680914722 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/3b1ddd12-bc61-4aff-9907-b08cd577ee3a-proxy-tls") pod "raw-sklearn-ea140-predictor-85487667c-ztpzz" (UID: "3b1ddd12-bc61-4aff-9907-b08cd577ee3a") : secret "raw-sklearn-ea140-predictor-serving-cert" not found
May 26 16:19:25.336624 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:19:25.336604    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"raw-sklearn-ea140-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3b1ddd12-bc61-4aff-9907-b08cd577ee3a-raw-sklearn-ea140-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-ea140-predictor-85487667c-ztpzz\" (UID: \"3b1ddd12-bc61-4aff-9907-b08cd577ee3a\") " pod="kserve-ci-e2e-test/raw-sklearn-ea140-predictor-85487667c-ztpzz"
May 26 16:19:25.350871 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:19:25.350844    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2hcd\" (UniqueName: \"kubernetes.io/projected/3b1ddd12-bc61-4aff-9907-b08cd577ee3a-kube-api-access-w2hcd\") pod \"raw-sklearn-ea140-predictor-85487667c-ztpzz\" (UID: \"3b1ddd12-bc61-4aff-9907-b08cd577ee3a\") " pod="kserve-ci-e2e-test/raw-sklearn-ea140-predictor-85487667c-ztpzz"
May 26 16:19:25.442309 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:19:25.442286    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-3dec36-predictor-6f8b45dcff-p5629_81669d50-52be-4ca4-98e5-7d3de6d4e7cc/storage-initializer/1.log"
May 26 16:19:25.442416 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:19:25.442351    2556 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-3dec36-predictor-6f8b45dcff-p5629"
May 26 16:19:25.537958 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:19:25.537834    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-init-fail-3dec36-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/81669d50-52be-4ca4-98e5-7d3de6d4e7cc-isvc-init-fail-3dec36-kube-rbac-proxy-sar-config\") pod \"81669d50-52be-4ca4-98e5-7d3de6d4e7cc\" (UID: \"81669d50-52be-4ca4-98e5-7d3de6d4e7cc\") "
May 26 16:19:25.537958 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:19:25.537940    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/81669d50-52be-4ca4-98e5-7d3de6d4e7cc-kserve-provision-location\") pod \"81669d50-52be-4ca4-98e5-7d3de6d4e7cc\" (UID: \"81669d50-52be-4ca4-98e5-7d3de6d4e7cc\") "
May 26 16:19:25.538165 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:19:25.537965    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/81669d50-52be-4ca4-98e5-7d3de6d4e7cc-cabundle-cert\") pod \"81669d50-52be-4ca4-98e5-7d3de6d4e7cc\" (UID: \"81669d50-52be-4ca4-98e5-7d3de6d4e7cc\") "
May 26 16:19:25.538165 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:19:25.538015    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/81669d50-52be-4ca4-98e5-7d3de6d4e7cc-proxy-tls\") pod \"81669d50-52be-4ca4-98e5-7d3de6d4e7cc\" (UID: \"81669d50-52be-4ca4-98e5-7d3de6d4e7cc\") "
May 26 16:19:25.538165 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:19:25.538082    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zk4v6\" (UniqueName: \"kubernetes.io/projected/81669d50-52be-4ca4-98e5-7d3de6d4e7cc-kube-api-access-zk4v6\") pod \"81669d50-52be-4ca4-98e5-7d3de6d4e7cc\" (UID: \"81669d50-52be-4ca4-98e5-7d3de6d4e7cc\") "
May 26 16:19:25.538321 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:19:25.538226    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81669d50-52be-4ca4-98e5-7d3de6d4e7cc-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "81669d50-52be-4ca4-98e5-7d3de6d4e7cc" (UID: "81669d50-52be-4ca4-98e5-7d3de6d4e7cc"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 16:19:25.538321 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:19:25.538276    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81669d50-52be-4ca4-98e5-7d3de6d4e7cc-isvc-init-fail-3dec36-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-init-fail-3dec36-kube-rbac-proxy-sar-config") pod "81669d50-52be-4ca4-98e5-7d3de6d4e7cc" (UID: "81669d50-52be-4ca4-98e5-7d3de6d4e7cc"). InnerVolumeSpecName "isvc-init-fail-3dec36-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 16:19:25.538407 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:19:25.538327    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81669d50-52be-4ca4-98e5-7d3de6d4e7cc-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "81669d50-52be-4ca4-98e5-7d3de6d4e7cc" (UID: "81669d50-52be-4ca4-98e5-7d3de6d4e7cc"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 16:19:25.538407 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:19:25.538363    2556 reconciler_common.go:299] "Volume detached for volume \"isvc-init-fail-3dec36-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/81669d50-52be-4ca4-98e5-7d3de6d4e7cc-isvc-init-fail-3dec36-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-19.ec2.internal\" DevicePath \"\""
May 26 16:19:25.538407 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:19:25.538384    2556 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/81669d50-52be-4ca4-98e5-7d3de6d4e7cc-kserve-provision-location\") on node \"ip-10-0-134-19.ec2.internal\" DevicePath \"\""
May 26 16:19:25.540070 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:19:25.540047    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/81669d50-52be-4ca4-98e5-7d3de6d4e7cc-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "81669d50-52be-4ca4-98e5-7d3de6d4e7cc" (UID: "81669d50-52be-4ca4-98e5-7d3de6d4e7cc"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 16:19:25.540070 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:19:25.540049    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81669d50-52be-4ca4-98e5-7d3de6d4e7cc-kube-api-access-zk4v6" (OuterVolumeSpecName: "kube-api-access-zk4v6") pod "81669d50-52be-4ca4-98e5-7d3de6d4e7cc" (UID: "81669d50-52be-4ca4-98e5-7d3de6d4e7cc"). InnerVolumeSpecName "kube-api-access-zk4v6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 16:19:25.639160 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:19:25.639130    2556 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/81669d50-52be-4ca4-98e5-7d3de6d4e7cc-cabundle-cert\") on node \"ip-10-0-134-19.ec2.internal\" DevicePath \"\""
May 26 16:19:25.639160 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:19:25.639162    2556 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/81669d50-52be-4ca4-98e5-7d3de6d4e7cc-proxy-tls\") on node \"ip-10-0-134-19.ec2.internal\" DevicePath \"\""
May 26 16:19:25.639360 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:19:25.639177    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zk4v6\" (UniqueName: \"kubernetes.io/projected/81669d50-52be-4ca4-98e5-7d3de6d4e7cc-kube-api-access-zk4v6\") on node \"ip-10-0-134-19.ec2.internal\" DevicePath \"\""
May 26 16:19:25.841282 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:19:25.841200    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3b1ddd12-bc61-4aff-9907-b08cd577ee3a-proxy-tls\") pod \"raw-sklearn-ea140-predictor-85487667c-ztpzz\" (UID: \"3b1ddd12-bc61-4aff-9907-b08cd577ee3a\") " pod="kserve-ci-e2e-test/raw-sklearn-ea140-predictor-85487667c-ztpzz"
May 26 16:19:25.843612 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:19:25.843592    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3b1ddd12-bc61-4aff-9907-b08cd577ee3a-proxy-tls\") pod \"raw-sklearn-ea140-predictor-85487667c-ztpzz\" (UID: \"3b1ddd12-bc61-4aff-9907-b08cd577ee3a\") " pod="kserve-ci-e2e-test/raw-sklearn-ea140-predictor-85487667c-ztpzz"
May 26 16:19:26.077719 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:19:26.077672    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-ea140-predictor-85487667c-ztpzz"
May 26 16:19:26.222585 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:19:26.222550    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-ea140-predictor-85487667c-ztpzz"]
May 26 16:19:26.225852 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:19:26.225825    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b1ddd12_bc61_4aff_9907_b08cd577ee3a.slice/crio-9e36c5b9235ea4648f4ccd3f6e6950eaeb987b23d56eebd501af4ff6d54c66d9 WatchSource:0}: Error finding container 9e36c5b9235ea4648f4ccd3f6e6950eaeb987b23d56eebd501af4ff6d54c66d9: Status 404 returned error can't find the container with id 9e36c5b9235ea4648f4ccd3f6e6950eaeb987b23d56eebd501af4ff6d54c66d9
May 26 16:19:26.319859 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:19:26.319831    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-3dec36-predictor-6f8b45dcff-p5629_81669d50-52be-4ca4-98e5-7d3de6d4e7cc/storage-initializer/1.log"
May 26 16:19:26.320264 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:19:26.319969    2556 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-3dec36-predictor-6f8b45dcff-p5629"
May 26 16:19:26.320264 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:19:26.319963    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-3dec36-predictor-6f8b45dcff-p5629" event={"ID":"81669d50-52be-4ca4-98e5-7d3de6d4e7cc","Type":"ContainerDied","Data":"c3838930622c69b4e3322e0bb1468843aedcbba280cf9894bce1c27fcc60de01"}
May 26 16:19:26.320264 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:19:26.320017    2556 scope.go:117] "RemoveContainer" containerID="a663d17f95a5559756b575fd7b6a83e9ccaaf69adf0c6a8655f8719711344bea"
May 26 16:19:26.321503 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:19:26.321361    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-ea140-predictor-85487667c-ztpzz" event={"ID":"3b1ddd12-bc61-4aff-9907-b08cd577ee3a","Type":"ContainerStarted","Data":"3022251c3fbe10dde82b0aa53a79ee266fd77b8d59d7f5632bf2a957652c274c"}
May 26 16:19:26.321503 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:19:26.321384    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-ea140-predictor-85487667c-ztpzz" event={"ID":"3b1ddd12-bc61-4aff-9907-b08cd577ee3a","Type":"ContainerStarted","Data":"9e36c5b9235ea4648f4ccd3f6e6950eaeb987b23d56eebd501af4ff6d54c66d9"}
May 26 16:19:26.403977 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:19:26.403902    2556 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-3dec36-predictor-6f8b45dcff-p5629"]
May 26 16:19:26.407970 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:19:26.407942    2556 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-3dec36-predictor-6f8b45dcff-p5629"]
May 26 16:19:26.771910 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:19:26.771827    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="81669d50-52be-4ca4-98e5-7d3de6d4e7cc" path="/var/lib/kubelet/pods/81669d50-52be-4ca4-98e5-7d3de6d4e7cc/volumes"
May 26 16:19:30.336656 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:19:30.336621    2556 generic.go:358] "Generic (PLEG): container finished" podID="3b1ddd12-bc61-4aff-9907-b08cd577ee3a" containerID="3022251c3fbe10dde82b0aa53a79ee266fd77b8d59d7f5632bf2a957652c274c" exitCode=0
May 26 16:19:30.337035 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:19:30.336695    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-ea140-predictor-85487667c-ztpzz" event={"ID":"3b1ddd12-bc61-4aff-9907-b08cd577ee3a","Type":"ContainerDied","Data":"3022251c3fbe10dde82b0aa53a79ee266fd77b8d59d7f5632bf2a957652c274c"}
May 26 16:19:31.342638 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:19:31.342606    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-ea140-predictor-85487667c-ztpzz" event={"ID":"3b1ddd12-bc61-4aff-9907-b08cd577ee3a","Type":"ContainerStarted","Data":"00458c6e2a489e62ea0390b877775e0923f3a106154eb86ded2738a48fac1a9c"}
May 26 16:19:31.342638 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:19:31.342643    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-ea140-predictor-85487667c-ztpzz" event={"ID":"3b1ddd12-bc61-4aff-9907-b08cd577ee3a","Type":"ContainerStarted","Data":"e3f12bad25cd927d5583112decf16c1f412d97a51ff9b947201e4b05ff276670"}
May 26 16:19:31.343071 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:19:31.342923    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-ea140-predictor-85487667c-ztpzz"
May 26 16:19:31.383192 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:19:31.383146    2556 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/raw-sklearn-ea140-predictor-85487667c-ztpzz" podStartSLOduration=6.383132684 podStartE2EDuration="6.383132684s" podCreationTimestamp="2026-05-26 16:19:25 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 16:19:31.379861451 +0000 UTC m=+1133.224478442" watchObservedRunningTime="2026-05-26 16:19:31.383132684 +0000 UTC m=+1133.227749675"
May 26 16:19:32.346064 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:19:32.346031    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-ea140-predictor-85487667c-ztpzz"
May 26 16:19:32.347341 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:19:32.347311    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-ea140-predictor-85487667c-ztpzz" podUID="3b1ddd12-bc61-4aff-9907-b08cd577ee3a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
May 26 16:19:33.348748 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:19:33.348706    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-ea140-predictor-85487667c-ztpzz" podUID="3b1ddd12-bc61-4aff-9907-b08cd577ee3a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
May 26 16:19:38.353300 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:19:38.353268    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-ea140-predictor-85487667c-ztpzz"
May 26 16:19:38.353764 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:19:38.353707    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-ea140-predictor-85487667c-ztpzz" podUID="3b1ddd12-bc61-4aff-9907-b08cd577ee3a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
May 26 16:19:48.354073 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:19:48.354032    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-ea140-predictor-85487667c-ztpzz" podUID="3b1ddd12-bc61-4aff-9907-b08cd577ee3a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
May 26 16:19:58.353810 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:19:58.353725    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-ea140-predictor-85487667c-ztpzz" podUID="3b1ddd12-bc61-4aff-9907-b08cd577ee3a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
May 26 16:20:08.353672 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:20:08.353632    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-ea140-predictor-85487667c-ztpzz" podUID="3b1ddd12-bc61-4aff-9907-b08cd577ee3a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
May 26 16:20:18.353710 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:20:18.353667    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-ea140-predictor-85487667c-ztpzz" podUID="3b1ddd12-bc61-4aff-9907-b08cd577ee3a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
May 26 16:20:28.354275 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:20:28.354235    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-ea140-predictor-85487667c-ztpzz" podUID="3b1ddd12-bc61-4aff-9907-b08cd577ee3a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
May 26 16:20:38.354808 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:20:38.354782    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-ea140-predictor-85487667c-ztpzz"
May 26 16:20:38.708513 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:20:38.708484    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sk7ns_4887c0d3-11fb-4287-bb94-a876a6c38803/ovn-acl-logging/0.log"
May 26 16:20:38.711366 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:20:38.711342    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sk7ns_4887c0d3-11fb-4287-bb94-a876a6c38803/ovn-acl-logging/0.log"
May 26 16:20:45.171361 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:20:45.171319    2556 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-ea140-predictor-85487667c-ztpzz"]
May 26 16:20:45.171858 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:20:45.171649    2556 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-ea140-predictor-85487667c-ztpzz" podUID="3b1ddd12-bc61-4aff-9907-b08cd577ee3a" containerName="kserve-container" containerID="cri-o://e3f12bad25cd927d5583112decf16c1f412d97a51ff9b947201e4b05ff276670" gracePeriod=30
May 26 16:20:45.171858 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:20:45.171729    2556 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-ea140-predictor-85487667c-ztpzz" podUID="3b1ddd12-bc61-4aff-9907-b08cd577ee3a" containerName="kube-rbac-proxy" containerID="cri-o://00458c6e2a489e62ea0390b877775e0923f3a106154eb86ded2738a48fac1a9c" gracePeriod=30
May 26 16:20:45.286454 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:20:45.286426    2556 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-5a02e-predictor-76c678f7d7-nvwcg"]
May 26 16:20:45.286811 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:20:45.286795    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81669d50-52be-4ca4-98e5-7d3de6d4e7cc" containerName="storage-initializer"
May 26 16:20:45.286893 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:20:45.286815    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="81669d50-52be-4ca4-98e5-7d3de6d4e7cc" containerName="storage-initializer"
May 26 16:20:45.286893 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:20:45.286831    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81669d50-52be-4ca4-98e5-7d3de6d4e7cc" containerName="storage-initializer"
May 26 16:20:45.286893 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:20:45.286840    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="81669d50-52be-4ca4-98e5-7d3de6d4e7cc" containerName="storage-initializer"
May 26 16:20:45.287050 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:20:45.286921    2556 memory_manager.go:356] "RemoveStaleState removing state" podUID="81669d50-52be-4ca4-98e5-7d3de6d4e7cc" containerName="storage-initializer"
May 26 16:20:45.287050 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:20:45.286936    2556 memory_manager.go:356] "RemoveStaleState removing state" podUID="81669d50-52be-4ca4-98e5-7d3de6d4e7cc" containerName="storage-initializer"
May 26 16:20:45.288834 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:20:45.288817    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-5a02e-predictor-76c678f7d7-nvwcg"
May 26 16:20:45.291575 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:20:45.291557    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-runtime-5a02e-predictor-serving-cert\""
May 26 16:20:45.291898 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:20:45.291874    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-runtime-5a02e-kube-rbac-proxy-sar-config\""
May 26 16:20:45.302727 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:20:45.302704    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-5a02e-predictor-76c678f7d7-nvwcg"]
May 26 16:20:45.337644 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:20:45.337618    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/05129dae-e202-4f9a-8aec-a47bd5cefa6d-proxy-tls\") pod \"raw-sklearn-runtime-5a02e-predictor-76c678f7d7-nvwcg\" (UID: \"05129dae-e202-4f9a-8aec-a47bd5cefa6d\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-5a02e-predictor-76c678f7d7-nvwcg"
May 26 16:20:45.337768 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:20:45.337662    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"raw-sklearn-runtime-5a02e-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/05129dae-e202-4f9a-8aec-a47bd5cefa6d-raw-sklearn-runtime-5a02e-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-5a02e-predictor-76c678f7d7-nvwcg\" (UID: \"05129dae-e202-4f9a-8aec-a47bd5cefa6d\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-5a02e-predictor-76c678f7d7-nvwcg"
May 26 16:20:45.337768 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:20:45.337703    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/05129dae-e202-4f9a-8aec-a47bd5cefa6d-kserve-provision-location\") pod \"raw-sklearn-runtime-5a02e-predictor-76c678f7d7-nvwcg\" (UID: \"05129dae-e202-4f9a-8aec-a47bd5cefa6d\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-5a02e-predictor-76c678f7d7-nvwcg"
May 26 16:20:45.337768 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:20:45.337742    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgcgn\" (UniqueName: \"kubernetes.io/projected/05129dae-e202-4f9a-8aec-a47bd5cefa6d-kube-api-access-hgcgn\") pod \"raw-sklearn-runtime-5a02e-predictor-76c678f7d7-nvwcg\" (UID: \"05129dae-e202-4f9a-8aec-a47bd5cefa6d\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-5a02e-predictor-76c678f7d7-nvwcg"
May 26 16:20:45.438887 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:20:45.438823    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-hgcgn\" (UniqueName: \"kubernetes.io/projected/05129dae-e202-4f9a-8aec-a47bd5cefa6d-kube-api-access-hgcgn\") pod \"raw-sklearn-runtime-5a02e-predictor-76c678f7d7-nvwcg\" (UID: \"05129dae-e202-4f9a-8aec-a47bd5cefa6d\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-5a02e-predictor-76c678f7d7-nvwcg"
May 26 16:20:45.438992 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:20:45.438894    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/05129dae-e202-4f9a-8aec-a47bd5cefa6d-proxy-tls\") pod \"raw-sklearn-runtime-5a02e-predictor-76c678f7d7-nvwcg\" (UID: \"05129dae-e202-4f9a-8aec-a47bd5cefa6d\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-5a02e-predictor-76c678f7d7-nvwcg"
May 26 16:20:45.438992 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:20:45.438924    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"raw-sklearn-runtime-5a02e-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/05129dae-e202-4f9a-8aec-a47bd5cefa6d-raw-sklearn-runtime-5a02e-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-5a02e-predictor-76c678f7d7-nvwcg\" (UID: \"05129dae-e202-4f9a-8aec-a47bd5cefa6d\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-5a02e-predictor-76c678f7d7-nvwcg"
May 26 16:20:45.438992 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:20:45.438965    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/05129dae-e202-4f9a-8aec-a47bd5cefa6d-kserve-provision-location\") pod \"raw-sklearn-runtime-5a02e-predictor-76c678f7d7-nvwcg\" (UID: \"05129dae-e202-4f9a-8aec-a47bd5cefa6d\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-5a02e-predictor-76c678f7d7-nvwcg"
May 26 16:20:45.439139 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:20:45.439044    2556 secret.go:189] Couldn't get secret kserve-ci-e2e-test/raw-sklearn-runtime-5a02e-predictor-serving-cert: secret "raw-sklearn-runtime-5a02e-predictor-serving-cert" not found
May 26 16:20:45.439182 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:20:45.439137    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05129dae-e202-4f9a-8aec-a47bd5cefa6d-proxy-tls podName:05129dae-e202-4f9a-8aec-a47bd5cefa6d nodeName:}" failed. No retries permitted until 2026-05-26 16:20:45.939115311 +0000 UTC m=+1207.783732297 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/05129dae-e202-4f9a-8aec-a47bd5cefa6d-proxy-tls") pod "raw-sklearn-runtime-5a02e-predictor-76c678f7d7-nvwcg" (UID: "05129dae-e202-4f9a-8aec-a47bd5cefa6d") : secret "raw-sklearn-runtime-5a02e-predictor-serving-cert" not found
May 26 16:20:45.439393 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:20:45.439376    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/05129dae-e202-4f9a-8aec-a47bd5cefa6d-kserve-provision-location\") pod \"raw-sklearn-runtime-5a02e-predictor-76c678f7d7-nvwcg\" (UID: \"05129dae-e202-4f9a-8aec-a47bd5cefa6d\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-5a02e-predictor-76c678f7d7-nvwcg"
May 26 16:20:45.439652 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:20:45.439635    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"raw-sklearn-runtime-5a02e-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/05129dae-e202-4f9a-8aec-a47bd5cefa6d-raw-sklearn-runtime-5a02e-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-5a02e-predictor-76c678f7d7-nvwcg\" (UID: \"05129dae-e202-4f9a-8aec-a47bd5cefa6d\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-5a02e-predictor-76c678f7d7-nvwcg"
May 26 16:20:45.450563 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:20:45.450541    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgcgn\" (UniqueName: \"kubernetes.io/projected/05129dae-e202-4f9a-8aec-a47bd5cefa6d-kube-api-access-hgcgn\") pod \"raw-sklearn-runtime-5a02e-predictor-76c678f7d7-nvwcg\" (UID: \"05129dae-e202-4f9a-8aec-a47bd5cefa6d\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-5a02e-predictor-76c678f7d7-nvwcg"
May 26 16:20:45.571992 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:20:45.571961    2556 generic.go:358] "Generic (PLEG): container finished" podID="3b1ddd12-bc61-4aff-9907-b08cd577ee3a" containerID="00458c6e2a489e62ea0390b877775e0923f3a106154eb86ded2738a48fac1a9c" exitCode=2
May 26 16:20:45.572166 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:20:45.572026    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-ea140-predictor-85487667c-ztpzz" event={"ID":"3b1ddd12-bc61-4aff-9907-b08cd577ee3a","Type":"ContainerDied","Data":"00458c6e2a489e62ea0390b877775e0923f3a106154eb86ded2738a48fac1a9c"}
May 26 16:20:45.943005 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:20:45.942961    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/05129dae-e202-4f9a-8aec-a47bd5cefa6d-proxy-tls\") pod \"raw-sklearn-runtime-5a02e-predictor-76c678f7d7-nvwcg\" (UID: \"05129dae-e202-4f9a-8aec-a47bd5cefa6d\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-5a02e-predictor-76c678f7d7-nvwcg"
May 26 16:20:45.945460 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:20:45.945427    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/05129dae-e202-4f9a-8aec-a47bd5cefa6d-proxy-tls\") pod \"raw-sklearn-runtime-5a02e-predictor-76c678f7d7-nvwcg\" (UID: \"05129dae-e202-4f9a-8aec-a47bd5cefa6d\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-5a02e-predictor-76c678f7d7-nvwcg"
May 26 16:20:46.199202 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:20:46.199118    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-5a02e-predictor-76c678f7d7-nvwcg"
May 26 16:20:46.356387 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:20:46.356363    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-5a02e-predictor-76c678f7d7-nvwcg"]
May 26 16:20:46.358863 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:20:46.358834    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05129dae_e202_4f9a_8aec_a47bd5cefa6d.slice/crio-20e1d24de7c02d34d999e18345c27baf6752bcfa53c6772d1f46d8faea209236 WatchSource:0}: Error finding container 20e1d24de7c02d34d999e18345c27baf6752bcfa53c6772d1f46d8faea209236: Status 404 returned error can't find the container with id 20e1d24de7c02d34d999e18345c27baf6752bcfa53c6772d1f46d8faea209236
May 26 16:20:46.576875 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:20:46.576787    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-5a02e-predictor-76c678f7d7-nvwcg" event={"ID":"05129dae-e202-4f9a-8aec-a47bd5cefa6d","Type":"ContainerStarted","Data":"c34c77a0e66394a4d472974a40f2e2e942e050a4f22aa3303ece8dd6eba533d0"}
May 26 16:20:46.576875 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:20:46.576822    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-5a02e-predictor-76c678f7d7-nvwcg" event={"ID":"05129dae-e202-4f9a-8aec-a47bd5cefa6d","Type":"ContainerStarted","Data":"20e1d24de7c02d34d999e18345c27baf6752bcfa53c6772d1f46d8faea209236"}
May 26 16:20:48.349274 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:20:48.349231    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-ea140-predictor-85487667c-ztpzz" podUID="3b1ddd12-bc61-4aff-9907-b08cd577ee3a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.36:8643/healthz\": dial tcp 10.133.0.36:8643: connect: connection refused"
May 26 16:20:48.354555 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:20:48.354522    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-ea140-predictor-85487667c-ztpzz" podUID="3b1ddd12-bc61-4aff-9907-b08cd577ee3a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
May 26 16:20:49.324046 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:20:49.324021    2556 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-ea140-predictor-85487667c-ztpzz"
May 26 16:20:49.372652 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:20:49.372624    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3b1ddd12-bc61-4aff-9907-b08cd577ee3a-kserve-provision-location\") pod \"3b1ddd12-bc61-4aff-9907-b08cd577ee3a\" (UID: \"3b1ddd12-bc61-4aff-9907-b08cd577ee3a\") "
May 26 16:20:49.373038 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:20:49.372668    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"raw-sklearn-ea140-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3b1ddd12-bc61-4aff-9907-b08cd577ee3a-raw-sklearn-ea140-kube-rbac-proxy-sar-config\") pod \"3b1ddd12-bc61-4aff-9907-b08cd577ee3a\" (UID: \"3b1ddd12-bc61-4aff-9907-b08cd577ee3a\") "
May 26 16:20:49.373038 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:20:49.372699    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3b1ddd12-bc61-4aff-9907-b08cd577ee3a-proxy-tls\") pod \"3b1ddd12-bc61-4aff-9907-b08cd577ee3a\" (UID: \"3b1ddd12-bc61-4aff-9907-b08cd577ee3a\") "
May 26 16:20:49.373038 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:20:49.372733    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w2hcd\" (UniqueName: \"kubernetes.io/projected/3b1ddd12-bc61-4aff-9907-b08cd577ee3a-kube-api-access-w2hcd\") pod \"3b1ddd12-bc61-4aff-9907-b08cd577ee3a\" (UID: \"3b1ddd12-bc61-4aff-9907-b08cd577ee3a\") "
May 26 16:20:49.373038 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:20:49.372976    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b1ddd12-bc61-4aff-9907-b08cd577ee3a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3b1ddd12-bc61-4aff-9907-b08cd577ee3a" (UID: "3b1ddd12-bc61-4aff-9907-b08cd577ee3a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 16:20:49.373262 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:20:49.373104    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b1ddd12-bc61-4aff-9907-b08cd577ee3a-raw-sklearn-ea140-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "raw-sklearn-ea140-kube-rbac-proxy-sar-config") pod "3b1ddd12-bc61-4aff-9907-b08cd577ee3a" (UID: "3b1ddd12-bc61-4aff-9907-b08cd577ee3a"). InnerVolumeSpecName "raw-sklearn-ea140-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 16:20:49.374748 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:20:49.374725    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b1ddd12-bc61-4aff-9907-b08cd577ee3a-kube-api-access-w2hcd" (OuterVolumeSpecName: "kube-api-access-w2hcd") pod "3b1ddd12-bc61-4aff-9907-b08cd577ee3a" (UID: "3b1ddd12-bc61-4aff-9907-b08cd577ee3a"). InnerVolumeSpecName "kube-api-access-w2hcd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 16:20:49.374836 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:20:49.374767    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b1ddd12-bc61-4aff-9907-b08cd577ee3a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3b1ddd12-bc61-4aff-9907-b08cd577ee3a" (UID: "3b1ddd12-bc61-4aff-9907-b08cd577ee3a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 16:20:49.473309 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:20:49.473256    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2hcd\" (UniqueName: \"kubernetes.io/projected/3b1ddd12-bc61-4aff-9907-b08cd577ee3a-kube-api-access-w2hcd\") on node \"ip-10-0-134-19.ec2.internal\" DevicePath \"\""
May 26 16:20:49.473309 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:20:49.473303    2556 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3b1ddd12-bc61-4aff-9907-b08cd577ee3a-kserve-provision-location\") on node \"ip-10-0-134-19.ec2.internal\" DevicePath \"\""
May 26 16:20:49.473309 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:20:49.473315    2556 reconciler_common.go:299] "Volume detached for volume \"raw-sklearn-ea140-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3b1ddd12-bc61-4aff-9907-b08cd577ee3a-raw-sklearn-ea140-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-19.ec2.internal\" DevicePath \"\""
May 26 16:20:49.473309 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:20:49.473326    2556 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3b1ddd12-bc61-4aff-9907-b08cd577ee3a-proxy-tls\") on node \"ip-10-0-134-19.ec2.internal\" DevicePath \"\""
May 26 16:20:49.596189 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:20:49.596155    2556 generic.go:358] "Generic (PLEG): container finished" podID="3b1ddd12-bc61-4aff-9907-b08cd577ee3a" containerID="e3f12bad25cd927d5583112decf16c1f412d97a51ff9b947201e4b05ff276670" exitCode=0
May 26 16:20:49.596344 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:20:49.596243    2556 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-ea140-predictor-85487667c-ztpzz"
May 26 16:20:49.596344 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:20:49.596239    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-ea140-predictor-85487667c-ztpzz" event={"ID":"3b1ddd12-bc61-4aff-9907-b08cd577ee3a","Type":"ContainerDied","Data":"e3f12bad25cd927d5583112decf16c1f412d97a51ff9b947201e4b05ff276670"}
May 26 16:20:49.596421 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:20:49.596351    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-ea140-predictor-85487667c-ztpzz" event={"ID":"3b1ddd12-bc61-4aff-9907-b08cd577ee3a","Type":"ContainerDied","Data":"9e36c5b9235ea4648f4ccd3f6e6950eaeb987b23d56eebd501af4ff6d54c66d9"}
May 26 16:20:49.596421 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:20:49.596366    2556 scope.go:117] "RemoveContainer" containerID="00458c6e2a489e62ea0390b877775e0923f3a106154eb86ded2738a48fac1a9c"
May 26 16:20:49.605056 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:20:49.605033    2556 scope.go:117] "RemoveContainer" containerID="e3f12bad25cd927d5583112decf16c1f412d97a51ff9b947201e4b05ff276670"
May 26 16:20:49.611766 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:20:49.611747    2556 scope.go:117] "RemoveContainer" containerID="3022251c3fbe10dde82b0aa53a79ee266fd77b8d59d7f5632bf2a957652c274c"
May 26 16:20:49.618180 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:20:49.618163    2556 scope.go:117] "RemoveContainer" containerID="00458c6e2a489e62ea0390b877775e0923f3a106154eb86ded2738a48fac1a9c"
May 26 16:20:49.618413 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:20:49.618396    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00458c6e2a489e62ea0390b877775e0923f3a106154eb86ded2738a48fac1a9c\": container with ID starting with 00458c6e2a489e62ea0390b877775e0923f3a106154eb86ded2738a48fac1a9c not found: ID does not exist" containerID="00458c6e2a489e62ea0390b877775e0923f3a106154eb86ded2738a48fac1a9c"
May 26 16:20:49.618460 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:20:49.618423    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00458c6e2a489e62ea0390b877775e0923f3a106154eb86ded2738a48fac1a9c"} err="failed to get container status \"00458c6e2a489e62ea0390b877775e0923f3a106154eb86ded2738a48fac1a9c\": rpc error: code = NotFound desc = could not find container \"00458c6e2a489e62ea0390b877775e0923f3a106154eb86ded2738a48fac1a9c\": container with ID starting with 00458c6e2a489e62ea0390b877775e0923f3a106154eb86ded2738a48fac1a9c not found: ID does not exist"
May 26 16:20:49.618460 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:20:49.618441    2556 scope.go:117] "RemoveContainer" containerID="e3f12bad25cd927d5583112decf16c1f412d97a51ff9b947201e4b05ff276670"
May 26 16:20:49.618685 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:20:49.618668    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3f12bad25cd927d5583112decf16c1f412d97a51ff9b947201e4b05ff276670\": container with ID starting with e3f12bad25cd927d5583112decf16c1f412d97a51ff9b947201e4b05ff276670 not found: ID does not exist" containerID="e3f12bad25cd927d5583112decf16c1f412d97a51ff9b947201e4b05ff276670"
May 26 16:20:49.618731 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:20:49.618692    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3f12bad25cd927d5583112decf16c1f412d97a51ff9b947201e4b05ff276670"} err="failed to get container status \"e3f12bad25cd927d5583112decf16c1f412d97a51ff9b947201e4b05ff276670\": rpc error: code = NotFound desc = could not find container \"e3f12bad25cd927d5583112decf16c1f412d97a51ff9b947201e4b05ff276670\": container with ID starting with e3f12bad25cd927d5583112decf16c1f412d97a51ff9b947201e4b05ff276670 not found: ID does not exist"
May 26 16:20:49.618731 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:20:49.618708    2556 scope.go:117] "RemoveContainer" containerID="3022251c3fbe10dde82b0aa53a79ee266fd77b8d59d7f5632bf2a957652c274c"
May 26 16:20:49.618915 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:20:49.618900    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3022251c3fbe10dde82b0aa53a79ee266fd77b8d59d7f5632bf2a957652c274c\": container with ID starting with 3022251c3fbe10dde82b0aa53a79ee266fd77b8d59d7f5632bf2a957652c274c not found: ID does not exist" containerID="3022251c3fbe10dde82b0aa53a79ee266fd77b8d59d7f5632bf2a957652c274c"
May 26 16:20:49.618954 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:20:49.618918    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3022251c3fbe10dde82b0aa53a79ee266fd77b8d59d7f5632bf2a957652c274c"} err="failed to get container status \"3022251c3fbe10dde82b0aa53a79ee266fd77b8d59d7f5632bf2a957652c274c\": rpc error: code = NotFound desc = could not find container \"3022251c3fbe10dde82b0aa53a79ee266fd77b8d59d7f5632bf2a957652c274c\": container with ID starting with 3022251c3fbe10dde82b0aa53a79ee266fd77b8d59d7f5632bf2a957652c274c not found: ID does not exist"
May 26 16:20:49.623254 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:20:49.623201    2556 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-ea140-predictor-85487667c-ztpzz"]
May 26 16:20:49.628041 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:20:49.628020    2556 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-ea140-predictor-85487667c-ztpzz"]
May 26 16:20:50.601013 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:20:50.600980    2556 generic.go:358] "Generic (PLEG): container finished" podID="05129dae-e202-4f9a-8aec-a47bd5cefa6d" containerID="c34c77a0e66394a4d472974a40f2e2e942e050a4f22aa3303ece8dd6eba533d0" exitCode=0
May 26 16:20:50.601463 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:20:50.601053    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-5a02e-predictor-76c678f7d7-nvwcg" event={"ID":"05129dae-e202-4f9a-8aec-a47bd5cefa6d","Type":"ContainerDied","Data":"c34c77a0e66394a4d472974a40f2e2e942e050a4f22aa3303ece8dd6eba533d0"}
May 26 16:20:50.772298 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:20:50.772265    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3b1ddd12-bc61-4aff-9907-b08cd577ee3a" path="/var/lib/kubelet/pods/3b1ddd12-bc61-4aff-9907-b08cd577ee3a/volumes"
May 26 16:20:51.606605 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:20:51.606573    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-5a02e-predictor-76c678f7d7-nvwcg" event={"ID":"05129dae-e202-4f9a-8aec-a47bd5cefa6d","Type":"ContainerStarted","Data":"af297d866f849fabd9a6685f7cd6d00fabe35ad3792a81d1270be070a4a092a1"}
May 26 16:20:51.606605 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:20:51.606610    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-5a02e-predictor-76c678f7d7-nvwcg" event={"ID":"05129dae-e202-4f9a-8aec-a47bd5cefa6d","Type":"ContainerStarted","Data":"59f6af45294f7ea527b433fbd90c856ae94f88980d03b127848ea1d07a8e77e8"}
May 26 16:20:51.607019 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:20:51.606877    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-5a02e-predictor-76c678f7d7-nvwcg"
May 26 16:20:51.607019 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:20:51.606985    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-5a02e-predictor-76c678f7d7-nvwcg"
May 26 16:20:51.608253 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:20:51.608227    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-5a02e-predictor-76c678f7d7-nvwcg" podUID="05129dae-e202-4f9a-8aec-a47bd5cefa6d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
May 26 16:20:51.632611 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:20:51.632561    2556 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/raw-sklearn-runtime-5a02e-predictor-76c678f7d7-nvwcg" podStartSLOduration=6.632545413 podStartE2EDuration="6.632545413s" podCreationTimestamp="2026-05-26 16:20:45 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 16:20:51.628883394 +0000 UTC m=+1213.473500386" watchObservedRunningTime="2026-05-26 16:20:51.632545413 +0000 UTC m=+1213.477162406"
May 26 16:20:52.610138 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:20:52.610085    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-5a02e-predictor-76c678f7d7-nvwcg" podUID="05129dae-e202-4f9a-8aec-a47bd5cefa6d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
May 26 16:20:57.620446 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:20:57.620415    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-5a02e-predictor-76c678f7d7-nvwcg"
May 26 16:20:57.621052 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:20:57.621022    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-5a02e-predictor-76c678f7d7-nvwcg" podUID="05129dae-e202-4f9a-8aec-a47bd5cefa6d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
May 26 16:21:07.621225 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:21:07.621185    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-5a02e-predictor-76c678f7d7-nvwcg" podUID="05129dae-e202-4f9a-8aec-a47bd5cefa6d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
May 26 16:21:17.621294 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:21:17.621252    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-5a02e-predictor-76c678f7d7-nvwcg" podUID="05129dae-e202-4f9a-8aec-a47bd5cefa6d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
May 26 16:21:27.621144 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:21:27.621048    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-5a02e-predictor-76c678f7d7-nvwcg" podUID="05129dae-e202-4f9a-8aec-a47bd5cefa6d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
May 26 16:21:37.621509 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:21:37.621468    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-5a02e-predictor-76c678f7d7-nvwcg" podUID="05129dae-e202-4f9a-8aec-a47bd5cefa6d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
May 26 16:21:47.621383 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:21:47.621344    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-5a02e-predictor-76c678f7d7-nvwcg" podUID="05129dae-e202-4f9a-8aec-a47bd5cefa6d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
May 26 16:21:57.622288 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:21:57.622256    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-5a02e-predictor-76c678f7d7-nvwcg"
May 26 16:22:05.420871 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:22:05.420838    2556 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-5a02e-predictor-76c678f7d7-nvwcg"]
May 26 16:22:05.421287 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:22:05.421221    2556 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-runtime-5a02e-predictor-76c678f7d7-nvwcg" podUID="05129dae-e202-4f9a-8aec-a47bd5cefa6d" containerName="kserve-container" containerID="cri-o://59f6af45294f7ea527b433fbd90c856ae94f88980d03b127848ea1d07a8e77e8" gracePeriod=30
May 26 16:22:05.421287 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:22:05.421237    2556 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-runtime-5a02e-predictor-76c678f7d7-nvwcg" podUID="05129dae-e202-4f9a-8aec-a47bd5cefa6d" containerName="kube-rbac-proxy" containerID="cri-o://af297d866f849fabd9a6685f7cd6d00fabe35ad3792a81d1270be070a4a092a1" gracePeriod=30
May 26 16:22:05.841771 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:22:05.841689    2556 generic.go:358] "Generic (PLEG): container finished" podID="05129dae-e202-4f9a-8aec-a47bd5cefa6d" containerID="af297d866f849fabd9a6685f7cd6d00fabe35ad3792a81d1270be070a4a092a1" exitCode=2
May 26 16:22:05.841913 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:22:05.841766    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-5a02e-predictor-76c678f7d7-nvwcg" event={"ID":"05129dae-e202-4f9a-8aec-a47bd5cefa6d","Type":"ContainerDied","Data":"af297d866f849fabd9a6685f7cd6d00fabe35ad3792a81d1270be070a4a092a1"}
May 26 16:22:07.610586 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:22:07.610541    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-5a02e-predictor-76c678f7d7-nvwcg" podUID="05129dae-e202-4f9a-8aec-a47bd5cefa6d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.37:8643/healthz\": dial tcp 10.133.0.37:8643: connect: connection refused"
May 26 16:22:07.621442 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:22:07.621411    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-5a02e-predictor-76c678f7d7-nvwcg" podUID="05129dae-e202-4f9a-8aec-a47bd5cefa6d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
May 26 16:22:09.456607 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:22:09.456585    2556 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-5a02e-predictor-76c678f7d7-nvwcg"
May 26 16:22:09.565076 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:22:09.565005    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/05129dae-e202-4f9a-8aec-a47bd5cefa6d-proxy-tls\") pod \"05129dae-e202-4f9a-8aec-a47bd5cefa6d\" (UID: \"05129dae-e202-4f9a-8aec-a47bd5cefa6d\") "
May 26 16:22:09.565076 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:22:09.565064    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"raw-sklearn-runtime-5a02e-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/05129dae-e202-4f9a-8aec-a47bd5cefa6d-raw-sklearn-runtime-5a02e-kube-rbac-proxy-sar-config\") pod \"05129dae-e202-4f9a-8aec-a47bd5cefa6d\" (UID: \"05129dae-e202-4f9a-8aec-a47bd5cefa6d\") "
May 26 16:22:09.565260 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:22:09.565113    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/05129dae-e202-4f9a-8aec-a47bd5cefa6d-kserve-provision-location\") pod \"05129dae-e202-4f9a-8aec-a47bd5cefa6d\" (UID: \"05129dae-e202-4f9a-8aec-a47bd5cefa6d\") "
May 26 16:22:09.565260 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:22:09.565147    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hgcgn\" (UniqueName: \"kubernetes.io/projected/05129dae-e202-4f9a-8aec-a47bd5cefa6d-kube-api-access-hgcgn\") pod \"05129dae-e202-4f9a-8aec-a47bd5cefa6d\" (UID: \"05129dae-e202-4f9a-8aec-a47bd5cefa6d\") "
May 26 16:22:09.565412 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:22:09.565390    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05129dae-e202-4f9a-8aec-a47bd5cefa6d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "05129dae-e202-4f9a-8aec-a47bd5cefa6d" (UID: "05129dae-e202-4f9a-8aec-a47bd5cefa6d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 16:22:09.565467 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:22:09.565441    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05129dae-e202-4f9a-8aec-a47bd5cefa6d-raw-sklearn-runtime-5a02e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "raw-sklearn-runtime-5a02e-kube-rbac-proxy-sar-config") pod "05129dae-e202-4f9a-8aec-a47bd5cefa6d" (UID: "05129dae-e202-4f9a-8aec-a47bd5cefa6d"). InnerVolumeSpecName "raw-sklearn-runtime-5a02e-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 16:22:09.567064 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:22:09.567045    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05129dae-e202-4f9a-8aec-a47bd5cefa6d-kube-api-access-hgcgn" (OuterVolumeSpecName: "kube-api-access-hgcgn") pod "05129dae-e202-4f9a-8aec-a47bd5cefa6d" (UID: "05129dae-e202-4f9a-8aec-a47bd5cefa6d"). InnerVolumeSpecName "kube-api-access-hgcgn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 16:22:09.567191 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:22:09.567173    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05129dae-e202-4f9a-8aec-a47bd5cefa6d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "05129dae-e202-4f9a-8aec-a47bd5cefa6d" (UID: "05129dae-e202-4f9a-8aec-a47bd5cefa6d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 16:22:09.665783 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:22:09.665742    2556 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/05129dae-e202-4f9a-8aec-a47bd5cefa6d-proxy-tls\") on node \"ip-10-0-134-19.ec2.internal\" DevicePath \"\""
May 26 16:22:09.665783 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:22:09.665781    2556 reconciler_common.go:299] "Volume detached for volume \"raw-sklearn-runtime-5a02e-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/05129dae-e202-4f9a-8aec-a47bd5cefa6d-raw-sklearn-runtime-5a02e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-19.ec2.internal\" DevicePath \"\""
May 26 16:22:09.665783 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:22:09.665793    2556 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/05129dae-e202-4f9a-8aec-a47bd5cefa6d-kserve-provision-location\") on node \"ip-10-0-134-19.ec2.internal\" DevicePath \"\""
May 26 16:22:09.665971 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:22:09.665804    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hgcgn\" (UniqueName: \"kubernetes.io/projected/05129dae-e202-4f9a-8aec-a47bd5cefa6d-kube-api-access-hgcgn\") on node \"ip-10-0-134-19.ec2.internal\" DevicePath \"\""
May 26 16:22:09.855850 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:22:09.855781    2556 generic.go:358] "Generic (PLEG): container finished" podID="05129dae-e202-4f9a-8aec-a47bd5cefa6d" containerID="59f6af45294f7ea527b433fbd90c856ae94f88980d03b127848ea1d07a8e77e8" exitCode=0
May 26 16:22:09.855850 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:22:09.855828    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-5a02e-predictor-76c678f7d7-nvwcg" event={"ID":"05129dae-e202-4f9a-8aec-a47bd5cefa6d","Type":"ContainerDied","Data":"59f6af45294f7ea527b433fbd90c856ae94f88980d03b127848ea1d07a8e77e8"}
May 26 16:22:09.855850 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:22:09.855850    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-5a02e-predictor-76c678f7d7-nvwcg" event={"ID":"05129dae-e202-4f9a-8aec-a47bd5cefa6d","Type":"ContainerDied","Data":"20e1d24de7c02d34d999e18345c27baf6752bcfa53c6772d1f46d8faea209236"}
May 26 16:22:09.856148 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:22:09.855865    2556 scope.go:117] "RemoveContainer" containerID="af297d866f849fabd9a6685f7cd6d00fabe35ad3792a81d1270be070a4a092a1"
May 26 16:22:09.856148 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:22:09.855983    2556 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-5a02e-predictor-76c678f7d7-nvwcg"
May 26 16:22:09.868861 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:22:09.868842    2556 scope.go:117] "RemoveContainer" containerID="59f6af45294f7ea527b433fbd90c856ae94f88980d03b127848ea1d07a8e77e8"
May 26 16:22:09.875436 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:22:09.875414    2556 scope.go:117] "RemoveContainer" containerID="c34c77a0e66394a4d472974a40f2e2e942e050a4f22aa3303ece8dd6eba533d0"
May 26 16:22:09.881689 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:22:09.881674    2556 scope.go:117] "RemoveContainer" containerID="af297d866f849fabd9a6685f7cd6d00fabe35ad3792a81d1270be070a4a092a1"
May 26 16:22:09.881899 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:22:09.881882    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af297d866f849fabd9a6685f7cd6d00fabe35ad3792a81d1270be070a4a092a1\": container with ID starting with af297d866f849fabd9a6685f7cd6d00fabe35ad3792a81d1270be070a4a092a1 not found: ID does not exist" containerID="af297d866f849fabd9a6685f7cd6d00fabe35ad3792a81d1270be070a4a092a1"
May 26 16:22:09.881968 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:22:09.881906    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af297d866f849fabd9a6685f7cd6d00fabe35ad3792a81d1270be070a4a092a1"} err="failed to get container status \"af297d866f849fabd9a6685f7cd6d00fabe35ad3792a81d1270be070a4a092a1\": rpc error: code = NotFound desc = could not find container \"af297d866f849fabd9a6685f7cd6d00fabe35ad3792a81d1270be070a4a092a1\": container with ID starting with af297d866f849fabd9a6685f7cd6d00fabe35ad3792a81d1270be070a4a092a1 not found: ID does not exist"
May 26 16:22:09.881968 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:22:09.881922    2556 scope.go:117] "RemoveContainer" containerID="59f6af45294f7ea527b433fbd90c856ae94f88980d03b127848ea1d07a8e77e8"
May 26 16:22:09.882122 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:22:09.882103    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59f6af45294f7ea527b433fbd90c856ae94f88980d03b127848ea1d07a8e77e8\": container with ID starting with 59f6af45294f7ea527b433fbd90c856ae94f88980d03b127848ea1d07a8e77e8 not found: ID does not exist" containerID="59f6af45294f7ea527b433fbd90c856ae94f88980d03b127848ea1d07a8e77e8"
May 26 16:22:09.882167 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:22:09.882125    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59f6af45294f7ea527b433fbd90c856ae94f88980d03b127848ea1d07a8e77e8"} err="failed to get container status \"59f6af45294f7ea527b433fbd90c856ae94f88980d03b127848ea1d07a8e77e8\": rpc error: code = NotFound desc = could not find container \"59f6af45294f7ea527b433fbd90c856ae94f88980d03b127848ea1d07a8e77e8\": container with ID starting with 59f6af45294f7ea527b433fbd90c856ae94f88980d03b127848ea1d07a8e77e8 not found: ID does not exist"
May 26 16:22:09.882167 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:22:09.882139    2556 scope.go:117] "RemoveContainer" containerID="c34c77a0e66394a4d472974a40f2e2e942e050a4f22aa3303ece8dd6eba533d0"
May 26 16:22:09.882362 ip-10-0-134-19 kubenswrapper[2556]: E0526 16:22:09.882340    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c34c77a0e66394a4d472974a40f2e2e942e050a4f22aa3303ece8dd6eba533d0\": container with ID starting with c34c77a0e66394a4d472974a40f2e2e942e050a4f22aa3303ece8dd6eba533d0 not found: ID does not exist" containerID="c34c77a0e66394a4d472974a40f2e2e942e050a4f22aa3303ece8dd6eba533d0"
May 26 16:22:09.882425 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:22:09.882371    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c34c77a0e66394a4d472974a40f2e2e942e050a4f22aa3303ece8dd6eba533d0"} err="failed to get container status \"c34c77a0e66394a4d472974a40f2e2e942e050a4f22aa3303ece8dd6eba533d0\": rpc error: code = NotFound desc = could not find container \"c34c77a0e66394a4d472974a40f2e2e942e050a4f22aa3303ece8dd6eba533d0\": container with ID starting with c34c77a0e66394a4d472974a40f2e2e942e050a4f22aa3303ece8dd6eba533d0 not found: ID does not exist"
May 26 16:22:09.885334 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:22:09.885315    2556 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-5a02e-predictor-76c678f7d7-nvwcg"]
May 26 16:22:09.890914 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:22:09.890887    2556 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-5a02e-predictor-76c678f7d7-nvwcg"]
May 26 16:22:10.771468 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:22:10.771432    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="05129dae-e202-4f9a-8aec-a47bd5cefa6d" path="/var/lib/kubelet/pods/05129dae-e202-4f9a-8aec-a47bd5cefa6d/volumes"
May 26 16:22:40.472362 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:22:40.472323    2556 ???:1] "http: TLS handshake error from 10.0.137.117:37200: EOF"
May 26 16:22:40.475338 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:22:40.475314    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-wc8kh_c0cc81a8-fc3f-4188-8e90-4640d751dfb5/global-pull-secret-syncer/0.log"
May 26 16:22:40.677281 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:22:40.677253    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-b2npd_02507d1e-2665-4380-aa87-3605e4411561/konnectivity-agent/0.log"
May 26 16:22:40.817353 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:22:40.817285    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-134-19.ec2.internal_c33db3f858f26d571aec8a6b75b8b9e2/haproxy/0.log"
May 26 16:22:44.366292 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:22:44.366263    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-665c9c8584-2v9mb_a06fc334-b590-4dad-a438-7ab5f9285f1f/monitoring-plugin/0.log"
May 26 16:22:44.532461 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:22:44.532431    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9q9lz_88e521b6-1620-4720-8a54-35939b468812/node-exporter/0.log"
May 26 16:22:44.571164 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:22:44.571139    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9q9lz_88e521b6-1620-4720-8a54-35939b468812/kube-rbac-proxy/0.log"
May 26 16:22:44.608224 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:22:44.608205    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9q9lz_88e521b6-1620-4720-8a54-35939b468812/init-textfile/0.log"
May 26 16:22:45.343651 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:22:45.343599    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-65b6b594d5-w7prx_5604c4cf-75ec-4fd0-989b-9e0316b35972/telemeter-client/0.log"
May 26 16:22:45.377037 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:22:45.377012    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-65b6b594d5-w7prx_5604c4cf-75ec-4fd0-989b-9e0316b35972/reload/0.log"
May 26 16:22:45.415701 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:22:45.415679    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-65b6b594d5-w7prx_5604c4cf-75ec-4fd0-989b-9e0316b35972/kube-rbac-proxy/0.log"
May 26 16:22:47.148051 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:22:47.148008    2556 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-must-gather-8mvc8/perf-node-gather-daemonset-rx62x"]
May 26 16:22:47.148439 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:22:47.148318    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b1ddd12-bc61-4aff-9907-b08cd577ee3a" containerName="storage-initializer"
May 26 16:22:47.148439 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:22:47.148330    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b1ddd12-bc61-4aff-9907-b08cd577ee3a" containerName="storage-initializer"
May 26 16:22:47.148439 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:22:47.148340    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b1ddd12-bc61-4aff-9907-b08cd577ee3a" containerName="kube-rbac-proxy"
May 26 16:22:47.148439 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:22:47.148346    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b1ddd12-bc61-4aff-9907-b08cd577ee3a" containerName="kube-rbac-proxy"
May 26 16:22:47.148439 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:22:47.148354    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05129dae-e202-4f9a-8aec-a47bd5cefa6d" containerName="storage-initializer"
May 26 16:22:47.148439 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:22:47.148360    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="05129dae-e202-4f9a-8aec-a47bd5cefa6d" containerName="storage-initializer"
May 26 16:22:47.148439 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:22:47.148367    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05129dae-e202-4f9a-8aec-a47bd5cefa6d" containerName="kserve-container"
May 26 16:22:47.148439 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:22:47.148372    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="05129dae-e202-4f9a-8aec-a47bd5cefa6d" containerName="kserve-container"
May 26 16:22:47.148439 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:22:47.148381    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b1ddd12-bc61-4aff-9907-b08cd577ee3a" containerName="kserve-container"
May 26 16:22:47.148439 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:22:47.148387    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b1ddd12-bc61-4aff-9907-b08cd577ee3a" containerName="kserve-container"
May 26 16:22:47.148439 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:22:47.148398    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05129dae-e202-4f9a-8aec-a47bd5cefa6d" containerName="kube-rbac-proxy"
May 26 16:22:47.148439 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:22:47.148403    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="05129dae-e202-4f9a-8aec-a47bd5cefa6d" containerName="kube-rbac-proxy"
May 26 16:22:47.148439 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:22:47.148446    2556 memory_manager.go:356] "RemoveStaleState removing state" podUID="3b1ddd12-bc61-4aff-9907-b08cd577ee3a" containerName="kube-rbac-proxy"
May 26 16:22:47.148836 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:22:47.148455    2556 memory_manager.go:356] "RemoveStaleState removing state" podUID="05129dae-e202-4f9a-8aec-a47bd5cefa6d" containerName="kserve-container"
May 26 16:22:47.148836 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:22:47.148462    2556 memory_manager.go:356] "RemoveStaleState removing state" podUID="05129dae-e202-4f9a-8aec-a47bd5cefa6d" containerName="kube-rbac-proxy"
May 26 16:22:47.148836 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:22:47.148469    2556 memory_manager.go:356] "RemoveStaleState removing state" podUID="3b1ddd12-bc61-4aff-9907-b08cd577ee3a" containerName="kserve-container"
May 26 16:22:47.152337 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:22:47.152317    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8mvc8/perf-node-gather-daemonset-rx62x"
May 26 16:22:47.155649 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:22:47.155621    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-8mvc8\"/\"kube-root-ca.crt\""
May 26 16:22:47.156936 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:22:47.156920    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-8mvc8\"/\"openshift-service-ca.crt\""
May 26 16:22:47.157029 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:22:47.156926    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-8mvc8\"/\"default-dockercfg-rbnlh\""
May 26 16:22:47.166998 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:22:47.166975    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-8mvc8/perf-node-gather-daemonset-rx62x"]
May 26 16:22:47.222345 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:22:47.222312    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/762d85cb-d1e7-49b0-ba21-1477e6886adc-sys\") pod \"perf-node-gather-daemonset-rx62x\" (UID: \"762d85cb-d1e7-49b0-ba21-1477e6886adc\") " pod="openshift-must-gather-8mvc8/perf-node-gather-daemonset-rx62x"
May 26 16:22:47.222345 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:22:47.222349    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tf7xf\" (UniqueName: \"kubernetes.io/projected/762d85cb-d1e7-49b0-ba21-1477e6886adc-kube-api-access-tf7xf\") pod \"perf-node-gather-daemonset-rx62x\" (UID: \"762d85cb-d1e7-49b0-ba21-1477e6886adc\") " pod="openshift-must-gather-8mvc8/perf-node-gather-daemonset-rx62x"
May 26 16:22:47.222516 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:22:47.222374    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/762d85cb-d1e7-49b0-ba21-1477e6886adc-podres\") pod \"perf-node-gather-daemonset-rx62x\" (UID: \"762d85cb-d1e7-49b0-ba21-1477e6886adc\") " pod="openshift-must-gather-8mvc8/perf-node-gather-daemonset-rx62x"
May 26 16:22:47.222516 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:22:47.222415    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/762d85cb-d1e7-49b0-ba21-1477e6886adc-lib-modules\") pod \"perf-node-gather-daemonset-rx62x\" (UID: \"762d85cb-d1e7-49b0-ba21-1477e6886adc\") " pod="openshift-must-gather-8mvc8/perf-node-gather-daemonset-rx62x"
May 26 16:22:47.222516 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:22:47.222434    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/762d85cb-d1e7-49b0-ba21-1477e6886adc-proc\") pod \"perf-node-gather-daemonset-rx62x\" (UID: \"762d85cb-d1e7-49b0-ba21-1477e6886adc\") " pod="openshift-must-gather-8mvc8/perf-node-gather-daemonset-rx62x"
May 26 16:22:47.322836 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:22:47.322805    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/762d85cb-d1e7-49b0-ba21-1477e6886adc-podres\") pod \"perf-node-gather-daemonset-rx62x\" (UID: \"762d85cb-d1e7-49b0-ba21-1477e6886adc\") " pod="openshift-must-gather-8mvc8/perf-node-gather-daemonset-rx62x"
May 26 16:22:47.322982 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:22:47.322848    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/762d85cb-d1e7-49b0-ba21-1477e6886adc-lib-modules\") pod \"perf-node-gather-daemonset-rx62x\" (UID: \"762d85cb-d1e7-49b0-ba21-1477e6886adc\") " pod="openshift-must-gather-8mvc8/perf-node-gather-daemonset-rx62x"
May 26 16:22:47.322982 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:22:47.322872    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/762d85cb-d1e7-49b0-ba21-1477e6886adc-proc\") pod \"perf-node-gather-daemonset-rx62x\" (UID: \"762d85cb-d1e7-49b0-ba21-1477e6886adc\") " pod="openshift-must-gather-8mvc8/perf-node-gather-daemonset-rx62x"
May 26 16:22:47.322982 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:22:47.322904    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/762d85cb-d1e7-49b0-ba21-1477e6886adc-sys\") pod \"perf-node-gather-daemonset-rx62x\" (UID: \"762d85cb-d1e7-49b0-ba21-1477e6886adc\") " pod="openshift-must-gather-8mvc8/perf-node-gather-daemonset-rx62x"
May 26 16:22:47.322982 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:22:47.322930    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-tf7xf\" (UniqueName: \"kubernetes.io/projected/762d85cb-d1e7-49b0-ba21-1477e6886adc-kube-api-access-tf7xf\") pod \"perf-node-gather-daemonset-rx62x\" (UID: \"762d85cb-d1e7-49b0-ba21-1477e6886adc\") " pod="openshift-must-gather-8mvc8/perf-node-gather-daemonset-rx62x"
May 26 16:22:47.322982 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:22:47.322967    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/762d85cb-d1e7-49b0-ba21-1477e6886adc-lib-modules\") pod \"perf-node-gather-daemonset-rx62x\" (UID: \"762d85cb-d1e7-49b0-ba21-1477e6886adc\") " pod="openshift-must-gather-8mvc8/perf-node-gather-daemonset-rx62x"
May 26 16:22:47.322982 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:22:47.322969    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/762d85cb-d1e7-49b0-ba21-1477e6886adc-podres\") pod \"perf-node-gather-daemonset-rx62x\" (UID: \"762d85cb-d1e7-49b0-ba21-1477e6886adc\") " pod="openshift-must-gather-8mvc8/perf-node-gather-daemonset-rx62x"
May 26 16:22:47.323297 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:22:47.323000    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/762d85cb-d1e7-49b0-ba21-1477e6886adc-proc\") pod \"perf-node-gather-daemonset-rx62x\" (UID: \"762d85cb-d1e7-49b0-ba21-1477e6886adc\") " pod="openshift-must-gather-8mvc8/perf-node-gather-daemonset-rx62x"
May 26 16:22:47.323297 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:22:47.323005    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/762d85cb-d1e7-49b0-ba21-1477e6886adc-sys\") pod \"perf-node-gather-daemonset-rx62x\" (UID: \"762d85cb-d1e7-49b0-ba21-1477e6886adc\") " pod="openshift-must-gather-8mvc8/perf-node-gather-daemonset-rx62x"
May 26 16:22:47.335064 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:22:47.335036    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tf7xf\" (UniqueName: \"kubernetes.io/projected/762d85cb-d1e7-49b0-ba21-1477e6886adc-kube-api-access-tf7xf\") pod \"perf-node-gather-daemonset-rx62x\" (UID: \"762d85cb-d1e7-49b0-ba21-1477e6886adc\") " pod="openshift-must-gather-8mvc8/perf-node-gather-daemonset-rx62x"
May 26 16:22:47.462021 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:22:47.461956    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8mvc8/perf-node-gather-daemonset-rx62x"
May 26 16:22:47.587057 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:22:47.587035    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-8mvc8/perf-node-gather-daemonset-rx62x"]
May 26 16:22:47.589658 ip-10-0-134-19 kubenswrapper[2556]: W0526 16:22:47.589630    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod762d85cb_d1e7_49b0_ba21_1477e6886adc.slice/crio-e154e0c4da986a967304d4065ac79a6b5a5cc4058c427775f9f0931cf3a078cf WatchSource:0}: Error finding container e154e0c4da986a967304d4065ac79a6b5a5cc4058c427775f9f0931cf3a078cf: Status 404 returned error can't find the container with id e154e0c4da986a967304d4065ac79a6b5a5cc4058c427775f9f0931cf3a078cf
May 26 16:22:47.591585 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:22:47.591568    2556 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 26 16:22:47.976944 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:22:47.976911    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8mvc8/perf-node-gather-daemonset-rx62x" event={"ID":"762d85cb-d1e7-49b0-ba21-1477e6886adc","Type":"ContainerStarted","Data":"8ed490b297d0664be0ea80b92d0421201cbf9aa038dea16487fc6e15dc6bee69"}
May 26 16:22:47.976944 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:22:47.976947    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8mvc8/perf-node-gather-daemonset-rx62x" event={"ID":"762d85cb-d1e7-49b0-ba21-1477e6886adc","Type":"ContainerStarted","Data":"e154e0c4da986a967304d4065ac79a6b5a5cc4058c427775f9f0931cf3a078cf"}
May 26 16:22:47.977205 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:22:47.977082    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-8mvc8/perf-node-gather-daemonset-rx62x"
May 26 16:22:47.996988 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:22:47.996947    2556 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-must-gather-8mvc8/perf-node-gather-daemonset-rx62x" podStartSLOduration=0.996934747 podStartE2EDuration="996.934747ms" podCreationTimestamp="2026-05-26 16:22:47 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 16:22:47.99503417 +0000 UTC m=+1329.839651163" watchObservedRunningTime="2026-05-26 16:22:47.996934747 +0000 UTC m=+1329.841551739"
May 26 16:22:49.037936 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:22:49.037910    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-blmvt_a131cc32-eb9c-4881-9af5-0985a6ad3b40/dns/0.log"
May 26 16:22:49.065561 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:22:49.065534    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-blmvt_a131cc32-eb9c-4881-9af5-0985a6ad3b40/kube-rbac-proxy/0.log"
May 26 16:22:49.211595 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:22:49.211567    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-ghdqr_5f8c28cc-c746-4f30-a75d-4d7c1a6a5475/dns-node-resolver/0.log"
May 26 16:22:49.821028 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:22:49.821000    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-67474545cf-rgctv_4a9f9941-c5fa-4e05-97f4-23aeefb46d13/registry/0.log"
May 26 16:22:49.906149 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:22:49.906122    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-lrkmg_bc458685-6108-4373-aa17-f6e49f2c6e1b/node-ca/0.log"
May 26 16:22:51.165248 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:22:51.165219    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-tkpb6_001b5d49-d123-4cec-85c4-e7f8927bf13a/serve-healthcheck-canary/0.log"
May 26 16:22:51.686474 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:22:51.686446    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-cq5gn_e3f3da40-e5da-4eab-a2bb-b7f75f9546c8/kube-rbac-proxy/0.log"
May 26 16:22:51.714808 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:22:51.714781    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-cq5gn_e3f3da40-e5da-4eab-a2bb-b7f75f9546c8/exporter/0.log"
May 26 16:22:51.743257 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:22:51.743234    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-cq5gn_e3f3da40-e5da-4eab-a2bb-b7f75f9546c8/extractor/0.log"
May 26 16:22:53.988992 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:22:53.988966    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-8mvc8/perf-node-gather-daemonset-rx62x"
May 26 16:22:54.151043 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:22:54.151015    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_llmisvc-controller-manager-6b94ff949c-szqfm_e628d10c-677b-48df-b75f-6efffbabe7fa/manager/0.log"
May 26 16:22:54.188295 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:22:54.188270    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-86f7b4b499-rkhfm_a2c73ac5-470d-4e56-9b23-03e3bdf4dfa8/server/0.log"
May 26 16:22:54.325854 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:22:54.325791    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-696fc77849-fhsd8_053a18f4-17b7-4dd4-9108-0e418f05b305/manager/0.log"
May 26 16:22:54.409852 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:22:54.409830    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-79f457656b-796rp_be62ba6f-4cd8-486d-8d5c-b20fd810d803/seaweedfs/0.log"
May 26 16:22:59.274000 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:22:59.273968    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-b985f5d89-62f95_39981191-a506-435c-8237-74dffe29862a/migrator/0.log"
May 26 16:22:59.321153 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:22:59.321122    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-b985f5d89-62f95_39981191-a506-435c-8237-74dffe29862a/graceful-termination/0.log"
May 26 16:23:00.903460 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:23:00.903435    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9znf9_391025fc-21eb-4786-8a0b-7f81c9eb90a4/kube-multus-additional-cni-plugins/0.log"
May 26 16:23:00.934816 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:23:00.934777    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9znf9_391025fc-21eb-4786-8a0b-7f81c9eb90a4/egress-router-binary-copy/0.log"
May 26 16:23:00.974438 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:23:00.974411    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9znf9_391025fc-21eb-4786-8a0b-7f81c9eb90a4/cni-plugins/0.log"
May 26 16:23:01.004901 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:23:01.004881    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9znf9_391025fc-21eb-4786-8a0b-7f81c9eb90a4/bond-cni-plugin/0.log"
May 26 16:23:01.037831 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:23:01.037808    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9znf9_391025fc-21eb-4786-8a0b-7f81c9eb90a4/routeoverride-cni/0.log"
May 26 16:23:01.071179 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:23:01.071159    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9znf9_391025fc-21eb-4786-8a0b-7f81c9eb90a4/whereabouts-cni-bincopy/0.log"
May 26 16:23:01.108063 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:23:01.108036    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9znf9_391025fc-21eb-4786-8a0b-7f81c9eb90a4/whereabouts-cni/0.log"
May 26 16:23:01.834982 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:23:01.834953    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-zc768_6b5a0115-0b0c-4d86-8e58-732e7814f4c9/kube-multus/0.log"
May 26 16:23:02.076708 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:23:02.076681    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-g9scb_aed39145-780f-4642-af53-ddccbc03c34c/network-metrics-daemon/0.log"
May 26 16:23:02.110481 ip-10-0-134-19 kubenswrapper[2556]: I0526 16:23:02.110418    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-g9scb_aed39145-780f-4642-af53-ddccbc03c34c/kube-rbac-proxy/0.log"