May 25 11:14:47.362629 ip-10-0-136-227 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
May 25 11:14:47.362641 ip-10-0-136-227 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
May 25 11:14:47.362650 ip-10-0-136-227 systemd[1]: kubelet.service: Failed with result 'resources'.
May 25 11:14:47.362960 ip-10-0-136-227 systemd[1]: Failed to start Kubernetes Kubelet.
May 25 11:14:57.380496 ip-10-0-136-227 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
May 25 11:14:57.380517 ip-10-0-136-227 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 7852e52987884fa0a7b4e97358965ee9 --
May 25 11:17:06.323532 ip-10-0-136-227 systemd[1]: Starting Kubernetes Kubelet...
May 25 11:17:06.976379 ip-10-0-136-227 kubenswrapper[2561]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
May 25 11:17:06.976379 ip-10-0-136-227 kubenswrapper[2561]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
May 25 11:17:06.976379 ip-10-0-136-227 kubenswrapper[2561]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
May 25 11:17:06.976379 ip-10-0-136-227 kubenswrapper[2561]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
May 25 11:17:06.976379 ip-10-0-136-227 kubenswrapper[2561]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
May 25 11:17:06.976379 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.840190    2561 server.go:212] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
May 25 11:17:06.976379 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.848477    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 25 11:17:06.976379 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.848489    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 25 11:17:06.976379 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.848492    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 25 11:17:06.976379 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.848495    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 25 11:17:06.976379 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.848498    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 25 11:17:06.976379 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.848501    2561 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 25 11:17:06.976379 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.848504    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 25 11:17:06.976379 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.848507    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 25 11:17:06.977887 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.848509    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 25 11:17:06.977887 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.848514    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 25 11:17:06.977887 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.848516    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 25 11:17:06.977887 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.848524    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 25 11:17:06.977887 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.848527    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 25 11:17:06.977887 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.848530    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 25 11:17:06.977887 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.848532    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 25 11:17:06.977887 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.848535    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 25 11:17:06.977887 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.848538    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 25 11:17:06.977887 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.848540    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 25 11:17:06.977887 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.848543    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 25 11:17:06.977887 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.848545    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 25 11:17:06.977887 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.848548    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 25 11:17:06.977887 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.848550    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 25 11:17:06.977887 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.848553    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 25 11:17:06.977887 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.848555    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 25 11:17:06.977887 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.848558    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 25 11:17:06.977887 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.848561    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 25 11:17:06.977887 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.848563    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 25 11:17:06.977887 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.848566    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 25 11:17:06.957568 ip-10-0-136-227 systemd[1]: Started Kubernetes Kubelet.
May 25 11:17:06.978829 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.848569    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 25 11:17:06.978829 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.848571    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 25 11:17:06.978829 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.848575    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 25 11:17:06.978829 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.848581    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 25 11:17:06.978829 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.848585    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 25 11:17:06.978829 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.848588    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 25 11:17:06.978829 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.848591    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 25 11:17:06.978829 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.848594    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
May 25 11:17:06.978829 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.848597    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 25 11:17:06.978829 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.848599    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 25 11:17:06.978829 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.848610    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 25 11:17:06.978829 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.848612    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 25 11:17:06.978829 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.848615    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 25 11:17:06.978829 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.848618    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 25 11:17:06.978829 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.848620    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
May 25 11:17:06.978829 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.848623    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 25 11:17:06.978829 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.848625    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 25 11:17:06.978829 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.848628    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 25 11:17:06.978829 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.848631    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 25 11:17:06.978829 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.848633    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 25 11:17:06.979986 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.848636    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 25 11:17:06.979986 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.848638    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 25 11:17:06.979986 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.848640    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 25 11:17:06.979986 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.848643    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 25 11:17:06.979986 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.848646    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 25 11:17:06.979986 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.848648    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 25 11:17:06.979986 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.848651    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 25 11:17:06.979986 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.848653    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 25 11:17:06.979986 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.848661    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 25 11:17:06.979986 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.848665    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 25 11:17:06.979986 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.848669    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 25 11:17:06.979986 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.848673    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 25 11:17:06.979986 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.848676    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 25 11:17:06.979986 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.848679    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 25 11:17:06.979986 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.848682    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 25 11:17:06.979986 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.848685    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 25 11:17:06.979986 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.848688    2561 feature_gate.go:328] unrecognized feature gate: Example2
May 25 11:17:06.979986 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.848691    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 25 11:17:06.979986 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.848693    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 25 11:17:06.980827 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.848696    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 25 11:17:06.980827 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.848699    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 25 11:17:06.980827 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.848702    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 25 11:17:06.980827 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.848704    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 25 11:17:06.980827 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.848713    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 25 11:17:06.980827 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.848716    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 25 11:17:06.980827 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.848718    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 25 11:17:06.980827 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.848720    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 25 11:17:06.980827 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.848723    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 25 11:17:06.980827 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.848726    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 25 11:17:06.980827 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.848728    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 25 11:17:06.980827 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.848731    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 25 11:17:06.980827 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.848733    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 25 11:17:06.980827 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.848736    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 25 11:17:06.980827 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.848738    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 25 11:17:06.980827 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.848741    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 25 11:17:06.980827 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.848743    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 25 11:17:06.980827 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.848746    2561 feature_gate.go:328] unrecognized feature gate: Example
May 25 11:17:06.980827 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.848748    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 25 11:17:06.980827 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.849149    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 25 11:17:06.981861 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.849153    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 25 11:17:06.981861 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.849156    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 25 11:17:06.981861 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.849159    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 25 11:17:06.981861 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.849162    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 25 11:17:06.981861 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.849165    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 25 11:17:06.981861 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.849168    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 25 11:17:06.981861 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.849171    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 25 11:17:06.981861 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.849174    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 25 11:17:06.981861 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.849177    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 25 11:17:06.981861 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.849179    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 25 11:17:06.981861 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.849182    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 25 11:17:06.981861 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.849185    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 25 11:17:06.981861 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.849187    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 25 11:17:06.981861 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.849192    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 25 11:17:06.981861 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.849195    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 25 11:17:06.981861 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.849198    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 25 11:17:06.981861 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.849202    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 25 11:17:06.981861 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.849205    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 25 11:17:06.981861 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.849208    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 25 11:17:06.982802 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.849211    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
May 25 11:17:06.982802 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.849214    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 25 11:17:06.982802 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.849217    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 25 11:17:06.982802 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.849220    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 25 11:17:06.982802 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.849223    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 25 11:17:06.982802 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.849225    2561 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 25 11:17:06.982802 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.849228    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 25 11:17:06.982802 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.849230    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 25 11:17:06.982802 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.849233    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 25 11:17:06.982802 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.849236    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 25 11:17:06.982802 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.849238    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 25 11:17:06.982802 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.849241    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 25 11:17:06.982802 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.849244    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
May 25 11:17:06.982802 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.849246    2561 feature_gate.go:328] unrecognized feature gate: Example
May 25 11:17:06.982802 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.849249    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 25 11:17:06.982802 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.849252    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 25 11:17:06.982802 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.849254    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 25 11:17:06.982802 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.849257    2561 feature_gate.go:328] unrecognized feature gate: Example2
May 25 11:17:06.982802 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.849260    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 25 11:17:06.982802 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.849262    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 25 11:17:06.983743 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.849281    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 25 11:17:06.983743 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.849284    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 25 11:17:06.983743 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.849286    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 25 11:17:06.983743 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.849289    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 25 11:17:06.983743 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.849292    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 25 11:17:06.983743 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.849294    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 25 11:17:06.983743 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.849297    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 25 11:17:06.983743 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.849299    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 25 11:17:06.983743 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.849302    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 25 11:17:06.983743 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.849304    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 25 11:17:06.983743 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.849308    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 25 11:17:06.983743 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.849312    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 25 11:17:06.983743 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.849314    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 25 11:17:06.983743 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.849317    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 25 11:17:06.983743 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.849320    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 25 11:17:06.983743 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.849322    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 25 11:17:06.983743 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.849325    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 25 11:17:06.983743 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.849327    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 25 11:17:06.983743 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.849330    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 25 11:17:06.983743 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.849333    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 25 11:17:06.984710 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.849335    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 25 11:17:06.984710 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.849338    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 25 11:17:06.984710 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.849344    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 25 11:17:06.984710 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.849347    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 25 11:17:06.984710 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.849349    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 25 11:17:06.984710 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.849352    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 25 11:17:06.984710 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.849354    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 25 11:17:06.984710 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.849357    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 25 11:17:06.984710 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.849360    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 25 11:17:06.984710 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.849363    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 25 11:17:06.984710 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.849366    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 25 11:17:06.984710 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.849370    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 25 11:17:06.984710 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.849373    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 25 11:17:06.984710 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.849375    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 25 11:17:06.984710 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.849378    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 25 11:17:06.984710 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.849381    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 25 11:17:06.984710 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.849395    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 25 11:17:06.984710 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.849400    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 25 11:17:06.984710 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.849404    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 25 11:17:06.985412 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.849407    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 25 11:17:06.985412 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.849411    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 25 11:17:06.985412 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.849415    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 25 11:17:06.985412 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.849418    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 25 11:17:06.985412 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.849421    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 25 11:17:06.985412 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.849424    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 25 11:17:06.985412 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.849428    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 25 11:17:06.985412 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.850237    2561 flags.go:64] FLAG: --address="0.0.0.0"
May 25 11:17:06.985412 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.850255    2561 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
May 25 11:17:06.985412 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.850263    2561 flags.go:64] FLAG: --anonymous-auth="true"
May 25 11:17:06.985412 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.850282    2561 flags.go:64] FLAG: --application-metrics-count-limit="100"
May 25 11:17:06.985412 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.850286    2561 flags.go:64] FLAG: --authentication-token-webhook="false"
May 25 11:17:06.985412 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.850290    2561 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
May 25 11:17:06.985412 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.850294    2561 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
May 25 11:17:06.985412 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.850298    2561 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
May 25 11:17:06.985412 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.850303    2561 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
May 25 11:17:06.985412 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.850307    2561 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
May 25 11:17:06.985412 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.850310    2561 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
May 25 11:17:06.985412 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.850314    2561 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
May 25 11:17:06.985412 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.850317    2561 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
May 25 11:17:06.985412 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.850320    2561 flags.go:64] FLAG: --cgroup-root=""
May 25 11:17:06.985412 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.850323    2561 flags.go:64] FLAG: --cgroups-per-qos="true"
May 25 11:17:06.985412 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.850326    2561 flags.go:64] FLAG: --client-ca-file=""
May 25 11:17:06.986113 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.850329    2561 flags.go:64] FLAG: --cloud-config=""
May 25 11:17:06.986113 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.850332    2561 flags.go:64] FLAG: --cloud-provider="external"
May 25 11:17:06.986113 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.850335    2561 flags.go:64] FLAG: --cluster-dns="[]"
May 25 11:17:06.986113 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.850338    2561 flags.go:64] FLAG: --cluster-domain=""
May 25 11:17:06.986113 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.850341    2561 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
May 25 11:17:06.986113 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.850344    2561 flags.go:64] FLAG: --config-dir=""
May 25 11:17:06.986113 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.850348    2561 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
May 25 11:17:06.986113 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.850351    2561 flags.go:64] FLAG: --container-log-max-files="5"
May 25 11:17:06.986113 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.850355    2561 flags.go:64] FLAG: --container-log-max-size="10Mi"
May 25 11:17:06.986113 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.850358    2561 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
May 25 11:17:06.986113 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.850361    2561 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
May 25 11:17:06.986113 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.850364    2561 flags.go:64] FLAG: --containerd-namespace="k8s.io"
May 25 11:17:06.986113 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.850367    2561 flags.go:64] FLAG: --contention-profiling="false"
May 25 11:17:06.986113 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.850370    2561 flags.go:64] FLAG: --cpu-cfs-quota="true"
May 25 11:17:06.986113 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.850375    2561 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
May 25 11:17:06.986113 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.850378    2561 flags.go:64] FLAG: --cpu-manager-policy="none"
May 25 11:17:06.986113 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.850381    2561 flags.go:64] FLAG: --cpu-manager-policy-options=""
May 25 11:17:06.986113 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.850385    2561 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
May 25 11:17:06.986113 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.850388    2561 flags.go:64] FLAG: --enable-controller-attach-detach="true"
May 25 11:17:06.986113 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.850391    2561 flags.go:64] FLAG: --enable-debugging-handlers="true"
May 25 11:17:06.986113 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.850394    2561 flags.go:64] FLAG: --enable-load-reader="false"
May 25 11:17:06.986113 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.850397    2561 flags.go:64] FLAG: --enable-server="true"
May 25 11:17:06.986113 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.850400    2561 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
May 25 11:17:06.986113 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.850405    2561 flags.go:64] FLAG: --event-burst="100"
May 25 11:17:06.986113 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.850410    2561 flags.go:64] FLAG: --event-qps="50"
May 25 11:17:06.986765 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.850413    2561 flags.go:64] FLAG: --event-storage-age-limit="default=0"
May 25 11:17:06.986765 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.850416    2561 flags.go:64] FLAG: --event-storage-event-limit="default=0"
May 25 11:17:06.986765 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.850419    2561 flags.go:64] FLAG: --eviction-hard=""
May 25 11:17:06.986765 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.850423    2561 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
May 25 11:17:06.986765 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.850426    2561 flags.go:64] FLAG: --eviction-minimum-reclaim=""
May 25 11:17:06.986765 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.850431    2561 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
May 25 11:17:06.986765 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.850435    2561 flags.go:64] FLAG: --eviction-soft=""
May 25 11:17:06.986765 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.850438    2561 flags.go:64] FLAG: --eviction-soft-grace-period=""
May 25 11:17:06.986765 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.850441    2561 flags.go:64] FLAG: --exit-on-lock-contention="false"
May 25 11:17:06.986765 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.850444    2561 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
May 25 11:17:06.986765 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.850447    2561 flags.go:64] FLAG: --experimental-mounter-path=""
May 25 11:17:06.986765 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.850450    2561 flags.go:64] FLAG: --fail-cgroupv1="false"
May 25 11:17:06.986765 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.850453    2561 flags.go:64] FLAG: --fail-swap-on="true"
May 25 11:17:06.986765 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.850456    2561 flags.go:64] FLAG: --feature-gates=""
May 25 11:17:06.986765 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.850460    2561 flags.go:64] FLAG: --file-check-frequency="20s"
May 25 11:17:06.986765 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.850463    2561 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
May 25 11:17:06.986765 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.850466    2561 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
May 25 11:17:06.986765 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.850469    2561 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
May 25 11:17:06.986765 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.850472    2561 flags.go:64] FLAG: --healthz-port="10248"
May 25 11:17:06.986765 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.850475    2561 flags.go:64] FLAG: --help="false"
May 25 11:17:06.986765 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.850478    2561 flags.go:64] FLAG: --hostname-override="ip-10-0-136-227.ec2.internal"
May 25 11:17:06.986765 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.850481    2561 flags.go:64] FLAG: --housekeeping-interval="10s"
May 25 11:17:06.986765 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.850485    2561 flags.go:64] FLAG: --http-check-frequency="20s"
May 25 11:17:06.986765 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.850488    2561 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
May 25 11:17:06.987396 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.850492    2561 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
May 25 11:17:06.987396 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.850495    2561 flags.go:64] FLAG: --image-gc-high-threshold="85"
May 25 11:17:06.987396 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.850498    2561 flags.go:64] FLAG: --image-gc-low-threshold="80"
May 25 11:17:06.987396 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.850500    2561 flags.go:64] FLAG: --image-service-endpoint=""
May 25 11:17:06.987396 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.850503    2561 flags.go:64] FLAG: --kernel-memcg-notification="false"
May 25 11:17:06.987396 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.850506    2561 flags.go:64] FLAG: --kube-api-burst="100"
May 25 11:17:06.987396 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.850509    2561 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
May 25 11:17:06.987396 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.850513    2561 flags.go:64] FLAG: --kube-api-qps="50"
May 25 11:17:06.987396 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.850516    2561 flags.go:64] FLAG: --kube-reserved=""
May 25 11:17:06.987396 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.850519    2561 flags.go:64] FLAG: --kube-reserved-cgroup=""
May 25 11:17:06.987396 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.850522    2561 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
May 25 11:17:06.987396 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.850525    2561 flags.go:64] FLAG: --kubelet-cgroups=""
May 25 11:17:06.987396 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.850528    2561 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
May 25 11:17:06.987396 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.850533    2561 flags.go:64] FLAG: --lock-file=""
May 25 11:17:06.987396 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.850536    2561 flags.go:64] FLAG: --log-cadvisor-usage="false"
May 25 11:17:06.987396 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.850539    2561 flags.go:64] FLAG: --log-flush-frequency="5s"
May 25 11:17:06.987396 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.850542    2561 flags.go:64] FLAG: --log-json-info-buffer-size="0"
May 25 11:17:06.987396 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.850547    2561 flags.go:64] FLAG: --log-json-split-stream="false"
May 25 11:17:06.987396 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.850550    2561 flags.go:64] FLAG: --log-text-info-buffer-size="0"
May 25 11:17:06.987396 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.850553    2561 flags.go:64] FLAG: --log-text-split-stream="false"
May 25 11:17:06.987396 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.850556    2561 flags.go:64] FLAG: --logging-format="text"
May 25 11:17:06.987396 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.850559    2561 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
May 25 11:17:06.987396 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.850562    2561 flags.go:64] FLAG: --make-iptables-util-chains="true"
May 25 11:17:06.987396 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.850565    2561 flags.go:64] FLAG: --manifest-url=""
May 25 11:17:06.987989 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.850568    2561 flags.go:64] FLAG: --manifest-url-header=""
May 25 11:17:06.987989 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.850572    2561 flags.go:64] FLAG: --max-housekeeping-interval="15s"
May 25 11:17:06.987989 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.850575    2561 flags.go:64] FLAG: --max-open-files="1000000"
May 25 11:17:06.987989 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.850579    2561 flags.go:64] FLAG: --max-pods="110"
May 25 11:17:06.987989 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.850582    2561 flags.go:64] FLAG: --maximum-dead-containers="-1"
May 25 11:17:06.987989 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.850585    2561 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
May 25 11:17:06.987989 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.850588    2561 flags.go:64] FLAG: --memory-manager-policy="None"
May 25 11:17:06.987989 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.850592    2561 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
May 25 11:17:06.987989 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.850595    2561 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
May 25 11:17:06.987989 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.850598    2561 flags.go:64] FLAG: --node-ip="0.0.0.0"
May 25 11:17:06.987989 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.850601    2561 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
May 25 11:17:06.987989 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.850609    2561 flags.go:64] FLAG: --node-status-max-images="50"
May 25 11:17:06.987989 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.850612    2561 flags.go:64] FLAG: --node-status-update-frequency="10s"
May 25 11:17:06.987989 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.850615    2561 flags.go:64] FLAG: --oom-score-adj="-999"
May 25 11:17:06.987989 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.850618    2561 flags.go:64] FLAG: --pod-cidr=""
May 25 11:17:06.987989 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.850622    2561 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:afe88671bf163c42b467d159dc6f0b784dbe28b1ed44e171edc9c06fd99cc6fc"
May 25 11:17:06.987989 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.850627    2561 flags.go:64] FLAG: --pod-manifest-path=""
May 25 11:17:06.987989 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.850630    2561 flags.go:64] FLAG: --pod-max-pids="-1"
May 25 11:17:06.987989 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.850633    2561 flags.go:64] FLAG: --pods-per-core="0"
May 25 11:17:06.987989 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.850636    2561 flags.go:64] FLAG: --port="10250"
May 25 11:17:06.987989 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.850639    2561 flags.go:64] FLAG: --protect-kernel-defaults="false"
May 25 11:17:06.987989 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.850643    2561 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0d22a5887c91f4122"
May 25 11:17:06.987989 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.850647    2561 flags.go:64] FLAG: --qos-reserved=""
May 25 11:17:06.987989 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.850650    2561 flags.go:64] FLAG: --read-only-port="10255"
May 25 11:17:06.988573 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.850653    2561 flags.go:64] FLAG: --register-node="true"
May 25 11:17:06.988573 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.850656    2561 flags.go:64] FLAG: --register-schedulable="true"
May 25 11:17:06.988573 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.850659    2561 flags.go:64] FLAG: --register-with-taints=""
May 25 11:17:06.988573 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.850667    2561 flags.go:64] FLAG: --registry-burst="10"
May 25 11:17:06.988573 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.850670    2561 flags.go:64] FLAG: --registry-qps="5"
May 25 11:17:06.988573 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.850673    2561 flags.go:64] FLAG: --reserved-cpus=""
May 25 11:17:06.988573 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.850676    2561 flags.go:64] FLAG: --reserved-memory=""
May 25 11:17:06.988573 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.850679    2561 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
May 25 11:17:06.988573 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.850682    2561 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
May 25 11:17:06.988573 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.850685    2561 flags.go:64] FLAG: --rotate-certificates="false"
May 25 11:17:06.988573 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.850688    2561 flags.go:64] FLAG: --rotate-server-certificates="false"
May 25 11:17:06.988573 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.850691    2561 flags.go:64] FLAG: --runonce="false"
May 25 11:17:06.988573 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.850694    2561 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
May 25 11:17:06.988573 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.850697    2561 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
May 25 11:17:06.988573 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.850700    2561 flags.go:64] FLAG: --seccomp-default="false"
May 25 11:17:06.988573 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.850703    2561 flags.go:64] FLAG: --serialize-image-pulls="true"
May 25 11:17:06.988573 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.850707    2561 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
May 25 11:17:06.988573 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.850710    2561 flags.go:64] FLAG: --storage-driver-db="cadvisor"
May 25 11:17:06.988573 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.850713    2561 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
May 25 11:17:06.988573 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.850716    2561 flags.go:64] FLAG: --storage-driver-password="root"
May 25 11:17:06.988573 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.850719    2561 flags.go:64] FLAG: --storage-driver-secure="false"
May 25 11:17:06.988573 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.850722    2561 flags.go:64] FLAG: --storage-driver-table="stats"
May 25 11:17:06.988573 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.850725    2561 flags.go:64] FLAG: --storage-driver-user="root"
May 25 11:17:06.988573 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.850728    2561 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
May 25 11:17:06.988573 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.850732    2561 flags.go:64] FLAG: --sync-frequency="1m0s"
May 25 11:17:06.988573 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.850735    2561 flags.go:64] FLAG: --system-cgroups=""
May 25 11:17:06.989254 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.850738    2561 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
May 25 11:17:06.989254 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.850744    2561 flags.go:64] FLAG: --system-reserved-cgroup=""
May 25 11:17:06.989254 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.850746    2561 flags.go:64] FLAG: --tls-cert-file=""
May 25 11:17:06.989254 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.850751    2561 flags.go:64] FLAG: --tls-cipher-suites="[]"
May 25 11:17:06.989254 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.850755    2561 flags.go:64] FLAG: --tls-min-version=""
May 25 11:17:06.989254 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.850758    2561 flags.go:64] FLAG: --tls-private-key-file=""
May 25 11:17:06.989254 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.850761    2561 flags.go:64] FLAG: --topology-manager-policy="none"
May 25 11:17:06.989254 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.850763    2561 flags.go:64] FLAG: --topology-manager-policy-options=""
May 25 11:17:06.989254 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.850767    2561 flags.go:64] FLAG: --topology-manager-scope="container"
May 25 11:17:06.989254 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.850769    2561 flags.go:64] FLAG: --v="2"
May 25 11:17:06.989254 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.850776    2561 flags.go:64] FLAG: --version="false"
May 25 11:17:06.989254 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.850781    2561 flags.go:64] FLAG: --vmodule=""
May 25 11:17:06.989254 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.850785    2561 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
May 25 11:17:06.989254 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.850789    2561 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
May 25 11:17:06.989254 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.850878    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 25 11:17:06.989254 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.850881    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 25 11:17:06.989254 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.850885    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 25 11:17:06.989254 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.850888    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 25 11:17:06.989254 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.850891    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 25 11:17:06.989254 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.850894    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 25 11:17:06.989254 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.850897    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 25 11:17:06.989254 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.850900    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 25 11:17:06.989254 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.850903    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 25 11:17:06.989852 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.850906    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 25 11:17:06.989852 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.850908    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 25 11:17:06.989852 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.850911    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 25 11:17:06.989852 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.850914    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 25 11:17:06.989852 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.850916    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 25 11:17:06.989852 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.850919    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 25 11:17:06.989852 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.850922    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 25 11:17:06.989852 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.850925    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 25 11:17:06.989852 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.850928    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 25 11:17:06.989852 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.850931    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 25 11:17:06.989852 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.850933    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 25 11:17:06.989852 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.850936    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 25 11:17:06.989852 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.850940    2561 feature_gate.go:328] unrecognized feature gate: Example
May 25 11:17:06.989852 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.850942    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 25 11:17:06.989852 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.850945    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
May 25 11:17:06.989852 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.850947    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 25 11:17:06.989852 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.850950    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 25 11:17:06.989852 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.850953    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 25 11:17:06.989852 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.850955    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 25 11:17:06.989852 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.850958    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 25 11:17:06.990434 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.850961    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 25 11:17:06.990434 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.850964    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 25 11:17:06.990434 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.850966    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 25 11:17:06.990434 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.850969    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 25 11:17:06.990434 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.850971    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 25 11:17:06.990434 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.850974    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 25 11:17:06.990434 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.850976    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 25 11:17:06.990434 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.850994    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 25 11:17:06.990434 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.850998    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 25 11:17:06.990434 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.851001    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
May 25 11:17:06.990434 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.851004    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 25 11:17:06.990434 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.851006    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 25 11:17:06.990434 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.851009    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 25 11:17:06.990434 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.851012    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 25 11:17:06.990434 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.851015    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 25 11:17:06.990434 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.851018    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 25 11:17:06.990434 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.851020    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 25 11:17:06.990434 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.851023    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 25 11:17:06.990434 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.851025    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 25 11:17:06.990434 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.851029    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 25 11:17:06.991095 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.851032    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 25 11:17:06.991095 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.851034    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 25 11:17:06.991095 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.851038    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 25 11:17:06.991095 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.851042    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 25 11:17:06.991095 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.851046    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 25 11:17:06.991095 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.851049    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 25 11:17:06.991095 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.851052    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 25 11:17:06.991095 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.851055    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 25 11:17:06.991095 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.851058    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 25 11:17:06.991095 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.851060    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 25 11:17:06.991095 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.851064    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 25 11:17:06.991095 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.851067    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 25 11:17:06.991095 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.851069    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 25 11:17:06.991095 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.851072    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 25 11:17:06.991095 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.851075    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 25 11:17:06.991095 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.851077    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 25 11:17:06.991095 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.851080    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 25 11:17:06.991095 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.851082    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 25 11:17:06.991095 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.851085    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 25 11:17:06.991767 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.851088    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 25 11:17:06.991767 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.851090    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 25 11:17:06.991767 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.851094    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 25 11:17:06.991767 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.851096    2561 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 25 11:17:06.991767 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.851099    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 25 11:17:06.991767 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.851102    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 25 11:17:06.991767 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.851104    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 25 11:17:06.991767 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.851107    2561 feature_gate.go:328] unrecognized feature gate: Example2
May 25 11:17:06.991767 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.851110    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 25 11:17:06.991767 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.851112    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 25 11:17:06.991767 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.851115    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 25 11:17:06.991767 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.851117    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 25 11:17:06.991767 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.851121    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 25 11:17:06.991767 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.851124    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 25 11:17:06.991767 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.851126    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 25 11:17:06.991767 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.851129    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 25 11:17:06.991767 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.851132    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 25 11:17:06.991767 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.851136    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 25 11:17:06.992214 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.851882    2561 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]}
May 25 11:17:06.992214 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.861637    2561 server.go:530] "Kubelet version" kubeletVersion="v1.33.11"
May 25 11:17:06.992214 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.861653    2561 server.go:532] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
May 25 11:17:06.992214 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.861702    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 25 11:17:06.992214 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.861706    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 25 11:17:06.992214 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.861710    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 25 11:17:06.992214 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.861713    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 25 11:17:06.992214 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.861716    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 25 11:17:06.992214 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.861718    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 25 11:17:06.992214 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.861722    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 25 11:17:06.992214 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.861725    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
May 25 11:17:06.992214 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.861728    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 25 11:17:06.992214 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.861731    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 25 11:17:06.992214 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.861734    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 25 11:17:06.992214 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.861737    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 25 11:17:06.992214 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.861739    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 25 11:17:06.992797 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.861742    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 25 11:17:06.992797 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.861745    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 25 11:17:06.992797 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.861747    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 25 11:17:06.992797 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.861750    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 25 11:17:06.992797 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.861752    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 25 11:17:06.992797 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.861755    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 25 11:17:06.992797 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.861758    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 25 11:17:06.992797 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.861760    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 25 11:17:06.992797 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.861763    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 25 11:17:06.992797 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.861769    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 25 11:17:06.992797 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.861772    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 25 11:17:06.992797 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.861775    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 25 11:17:06.992797 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.861777    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 25 11:17:06.992797 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.861780    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 25 11:17:06.992797 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.861783    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 25 11:17:06.992797 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.861785    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 25 11:17:06.992797 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.861788    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 25 11:17:06.992797 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.861791    2561 feature_gate.go:328] unrecognized feature gate: Example2
May 25 11:17:06.992797 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.861794    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 25 11:17:06.992797 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.861797    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 25 11:17:06.993590 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.861801    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 25 11:17:06.993590 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.861804    2561 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 25 11:17:06.993590 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.861807    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 25 11:17:06.993590 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.861810    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 25 11:17:06.993590 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.861812    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 25 11:17:06.993590 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.861815    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 25 11:17:06.993590 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.861818    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 25 11:17:06.993590 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.861821    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 25 11:17:06.993590 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.861823    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 25 11:17:06.993590 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.861826    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 25 11:17:06.993590 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.861829    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 25 11:17:06.993590 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.861831    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 25 11:17:06.993590 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.861834    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 25 11:17:06.993590 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.861837    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 25 11:17:06.993590 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.861839    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 25 11:17:06.993590 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.861842    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 25 11:17:06.993590 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.861845    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 25 11:17:06.993590 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.861848    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 25 11:17:06.993590 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.861850    2561 feature_gate.go:328] unrecognized feature gate: Example
May 25 11:17:06.993590 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.861853    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 25 11:17:06.994155 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.861855    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 25 11:17:06.994155 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.861858    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 25 11:17:06.994155 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.861861    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 25 11:17:06.994155 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.861864    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 25 11:17:06.994155 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.861867    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 25 11:17:06.994155 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.861870    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
May 25 11:17:06.994155 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.861874    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 25 11:17:06.994155 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.861878    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 25 11:17:06.994155 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.861883    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 25 11:17:06.994155 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.861886    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 25 11:17:06.994155 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.861889    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 25 11:17:06.994155 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.861891    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 25 11:17:06.994155 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.861894    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 25 11:17:06.994155 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.861897    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 25 11:17:06.994155 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.861899    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 25 11:17:06.994155 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.861902    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 25 11:17:06.994155 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.861904    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 25 11:17:06.994155 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.861907    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 25 11:17:06.994155 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.861910    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 25 11:17:06.994155 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.861912    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 25 11:17:06.994727 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.861915    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 25 11:17:06.994727 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.861918    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 25 11:17:06.994727 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.861920    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 25 11:17:06.994727 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.861925    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 25 11:17:06.994727 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.861928    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 25 11:17:06.994727 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.861931    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 25 11:17:06.994727 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.861933    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 25 11:17:06.994727 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.861936    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 25 11:17:06.994727 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.861939    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 25 11:17:06.994727 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.861942    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 25 11:17:06.994727 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.861948    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 25 11:17:06.994727 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.861950    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 25 11:17:06.994727 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.861953    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 25 11:17:06.994727 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.861958    2561 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]}
May 25 11:17:06.994727 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.862050    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 25 11:17:06.995110 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.862055    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 25 11:17:06.995110 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.862058    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 25 11:17:06.995110 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.862061    2561 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 25 11:17:06.995110 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.862065    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 25 11:17:06.995110 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.862069    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 25 11:17:06.995110 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.862072    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 25 11:17:06.995110 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.862075    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
May 25 11:17:06.995110 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.862078    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 25 11:17:06.995110 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.862080    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 25 11:17:06.995110 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.862083    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 25 11:17:06.995110 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.862085    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 25 11:17:06.995110 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.862088    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 25 11:17:06.995110 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.862091    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 25 11:17:06.995110 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.862093    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 25 11:17:06.995110 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.862096    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 25 11:17:06.995110 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.862099    2561 feature_gate.go:328] unrecognized feature gate: Example2
May 25 11:17:06.995110 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.862101    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 25 11:17:06.995110 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.862105    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 25 11:17:06.995110 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.862109    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 25 11:17:06.995595 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.862111    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 25 11:17:06.995595 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.862115    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 25 11:17:06.995595 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.862118    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 25 11:17:06.995595 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.862122    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 25 11:17:06.995595 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.862124    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 25 11:17:06.995595 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.862127    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 25 11:17:06.995595 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.862131    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 25 11:17:06.995595 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.862134    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 25 11:17:06.995595 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.862138    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 25 11:17:06.995595 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.862141    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 25 11:17:06.995595 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.862144    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 25 11:17:06.995595 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.862147    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 25 11:17:06.995595 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.862150    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 25 11:17:06.995595 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.862152    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 25 11:17:06.995595 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.862155    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 25 11:17:06.995595 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.862157    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 25 11:17:06.995595 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.862160    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 25 11:17:06.995595 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.862165    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 25 11:17:06.995595 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.862167    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 25 11:17:06.995595 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.862170    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 25 11:17:06.996086 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.862172    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 25 11:17:06.996086 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.862175    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 25 11:17:06.996086 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.862178    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 25 11:17:06.996086 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.862180    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 25 11:17:06.996086 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.862183    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 25 11:17:06.996086 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.862186    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 25 11:17:06.996086 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.862190    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 25 11:17:06.996086 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.862194    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 25 11:17:06.996086 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.862196    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 25 11:17:06.996086 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.862199    2561 feature_gate.go:328] unrecognized feature gate: Example
May 25 11:17:06.996086 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.862202    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 25 11:17:06.996086 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.862204    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 25 11:17:06.996086 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.862207    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 25 11:17:06.996086 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.862210    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 25 11:17:06.996086 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.862212    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 25 11:17:06.996086 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.862216    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 25 11:17:06.996086 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.862218    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 25 11:17:06.996086 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.862221    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 25 11:17:06.996086 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.862224    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 25 11:17:06.996086 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.862226    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 25 11:17:06.996658 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.862230    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 25 11:17:06.996658 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.862232    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 25 11:17:06.996658 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.862235    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 25 11:17:06.996658 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.862238    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 25 11:17:06.996658 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.862240    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
May 25 11:17:06.996658 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.862243    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 25 11:17:06.996658 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.862253    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 25 11:17:06.996658 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.862256    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 25 11:17:06.996658 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.862259    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 25 11:17:06.996658 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.862263    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 25 11:17:06.996658 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.862282    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 25 11:17:06.996658 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.862286    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 25 11:17:06.996658 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.862289    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 25 11:17:06.996658 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.862291    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 25 11:17:06.996658 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.862294    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 25 11:17:06.996658 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.862297    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 25 11:17:06.996658 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.862299    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 25 11:17:06.996658 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.862302    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 25 11:17:06.996658 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.862305    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 25 11:17:06.996658 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.862307    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 25 11:17:06.997129 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.862310    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 25 11:17:06.997129 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.862312    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 25 11:17:06.997129 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.862315    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 25 11:17:06.997129 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.862318    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 25 11:17:06.997129 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.862321    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 25 11:17:06.997129 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:06.862323    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 25 11:17:06.997129 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.862328    2561 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]}
May 25 11:17:06.997129 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.863109    2561 server.go:962] "Client rotation is on, will bootstrap in background"
May 25 11:17:06.997129 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.867881    2561 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
May 25 11:17:06.997129 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.868924    2561 server.go:1019] "Starting client certificate rotation"
May 25 11:17:06.997129 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.869015    2561 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
May 25 11:17:06.997129 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.869045    2561 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
May 25 11:17:06.997129 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.896602    2561 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
May 25 11:17:06.997129 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.899840    2561 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
May 25 11:17:06.997500 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.913502    2561 log.go:25] "Validated CRI v1 runtime API"
May 25 11:17:06.997500 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.921192    2561 log.go:25] "Validated CRI v1 image API"
May 25 11:17:06.997500 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.922566    2561 server.go:1452] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
May 25 11:17:06.997500 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.925911    2561 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 b6330c71-f92d-4260-859c-7494512286e1:/dev/nvme0n1p4 f6e907ca-4d28-48d9-9758-814b592a958a:/dev/nvme0n1p3]
May 25 11:17:06.997500 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.925928    2561 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
May 25 11:17:06.997500 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.932091    2561 reflector.go:430] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
May 25 11:17:06.997659 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.933988    2561 manager.go:217] Machine: {Timestamp:2026-05-25 11:17:06.933514045 +0000 UTC m=+0.478691639 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3146394 MemoryCapacity:33164488704 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec24303feab17fe7e8d4f61f27b13dde SystemUUID:ec24303f-eab1-7fe7-e8d4-f61f27b13dde BootID:7852e529-8788-4fa0-a7b4-e97358965ee9 Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6103040 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582242304 Type:vfs Inodes:4048399 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:98:ce:93:6d:17 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:98:ce:93:6d:17 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:76:08:41:57:ff:b3 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164488704 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
May 25 11:17:06.997659 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.934090    2561 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
May 25 11:17:06.997659 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.934191    2561 manager.go:233] Version: {KernelVersion:5.14.0-570.114.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260512-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
May 25 11:17:06.997659 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.938956    2561 container_manager_linux.go:270] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
May 25 11:17:06.997659 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.938986    2561 container_manager_linux.go:275] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-136-227.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
May 25 11:17:06.997659 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.939126    2561 topology_manager.go:138] "Creating topology manager with none policy"
May 25 11:17:06.997659 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.939134    2561 container_manager_linux.go:306] "Creating device plugin manager"
May 25 11:17:06.997659 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.939141    2561 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
May 25 11:17:06.997659 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.939986    2561 server.go:72] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
May 25 11:17:06.997659 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.942336    2561 state_mem.go:36] "Initialized new in-memory state store"
May 25 11:17:06.997659 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.942441    2561 server.go:1267] "Using root directory" path="/var/lib/kubelet"
May 25 11:17:06.997659 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.945631    2561 kubelet.go:491] "Attempting to sync node with API server"
May 25 11:17:06.997659 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.945642    2561 kubelet.go:386] "Adding static pod path" path="/etc/kubernetes/manifests"
May 25 11:17:06.997659 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.945653    2561 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
May 25 11:17:06.997659 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.945661    2561 kubelet.go:397] "Adding apiserver pod source"
May 25 11:17:06.997659 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.945669    2561 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
May 25 11:17:06.997659 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.946838    2561 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
May 25 11:17:06.997659 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.946853    2561 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
May 25 11:17:06.998136 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.950210    2561 kuberuntime_manager.go:279] "Container runtime initialized" containerRuntime="cri-o" version="1.33.11-2.rhaos4.20.gitb2a8320.el9" apiVersion="v1"
May 25 11:17:06.998136 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.952372    2561 kubelet.go:953] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
May 25 11:17:06.998136 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.953306    2561 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
May 25 11:17:06.998136 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.953322    2561 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
May 25 11:17:06.998136 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.953328    2561 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
May 25 11:17:06.998136 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.953335    2561 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
May 25 11:17:06.998136 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.953344    2561 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
May 25 11:17:06.998136 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.953349    2561 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/secret"
May 25 11:17:06.998136 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.953355    2561 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
May 25 11:17:06.998136 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.953361    2561 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
May 25 11:17:06.998136 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.953369    2561 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/fc"
May 25 11:17:06.998136 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.953376    2561 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
May 25 11:17:06.998136 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.953384    2561 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/projected"
May 25 11:17:06.998136 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.953819    2561 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
May 25 11:17:06.998136 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.953844    2561 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/csi"
May 25 11:17:06.998136 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.953853    2561 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/image"
May 25 11:17:06.998136 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.956842    2561 watchdog_linux.go:99] "Systemd watchdog is not enabled"
May 25 11:17:06.998136 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.956877    2561 server.go:1295] "Started kubelet"
May 25 11:17:06.998136 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.956929    2561 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
May 25 11:17:06.998136 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.956994    2561 ratelimit.go:55] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
May 25 11:17:06.998136 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.957045    2561 server_v1.go:47] "podresources" method="list" useActivePods=true
May 25 11:17:06.998136 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.958542    2561 server.go:255] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
May 25 11:17:06.998136 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.959053    2561 server.go:317] "Adding debug handlers to kubelet server"
May 25 11:17:06.998136 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:06.961235    2561 reflector.go:200] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
May 25 11:17:06.998136 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.961255    2561 csi_plugin.go:988] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-136-227.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
May 25 11:17:06.998136 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:06.961298    2561 reflector.go:200] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-136-227.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
May 25 11:17:06.998136 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:06.961345    2561 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-227.ec2.internal.18b2cad6ddbed538  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-227.ec2.internal,UID:ip-10-0-136-227.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-136-227.ec2.internal,},FirstTimestamp:2026-05-25 11:17:06.956850488 +0000 UTC m=+0.502028081,LastTimestamp:2026-05-25 11:17:06.956850488 +0000 UTC m=+0.502028081,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-227.ec2.internal,}"
May 25 11:17:06.998136 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:06.967005    2561 kubelet.go:1618] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
May 25 11:17:06.998136 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.967153    2561 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
May 25 11:17:06.998136 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.967748    2561 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
May 25 11:17:06.998136 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.967885    2561 volume_manager.go:295] "The desired_state_of_world populator starts"
May 25 11:17:06.998136 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.967899    2561 volume_manager.go:297] "Starting Kubelet Volume Manager"
May 25 11:17:06.998136 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.967968    2561 desired_state_of_world_populator.go:150] "Desired state populator starts to run"
May 25 11:17:06.998136 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.968024    2561 reconstruct.go:97] "Volume reconstruction finished"
May 25 11:17:06.998136 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.968034    2561 reconciler.go:26] "Reconciler: start to sync state"
May 25 11:17:06.998136 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.968568    2561 factory.go:153] Registering CRI-O factory
May 25 11:17:06.998136 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:06.968592    2561 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-136-227.ec2.internal\" not found"
May 25 11:17:06.998136 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.968605    2561 factory.go:223] Registration of the crio container factory successfully
May 25 11:17:06.998136 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.968667    2561 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
May 25 11:17:06.998136 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.968679    2561 factory.go:55] Registering systemd factory
May 25 11:17:06.998136 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.968686    2561 factory.go:223] Registration of the systemd container factory successfully
May 25 11:17:06.999183 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.968702    2561 factory.go:103] Registering Raw factory
May 25 11:17:06.999183 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.968714    2561 manager.go:1196] Started watching for new ooms in manager
May 25 11:17:06.999183 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.969149    2561 manager.go:319] Starting recovery of all containers
May 25 11:17:06.999183 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:06.970222    2561 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-136-227.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
May 25 11:17:06.999183 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:06.970392    2561 reflector.go:200] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
May 25 11:17:06.999183 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.981637    2561 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-qhfhs"
May 25 11:17:06.999183 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.984959    2561 manager.go:324] Recovery completed
May 25 11:17:06.999183 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.989023    2561 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 25 11:17:06.999183 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.989138    2561 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-qhfhs"
May 25 11:17:06.999183 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.990750    2561 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-136-227.ec2.internal" event="NodeHasSufficientMemory"
May 25 11:17:06.999183 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.990780    2561 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-136-227.ec2.internal" event="NodeHasNoDiskPressure"
May 25 11:17:06.999183 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.990793    2561 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-136-227.ec2.internal" event="NodeHasSufficientPID"
May 25 11:17:06.999183 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.991529    2561 cpu_manager.go:222] "Starting CPU manager" policy="none"
May 25 11:17:06.999183 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.991542    2561 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
May 25 11:17:06.999183 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.991559    2561 state_mem.go:36] "Initialized new in-memory state store"
May 25 11:17:06.999183 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.994262    2561 policy_none.go:49] "None policy: Start"
May 25 11:17:06.999183 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.994435    2561 memory_manager.go:186] "Starting memorymanager" policy="None"
May 25 11:17:06.999183 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:06.994446    2561 state_mem.go:35] "Initializing new in-memory state store"
May 25 11:17:07.047864 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:07.042814    2561 manager.go:341] "Starting Device Plugin manager"
May 25 11:17:07.047864 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:07.042843    2561 manager.go:517] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
May 25 11:17:07.047864 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:07.042852    2561 server.go:85] "Starting device plugin registration server"
May 25 11:17:07.047864 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:07.043020    2561 eviction_manager.go:189] "Eviction manager: starting control loop"
May 25 11:17:07.047864 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:07.043031    2561 container_log_manager.go:189] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
May 25 11:17:07.047864 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:07.043153    2561 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
May 25 11:17:07.047864 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:07.043212    2561 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
May 25 11:17:07.047864 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:07.043219    2561 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
May 25 11:17:07.047864 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:07.043700    2561 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
May 25 11:17:07.047864 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:07.043730    2561 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-136-227.ec2.internal\" not found"
May 25 11:17:07.110858 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:07.110823    2561 kubelet_network_linux.go:49] "Initialized iptables rules." protocol="IPv4"
May 25 11:17:07.112039 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:07.112022    2561 kubelet_network_linux.go:49] "Initialized iptables rules." protocol="IPv6"
May 25 11:17:07.112095 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:07.112050    2561 status_manager.go:230] "Starting to sync pod status with apiserver"
May 25 11:17:07.112095 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:07.112064    2561 watchdog_linux.go:127] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
May 25 11:17:07.112095 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:07.112071    2561 kubelet.go:2451] "Starting kubelet main sync loop"
May 25 11:17:07.112235 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:07.112109    2561 kubelet.go:2475] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
May 25 11:17:07.118172 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:07.118152    2561 reflector.go:430] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
May 25 11:17:07.143244 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:07.143230    2561 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 25 11:17:07.144095 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:07.144078    2561 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-136-227.ec2.internal" event="NodeHasSufficientMemory"
May 25 11:17:07.144155 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:07.144109    2561 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-136-227.ec2.internal" event="NodeHasNoDiskPressure"
May 25 11:17:07.144155 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:07.144123    2561 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-136-227.ec2.internal" event="NodeHasSufficientPID"
May 25 11:17:07.144155 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:07.144144    2561 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-136-227.ec2.internal"
May 25 11:17:07.156275 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:07.156251    2561 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-136-227.ec2.internal"
May 25 11:17:07.156388 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:07.156280    2561 kubelet_node_status.go:597] "Error updating node status, will retry" err="error getting node \"ip-10-0-136-227.ec2.internal\": node \"ip-10-0-136-227.ec2.internal\" not found"
May 25 11:17:07.170115 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:07.170094    2561 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-136-227.ec2.internal\" not found"
May 25 11:17:07.212333 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:07.212303    2561 kubelet.go:2537] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-227.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-136-227.ec2.internal"]
May 25 11:17:07.212399 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:07.212379    2561 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 25 11:17:07.213747 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:07.213732    2561 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-136-227.ec2.internal" event="NodeHasSufficientMemory"
May 25 11:17:07.213791 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:07.213757    2561 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-136-227.ec2.internal" event="NodeHasNoDiskPressure"
May 25 11:17:07.213791 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:07.213767    2561 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-136-227.ec2.internal" event="NodeHasSufficientPID"
May 25 11:17:07.215872 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:07.215860    2561 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 25 11:17:07.216049 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:07.216037    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-227.ec2.internal"
May 25 11:17:07.216096 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:07.216063    2561 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 25 11:17:07.216501 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:07.216487    2561 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-136-227.ec2.internal" event="NodeHasSufficientMemory"
May 25 11:17:07.216566 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:07.216515    2561 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-136-227.ec2.internal" event="NodeHasNoDiskPressure"
May 25 11:17:07.216566 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:07.216557    2561 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-136-227.ec2.internal" event="NodeHasSufficientPID"
May 25 11:17:07.216566 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:07.216557    2561 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-136-227.ec2.internal" event="NodeHasSufficientMemory"
May 25 11:17:07.216707 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:07.216581    2561 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-136-227.ec2.internal" event="NodeHasNoDiskPressure"
May 25 11:17:07.216707 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:07.216595    2561 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-136-227.ec2.internal" event="NodeHasSufficientPID"
May 25 11:17:07.218710 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:07.218694    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-227.ec2.internal"
May 25 11:17:07.218792 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:07.218720    2561 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 25 11:17:07.219318 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:07.219303    2561 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-136-227.ec2.internal" event="NodeHasSufficientMemory"
May 25 11:17:07.219388 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:07.219328    2561 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-136-227.ec2.internal" event="NodeHasNoDiskPressure"
May 25 11:17:07.219388 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:07.219337    2561 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-136-227.ec2.internal" event="NodeHasSufficientPID"
May 25 11:17:07.243798 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:07.243755    2561 kubelet.go:3336] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-227.ec2.internal\" not found" node="ip-10-0-136-227.ec2.internal"
May 25 11:17:07.247928 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:07.247914    2561 kubelet.go:3336] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-227.ec2.internal\" not found" node="ip-10-0-136-227.ec2.internal"
May 25 11:17:07.269881 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:07.269858    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f96ecd3e8e97719fd70e0de354f8eb4d-config\") pod \"kube-apiserver-proxy-ip-10-0-136-227.ec2.internal\" (UID: \"f96ecd3e8e97719fd70e0de354f8eb4d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-227.ec2.internal"
May 25 11:17:07.269968 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:07.269886    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/17c74a3057617237e14ccfbc812fc82f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-227.ec2.internal\" (UID: \"17c74a3057617237e14ccfbc812fc82f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-227.ec2.internal"
May 25 11:17:07.269968 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:07.269911    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/17c74a3057617237e14ccfbc812fc82f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-227.ec2.internal\" (UID: \"17c74a3057617237e14ccfbc812fc82f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-227.ec2.internal"
May 25 11:17:07.271050 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:07.271037    2561 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-136-227.ec2.internal\" not found"
May 25 11:17:07.370632 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:07.370613    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f96ecd3e8e97719fd70e0de354f8eb4d-config\") pod \"kube-apiserver-proxy-ip-10-0-136-227.ec2.internal\" (UID: \"f96ecd3e8e97719fd70e0de354f8eb4d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-227.ec2.internal"
May 25 11:17:07.370706 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:07.370635    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/17c74a3057617237e14ccfbc812fc82f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-227.ec2.internal\" (UID: \"17c74a3057617237e14ccfbc812fc82f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-227.ec2.internal"
May 25 11:17:07.370706 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:07.370654    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/17c74a3057617237e14ccfbc812fc82f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-227.ec2.internal\" (UID: \"17c74a3057617237e14ccfbc812fc82f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-227.ec2.internal"
May 25 11:17:07.370798 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:07.370716    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/17c74a3057617237e14ccfbc812fc82f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-227.ec2.internal\" (UID: \"17c74a3057617237e14ccfbc812fc82f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-227.ec2.internal"
May 25 11:17:07.370798 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:07.370767    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f96ecd3e8e97719fd70e0de354f8eb4d-config\") pod \"kube-apiserver-proxy-ip-10-0-136-227.ec2.internal\" (UID: \"f96ecd3e8e97719fd70e0de354f8eb4d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-227.ec2.internal"
May 25 11:17:07.370798 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:07.370791    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/17c74a3057617237e14ccfbc812fc82f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-227.ec2.internal\" (UID: \"17c74a3057617237e14ccfbc812fc82f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-227.ec2.internal"
May 25 11:17:07.371656 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:07.371642    2561 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-136-227.ec2.internal\" not found"
May 25 11:17:07.472312 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:07.472292    2561 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-136-227.ec2.internal\" not found"
May 25 11:17:07.546531 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:07.546473    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-227.ec2.internal"
May 25 11:17:07.550182 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:07.550166    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-227.ec2.internal"
May 25 11:17:07.573012 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:07.572994    2561 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-136-227.ec2.internal\" not found"
May 25 11:17:07.673464 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:07.673429    2561 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-136-227.ec2.internal\" not found"
May 25 11:17:07.773843 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:07.773820    2561 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-136-227.ec2.internal\" not found"
May 25 11:17:07.870151 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:07.870089    2561 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
May 25 11:17:07.870235 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:07.870217    2561 reflector.go:556] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
May 25 11:17:07.873964 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:07.873948    2561 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-136-227.ec2.internal\" not found"
May 25 11:17:07.887746 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:07.887729    2561 reflector.go:430] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
May 25 11:17:07.967309 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:07.967283    2561 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
May 25 11:17:07.974829 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:07.974810    2561 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-136-227.ec2.internal\" not found"
May 25 11:17:07.977825 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:07.977802    2561 reflector.go:430] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
May 25 11:17:07.991819 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:07.991784    2561 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-05-24 11:12:06 +0000 UTC" deadline="2027-10-28 07:14:41.36207964 +0000 UTC"
May 25 11:17:07.991915 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:07.991832    2561 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12499h57m33.370263694s"
May 25 11:17:08.075421 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:08.075385    2561 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-136-227.ec2.internal\" not found"
May 25 11:17:08.096057 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:08.096024    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17c74a3057617237e14ccfbc812fc82f.slice/crio-aefb5aa416c6e40ae363a082e81431d8f1bbd840074a34c9f4d8ed5ec971726e WatchSource:0}: Error finding container aefb5aa416c6e40ae363a082e81431d8f1bbd840074a34c9f4d8ed5ec971726e: Status 404 returned error can't find the container with id aefb5aa416c6e40ae363a082e81431d8f1bbd840074a34c9f4d8ed5ec971726e
May 25 11:17:08.096218 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:08.096203    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf96ecd3e8e97719fd70e0de354f8eb4d.slice/crio-d9648f2cba6178faddfd9f740cd4c7cfab4b55f99dab6c22058571666126c02f WatchSource:0}: Error finding container d9648f2cba6178faddfd9f740cd4c7cfab4b55f99dab6c22058571666126c02f: Status 404 returned error can't find the container with id d9648f2cba6178faddfd9f740cd4c7cfab4b55f99dab6c22058571666126c02f
May 25 11:17:08.100021 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:08.099998    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 25 11:17:08.105260 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:08.105240    2561 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-tths5"
May 25 11:17:08.113582 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:08.113563    2561 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-tths5"
May 25 11:17:08.115362 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:08.115325    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-227.ec2.internal" event={"ID":"17c74a3057617237e14ccfbc812fc82f","Type":"ContainerStarted","Data":"aefb5aa416c6e40ae363a082e81431d8f1bbd840074a34c9f4d8ed5ec971726e"}
May 25 11:17:08.116089 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:08.116072    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-227.ec2.internal" event={"ID":"f96ecd3e8e97719fd70e0de354f8eb4d","Type":"ContainerStarted","Data":"d9648f2cba6178faddfd9f740cd4c7cfab4b55f99dab6c22058571666126c02f"}
May 25 11:17:08.176469 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:08.176442    2561 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-136-227.ec2.internal\" not found"
May 25 11:17:08.276878 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:08.276835    2561 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-136-227.ec2.internal\" not found"
May 25 11:17:08.377263 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:08.377235    2561 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-136-227.ec2.internal\" not found"
May 25 11:17:08.458508 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:08.458438    2561 reflector.go:430] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
May 25 11:17:08.468046 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:08.468023    2561 kubelet.go:3340] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-227.ec2.internal"
May 25 11:17:08.469456 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:08.469440    2561 reflector.go:430] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
May 25 11:17:08.482066 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:08.482039    2561 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
May 25 11:17:08.483002 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:08.482984    2561 kubelet.go:3340] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-227.ec2.internal"
May 25 11:17:08.492504 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:08.492477    2561 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
May 25 11:17:08.920080 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:08.920032    2561 reflector.go:430] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
May 25 11:17:08.946220 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:08.946193    2561 apiserver.go:52] "Watching apiserver"
May 25 11:17:08.954154 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:08.954130    2561 reflector.go:430] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
May 25 11:17:08.954559 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:08.954531    2561 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-target-rl7hd","openshift-network-operator/iptables-alerter-4xz76","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5ldrp","openshift-cluster-node-tuning-operator/tuned-k2fsn","openshift-image-registry/node-ca-q8ng6","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-227.ec2.internal","openshift-ovn-kubernetes/ovnkube-node-thvg5","kube-system/konnectivity-agent-mr48p","kube-system/kube-apiserver-proxy-ip-10-0-136-227.ec2.internal","openshift-dns/node-resolver-xvcpl","openshift-multus/multus-additional-cni-plugins-dmzql","openshift-multus/multus-bxzz6","openshift-multus/network-metrics-daemon-444mb"]
May 25 11:17:08.957887 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:08.957863    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-thvg5"
May 25 11:17:08.960057 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:08.960032    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-4xz76"
May 25 11:17:08.960355 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:08.960333    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
May 25 11:17:08.960540 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:08.960523    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
May 25 11:17:08.960851 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:08.960827    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
May 25 11:17:08.961174 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:08.961033    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
May 25 11:17:08.961174 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:08.961101    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-4vbk9\""
May 25 11:17:08.961342 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:08.961288    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
May 25 11:17:08.961557 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:08.961541    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
May 25 11:17:08.962525 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:08.962316    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
May 25 11:17:08.962629 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:08.962528    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
May 25 11:17:08.962783 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:08.962765    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-qlvgr\""
May 25 11:17:08.963177 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:08.963160    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
May 25 11:17:08.967460 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:08.967410    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-k2fsn"
May 25 11:17:08.967460 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:08.967437    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5ldrp"
May 25 11:17:08.970009 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:08.969849    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
May 25 11:17:08.970157 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:08.970135    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
May 25 11:17:08.970241 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:08.970224    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-4f9nv\""
May 25 11:17:08.970500 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:08.970478    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-bql8n\""
May 25 11:17:08.970623 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:08.970502    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
May 25 11:17:08.970735 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:08.970715    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
May 25 11:17:08.970831 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:08.970745    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
May 25 11:17:08.972414 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:08.972395    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-q8ng6"
May 25 11:17:08.974560 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:08.974430    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
May 25 11:17:08.974721 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:08.974643    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rl7hd"
May 25 11:17:08.974721 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:08.974699    2561 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rl7hd" podUID="2b05b78f-6ab1-4f94-8273-e5fa70b78d15"
May 25 11:17:08.974840 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:08.974762    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-mr48p"
May 25 11:17:08.975319 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:08.975175    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
May 25 11:17:08.975319 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:08.975188    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
May 25 11:17:08.975319 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:08.975205    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-plq52\""
May 25 11:17:08.976850 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:08.976830    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-lkmss\""
May 25 11:17:08.976951 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:08.976904    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
May 25 11:17:08.977012 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:08.976957    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-xvcpl"
May 25 11:17:08.977120 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:08.977104    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
May 25 11:17:08.978759 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:08.978732    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b0e8a1a4-c0cb-443e-8c38-9abcf242fc84-sys\") pod \"tuned-k2fsn\" (UID: \"b0e8a1a4-c0cb-443e-8c38-9abcf242fc84\") " pod="openshift-cluster-node-tuning-operator/tuned-k2fsn"
May 25 11:17:08.979427 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:08.978770    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/5b2bf5ae-a8a4-48e9-9afc-ee576988791e-host-kubelet\") pod \"ovnkube-node-thvg5\" (UID: \"5b2bf5ae-a8a4-48e9-9afc-ee576988791e\") " pod="openshift-ovn-kubernetes/ovnkube-node-thvg5"
May 25 11:17:08.979427 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:08.978799    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5b2bf5ae-a8a4-48e9-9afc-ee576988791e-host-run-netns\") pod \"ovnkube-node-thvg5\" (UID: \"5b2bf5ae-a8a4-48e9-9afc-ee576988791e\") " pod="openshift-ovn-kubernetes/ovnkube-node-thvg5"
May 25 11:17:08.979427 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:08.978824    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/5b2bf5ae-a8a4-48e9-9afc-ee576988791e-log-socket\") pod \"ovnkube-node-thvg5\" (UID: \"5b2bf5ae-a8a4-48e9-9afc-ee576988791e\") " pod="openshift-ovn-kubernetes/ovnkube-node-thvg5"
May 25 11:17:08.979427 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:08.978845    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/5b2bf5ae-a8a4-48e9-9afc-ee576988791e-host-cni-netd\") pod \"ovnkube-node-thvg5\" (UID: \"5b2bf5ae-a8a4-48e9-9afc-ee576988791e\") " pod="openshift-ovn-kubernetes/ovnkube-node-thvg5"
May 25 11:17:08.979427 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:08.978860    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/71885334-70ca-483b-8bd9-fdbfecb77949-kubelet-dir\") pod \"aws-ebs-csi-driver-node-5ldrp\" (UID: \"71885334-70ca-483b-8bd9-fdbfecb77949\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5ldrp"
May 25 11:17:08.979427 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:08.978905    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/71885334-70ca-483b-8bd9-fdbfecb77949-device-dir\") pod \"aws-ebs-csi-driver-node-5ldrp\" (UID: \"71885334-70ca-483b-8bd9-fdbfecb77949\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5ldrp"
May 25 11:17:08.979427 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:08.978930    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/5b2bf5ae-a8a4-48e9-9afc-ee576988791e-run-ovn\") pod \"ovnkube-node-thvg5\" (UID: \"5b2bf5ae-a8a4-48e9-9afc-ee576988791e\") " pod="openshift-ovn-kubernetes/ovnkube-node-thvg5"
May 25 11:17:08.979427 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:08.978951    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5b2bf5ae-a8a4-48e9-9afc-ee576988791e-host-cni-bin\") pod \"ovnkube-node-thvg5\" (UID: \"5b2bf5ae-a8a4-48e9-9afc-ee576988791e\") " pod="openshift-ovn-kubernetes/ovnkube-node-thvg5"
May 25 11:17:08.979427 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:08.978974    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5b2bf5ae-a8a4-48e9-9afc-ee576988791e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-thvg5\" (UID: \"5b2bf5ae-a8a4-48e9-9afc-ee576988791e\") " pod="openshift-ovn-kubernetes/ovnkube-node-thvg5"
May 25 11:17:08.979427 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:08.979004    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a8a3a45a-3dba-4919-895a-ef467fbd4343-host-slash\") pod \"iptables-alerter-4xz76\" (UID: \"a8a3a45a-3dba-4919-895a-ef467fbd4343\") " pod="openshift-network-operator/iptables-alerter-4xz76"
May 25 11:17:08.979427 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:08.979029    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/71885334-70ca-483b-8bd9-fdbfecb77949-sys-fs\") pod \"aws-ebs-csi-driver-node-5ldrp\" (UID: \"71885334-70ca-483b-8bd9-fdbfecb77949\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5ldrp"
May 25 11:17:08.979427 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:08.979053    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b0e8a1a4-c0cb-443e-8c38-9abcf242fc84-host\") pod \"tuned-k2fsn\" (UID: \"b0e8a1a4-c0cb-443e-8c38-9abcf242fc84\") " pod="openshift-cluster-node-tuning-operator/tuned-k2fsn"
May 25 11:17:08.979427 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:08.979091    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9a250955-746b-416a-b798-283a0ea46864-serviceca\") pod \"node-ca-q8ng6\" (UID: \"9a250955-746b-416a-b798-283a0ea46864\") " pod="openshift-image-registry/node-ca-q8ng6"
May 25 11:17:08.979427 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:08.979146    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d68xg\" (UniqueName: \"kubernetes.io/projected/71885334-70ca-483b-8bd9-fdbfecb77949-kube-api-access-d68xg\") pod \"aws-ebs-csi-driver-node-5ldrp\" (UID: \"71885334-70ca-483b-8bd9-fdbfecb77949\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5ldrp"
May 25 11:17:08.979427 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:08.979177    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5b2bf5ae-a8a4-48e9-9afc-ee576988791e-host-slash\") pod \"ovnkube-node-thvg5\" (UID: \"5b2bf5ae-a8a4-48e9-9afc-ee576988791e\") " pod="openshift-ovn-kubernetes/ovnkube-node-thvg5"
May 25 11:17:08.979427 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:08.979185    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
May 25 11:17:08.979427 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:08.979209    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/5b2bf5ae-a8a4-48e9-9afc-ee576988791e-run-systemd\") pod \"ovnkube-node-thvg5\" (UID: \"5b2bf5ae-a8a4-48e9-9afc-ee576988791e\") " pod="openshift-ovn-kubernetes/ovnkube-node-thvg5"
May 25 11:17:08.980218 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:08.979233    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5b2bf5ae-a8a4-48e9-9afc-ee576988791e-etc-openvswitch\") pod \"ovnkube-node-thvg5\" (UID: \"5b2bf5ae-a8a4-48e9-9afc-ee576988791e\") " pod="openshift-ovn-kubernetes/ovnkube-node-thvg5"
May 25 11:17:08.980218 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:08.979255    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/5b2bf5ae-a8a4-48e9-9afc-ee576988791e-env-overrides\") pod \"ovnkube-node-thvg5\" (UID: \"5b2bf5ae-a8a4-48e9-9afc-ee576988791e\") " pod="openshift-ovn-kubernetes/ovnkube-node-thvg5"
May 25 11:17:08.980218 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:08.979307    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qj5l\" (UniqueName: \"kubernetes.io/projected/5b2bf5ae-a8a4-48e9-9afc-ee576988791e-kube-api-access-6qj5l\") pod \"ovnkube-node-thvg5\" (UID: \"5b2bf5ae-a8a4-48e9-9afc-ee576988791e\") " pod="openshift-ovn-kubernetes/ovnkube-node-thvg5"
May 25 11:17:08.980218 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:08.979334    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5nqq\" (UniqueName: \"kubernetes.io/projected/a8a3a45a-3dba-4919-895a-ef467fbd4343-kube-api-access-q5nqq\") pod \"iptables-alerter-4xz76\" (UID: \"a8a3a45a-3dba-4919-895a-ef467fbd4343\") " pod="openshift-network-operator/iptables-alerter-4xz76"
May 25 11:17:08.980218 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:08.979360    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/71885334-70ca-483b-8bd9-fdbfecb77949-socket-dir\") pod \"aws-ebs-csi-driver-node-5ldrp\" (UID: \"71885334-70ca-483b-8bd9-fdbfecb77949\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5ldrp"
May 25 11:17:08.980218 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:08.979384    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5b2bf5ae-a8a4-48e9-9afc-ee576988791e-var-lib-openvswitch\") pod \"ovnkube-node-thvg5\" (UID: \"5b2bf5ae-a8a4-48e9-9afc-ee576988791e\") " pod="openshift-ovn-kubernetes/ovnkube-node-thvg5"
May 25 11:17:08.980218 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:08.979406    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/5b2bf5ae-a8a4-48e9-9afc-ee576988791e-node-log\") pod \"ovnkube-node-thvg5\" (UID: \"5b2bf5ae-a8a4-48e9-9afc-ee576988791e\") " pod="openshift-ovn-kubernetes/ovnkube-node-thvg5"
May 25 11:17:08.980218 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:08.979428    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
May 25 11:17:08.980218 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:08.979428    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/71885334-70ca-483b-8bd9-fdbfecb77949-etc-selinux\") pod \"aws-ebs-csi-driver-node-5ldrp\" (UID: \"71885334-70ca-483b-8bd9-fdbfecb77949\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5ldrp"
May 25 11:17:08.980218 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:08.979565    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-bwcnv\""
May 25 11:17:08.980218 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:08.979576    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b0e8a1a4-c0cb-443e-8c38-9abcf242fc84-etc-systemd\") pod \"tuned-k2fsn\" (UID: \"b0e8a1a4-c0cb-443e-8c38-9abcf242fc84\") " pod="openshift-cluster-node-tuning-operator/tuned-k2fsn"
May 25 11:17:08.980218 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:08.979602    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b0e8a1a4-c0cb-443e-8c38-9abcf242fc84-run\") pod \"tuned-k2fsn\" (UID: \"b0e8a1a4-c0cb-443e-8c38-9abcf242fc84\") " pod="openshift-cluster-node-tuning-operator/tuned-k2fsn"
May 25 11:17:08.980218 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:08.979639    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b0e8a1a4-c0cb-443e-8c38-9abcf242fc84-lib-modules\") pod \"tuned-k2fsn\" (UID: \"b0e8a1a4-c0cb-443e-8c38-9abcf242fc84\") " pod="openshift-cluster-node-tuning-operator/tuned-k2fsn"
May 25 11:17:08.980218 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:08.979664    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b0e8a1a4-c0cb-443e-8c38-9abcf242fc84-etc-tuned\") pod \"tuned-k2fsn\" (UID: \"b0e8a1a4-c0cb-443e-8c38-9abcf242fc84\") " pod="openshift-cluster-node-tuning-operator/tuned-k2fsn"
May 25 11:17:08.980218 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:08.979704    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9a250955-746b-416a-b798-283a0ea46864-host\") pod \"node-ca-q8ng6\" (UID: \"9a250955-746b-416a-b798-283a0ea46864\") " pod="openshift-image-registry/node-ca-q8ng6"
May 25 11:17:08.980218 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:08.979727    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5b2bf5ae-a8a4-48e9-9afc-ee576988791e-host-run-ovn-kubernetes\") pod \"ovnkube-node-thvg5\" (UID: \"5b2bf5ae-a8a4-48e9-9afc-ee576988791e\") " pod="openshift-ovn-kubernetes/ovnkube-node-thvg5"
May 25 11:17:08.980218 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:08.979760    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/5b2bf5ae-a8a4-48e9-9afc-ee576988791e-ovnkube-script-lib\") pod \"ovnkube-node-thvg5\" (UID: \"5b2bf5ae-a8a4-48e9-9afc-ee576988791e\") " pod="openshift-ovn-kubernetes/ovnkube-node-thvg5"
May 25 11:17:08.981071 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:08.979790    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b0e8a1a4-c0cb-443e-8c38-9abcf242fc84-var-lib-kubelet\") pod \"tuned-k2fsn\" (UID: \"b0e8a1a4-c0cb-443e-8c38-9abcf242fc84\") " pod="openshift-cluster-node-tuning-operator/tuned-k2fsn"
May 25 11:17:08.981071 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:08.979833    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b0e8a1a4-c0cb-443e-8c38-9abcf242fc84-tmp\") pod \"tuned-k2fsn\" (UID: \"b0e8a1a4-c0cb-443e-8c38-9abcf242fc84\") " pod="openshift-cluster-node-tuning-operator/tuned-k2fsn"
May 25 11:17:08.981071 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:08.979871    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gg7wx\" (UniqueName: \"kubernetes.io/projected/b0e8a1a4-c0cb-443e-8c38-9abcf242fc84-kube-api-access-gg7wx\") pod \"tuned-k2fsn\" (UID: \"b0e8a1a4-c0cb-443e-8c38-9abcf242fc84\") " pod="openshift-cluster-node-tuning-operator/tuned-k2fsn"
May 25 11:17:08.981071 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:08.979924    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5b2bf5ae-a8a4-48e9-9afc-ee576988791e-run-openvswitch\") pod \"ovnkube-node-thvg5\" (UID: \"5b2bf5ae-a8a4-48e9-9afc-ee576988791e\") " pod="openshift-ovn-kubernetes/ovnkube-node-thvg5"
May 25 11:17:08.981071 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:08.979962    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/5b2bf5ae-a8a4-48e9-9afc-ee576988791e-ovnkube-config\") pod \"ovnkube-node-thvg5\" (UID: \"5b2bf5ae-a8a4-48e9-9afc-ee576988791e\") " pod="openshift-ovn-kubernetes/ovnkube-node-thvg5"
May 25 11:17:08.981071 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:08.980002    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b0e8a1a4-c0cb-443e-8c38-9abcf242fc84-etc-modprobe-d\") pod \"tuned-k2fsn\" (UID: \"b0e8a1a4-c0cb-443e-8c38-9abcf242fc84\") " pod="openshift-cluster-node-tuning-operator/tuned-k2fsn"
May 25 11:17:08.981071 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:08.980039    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b0e8a1a4-c0cb-443e-8c38-9abcf242fc84-etc-sysconfig\") pod \"tuned-k2fsn\" (UID: \"b0e8a1a4-c0cb-443e-8c38-9abcf242fc84\") " pod="openshift-cluster-node-tuning-operator/tuned-k2fsn"
May 25 11:17:08.981071 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:08.980070    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b0e8a1a4-c0cb-443e-8c38-9abcf242fc84-etc-sysctl-d\") pod \"tuned-k2fsn\" (UID: \"b0e8a1a4-c0cb-443e-8c38-9abcf242fc84\") " pod="openshift-cluster-node-tuning-operator/tuned-k2fsn"
May 25 11:17:08.981071 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:08.980111    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pq5fc\" (UniqueName: \"kubernetes.io/projected/9a250955-746b-416a-b798-283a0ea46864-kube-api-access-pq5fc\") pod \"node-ca-q8ng6\" (UID: \"9a250955-746b-416a-b798-283a0ea46864\") " pod="openshift-image-registry/node-ca-q8ng6"
May 25 11:17:08.981071 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:08.980140    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/5b2bf5ae-a8a4-48e9-9afc-ee576988791e-systemd-units\") pod \"ovnkube-node-thvg5\" (UID: \"5b2bf5ae-a8a4-48e9-9afc-ee576988791e\") " pod="openshift-ovn-kubernetes/ovnkube-node-thvg5"
May 25 11:17:08.981071 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:08.980178    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/5b2bf5ae-a8a4-48e9-9afc-ee576988791e-ovn-node-metrics-cert\") pod \"ovnkube-node-thvg5\" (UID: \"5b2bf5ae-a8a4-48e9-9afc-ee576988791e\") " pod="openshift-ovn-kubernetes/ovnkube-node-thvg5"
May 25 11:17:08.981071 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:08.980203    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a8a3a45a-3dba-4919-895a-ef467fbd4343-iptables-alerter-script\") pod \"iptables-alerter-4xz76\" (UID: \"a8a3a45a-3dba-4919-895a-ef467fbd4343\") " pod="openshift-network-operator/iptables-alerter-4xz76"
May 25 11:17:08.981071 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:08.980225    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/71885334-70ca-483b-8bd9-fdbfecb77949-registration-dir\") pod \"aws-ebs-csi-driver-node-5ldrp\" (UID: \"71885334-70ca-483b-8bd9-fdbfecb77949\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5ldrp"
May 25 11:17:08.981071 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:08.980246    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b0e8a1a4-c0cb-443e-8c38-9abcf242fc84-etc-kubernetes\") pod \"tuned-k2fsn\" (UID: \"b0e8a1a4-c0cb-443e-8c38-9abcf242fc84\") " pod="openshift-cluster-node-tuning-operator/tuned-k2fsn"
May 25 11:17:08.981071 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:08.980288    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b0e8a1a4-c0cb-443e-8c38-9abcf242fc84-etc-sysctl-conf\") pod \"tuned-k2fsn\" (UID: \"b0e8a1a4-c0cb-443e-8c38-9abcf242fc84\") " pod="openshift-cluster-node-tuning-operator/tuned-k2fsn"
May 25 11:17:08.982365 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:08.981898    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-dmzql"
May 25 11:17:08.982365 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:08.981970    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-bxzz6"
May 25 11:17:08.984174 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:08.984105    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-444mb"
May 25 11:17:08.984174 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:08.984157    2561 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-444mb" podUID="ead20bcd-df93-4d45-b5bb-470a8fe373e7"
May 25 11:17:08.984552 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:08.984532    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-mb4qd\""
May 25 11:17:08.984653 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:08.984575    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
May 25 11:17:08.984653 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:08.984575    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
May 25 11:17:08.984653 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:08.984630    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
May 25 11:17:08.984828 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:08.984787    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
May 25 11:17:08.985079 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:08.985062    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
May 25 11:17:08.985241 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:08.985223    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
May 25 11:17:08.985392 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:08.985332    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-qpp88\""
May 25 11:17:09.068797 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.068775    2561 desired_state_of_world_populator.go:158] "Finished populating initial desired state of world"
May 25 11:17:09.081204 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.081180    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b0e8a1a4-c0cb-443e-8c38-9abcf242fc84-etc-kubernetes\") pod \"tuned-k2fsn\" (UID: \"b0e8a1a4-c0cb-443e-8c38-9abcf242fc84\") " pod="openshift-cluster-node-tuning-operator/tuned-k2fsn"
May 25 11:17:09.081337 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.081213    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b0e8a1a4-c0cb-443e-8c38-9abcf242fc84-sys\") pod \"tuned-k2fsn\" (UID: \"b0e8a1a4-c0cb-443e-8c38-9abcf242fc84\") " pod="openshift-cluster-node-tuning-operator/tuned-k2fsn"
May 25 11:17:09.081337 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.081234    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a3d75818-4ca3-4bb1-a3dd-28ae4ce71d09-system-cni-dir\") pod \"multus-bxzz6\" (UID: \"a3d75818-4ca3-4bb1-a3dd-28ae4ce71d09\") " pod="openshift-multus/multus-bxzz6"
May 25 11:17:09.081337 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.081254    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a3d75818-4ca3-4bb1-a3dd-28ae4ce71d09-host-var-lib-cni-multus\") pod \"multus-bxzz6\" (UID: \"a3d75818-4ca3-4bb1-a3dd-28ae4ce71d09\") " pod="openshift-multus/multus-bxzz6"
May 25 11:17:09.081337 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.081293    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/5b2bf5ae-a8a4-48e9-9afc-ee576988791e-host-kubelet\") pod \"ovnkube-node-thvg5\" (UID: \"5b2bf5ae-a8a4-48e9-9afc-ee576988791e\") " pod="openshift-ovn-kubernetes/ovnkube-node-thvg5"
May 25 11:17:09.081337 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.081321    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5b2bf5ae-a8a4-48e9-9afc-ee576988791e-host-run-netns\") pod \"ovnkube-node-thvg5\" (UID: \"5b2bf5ae-a8a4-48e9-9afc-ee576988791e\") " pod="openshift-ovn-kubernetes/ovnkube-node-thvg5"
May 25 11:17:09.081337 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.081325    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b0e8a1a4-c0cb-443e-8c38-9abcf242fc84-sys\") pod \"tuned-k2fsn\" (UID: \"b0e8a1a4-c0cb-443e-8c38-9abcf242fc84\") " pod="openshift-cluster-node-tuning-operator/tuned-k2fsn"
May 25 11:17:09.081337 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.081328    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b0e8a1a4-c0cb-443e-8c38-9abcf242fc84-etc-kubernetes\") pod \"tuned-k2fsn\" (UID: \"b0e8a1a4-c0cb-443e-8c38-9abcf242fc84\") " pod="openshift-cluster-node-tuning-operator/tuned-k2fsn"
May 25 11:17:09.081649 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.081343    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/5b2bf5ae-a8a4-48e9-9afc-ee576988791e-log-socket\") pod \"ovnkube-node-thvg5\" (UID: \"5b2bf5ae-a8a4-48e9-9afc-ee576988791e\") " pod="openshift-ovn-kubernetes/ovnkube-node-thvg5"
May 25 11:17:09.081649 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.081355    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/5b2bf5ae-a8a4-48e9-9afc-ee576988791e-host-kubelet\") pod \"ovnkube-node-thvg5\" (UID: \"5b2bf5ae-a8a4-48e9-9afc-ee576988791e\") " pod="openshift-ovn-kubernetes/ovnkube-node-thvg5"
May 25 11:17:09.081649 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.081377    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5b2bf5ae-a8a4-48e9-9afc-ee576988791e-host-run-netns\") pod \"ovnkube-node-thvg5\" (UID: \"5b2bf5ae-a8a4-48e9-9afc-ee576988791e\") " pod="openshift-ovn-kubernetes/ovnkube-node-thvg5"
May 25 11:17:09.081649 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.081383    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/5b2bf5ae-a8a4-48e9-9afc-ee576988791e-log-socket\") pod \"ovnkube-node-thvg5\" (UID: \"5b2bf5ae-a8a4-48e9-9afc-ee576988791e\") " pod="openshift-ovn-kubernetes/ovnkube-node-thvg5"
May 25 11:17:09.081649 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.081383    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/5b2bf5ae-a8a4-48e9-9afc-ee576988791e-host-cni-netd\") pod \"ovnkube-node-thvg5\" (UID: \"5b2bf5ae-a8a4-48e9-9afc-ee576988791e\") " pod="openshift-ovn-kubernetes/ovnkube-node-thvg5"
May 25 11:17:09.081649 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.081416    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/71885334-70ca-483b-8bd9-fdbfecb77949-kubelet-dir\") pod \"aws-ebs-csi-driver-node-5ldrp\" (UID: \"71885334-70ca-483b-8bd9-fdbfecb77949\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5ldrp"
May 25 11:17:09.081649 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.081420    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/5b2bf5ae-a8a4-48e9-9afc-ee576988791e-host-cni-netd\") pod \"ovnkube-node-thvg5\" (UID: \"5b2bf5ae-a8a4-48e9-9afc-ee576988791e\") " pod="openshift-ovn-kubernetes/ovnkube-node-thvg5"
May 25 11:17:09.081649 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.081442    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a3d75818-4ca3-4bb1-a3dd-28ae4ce71d09-hostroot\") pod \"multus-bxzz6\" (UID: \"a3d75818-4ca3-4bb1-a3dd-28ae4ce71d09\") " pod="openshift-multus/multus-bxzz6"
May 25 11:17:09.081649 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.081467    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a3d75818-4ca3-4bb1-a3dd-28ae4ce71d09-host-run-multus-certs\") pod \"multus-bxzz6\" (UID: \"a3d75818-4ca3-4bb1-a3dd-28ae4ce71d09\") " pod="openshift-multus/multus-bxzz6"
May 25 11:17:09.081649 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.081490    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7c97daaa-b9c1-4ee0-ade1-e8de392e496f-system-cni-dir\") pod \"multus-additional-cni-plugins-dmzql\" (UID: \"7c97daaa-b9c1-4ee0-ade1-e8de392e496f\") " pod="openshift-multus/multus-additional-cni-plugins-dmzql"
May 25 11:17:09.081649 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.081501    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/71885334-70ca-483b-8bd9-fdbfecb77949-kubelet-dir\") pod \"aws-ebs-csi-driver-node-5ldrp\" (UID: \"71885334-70ca-483b-8bd9-fdbfecb77949\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5ldrp"
May 25 11:17:09.081649 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.081507    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5b2bf5ae-a8a4-48e9-9afc-ee576988791e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-thvg5\" (UID: \"5b2bf5ae-a8a4-48e9-9afc-ee576988791e\") " pod="openshift-ovn-kubernetes/ovnkube-node-thvg5"
May 25 11:17:09.081649 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.081544    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5b2bf5ae-a8a4-48e9-9afc-ee576988791e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-thvg5\" (UID: \"5b2bf5ae-a8a4-48e9-9afc-ee576988791e\") " pod="openshift-ovn-kubernetes/ovnkube-node-thvg5"
May 25 11:17:09.081649 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.081546    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/71885334-70ca-483b-8bd9-fdbfecb77949-sys-fs\") pod \"aws-ebs-csi-driver-node-5ldrp\" (UID: \"71885334-70ca-483b-8bd9-fdbfecb77949\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5ldrp"
May 25 11:17:09.081649 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.081572    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b0e8a1a4-c0cb-443e-8c38-9abcf242fc84-host\") pod \"tuned-k2fsn\" (UID: \"b0e8a1a4-c0cb-443e-8c38-9abcf242fc84\") " pod="openshift-cluster-node-tuning-operator/tuned-k2fsn"
May 25 11:17:09.081649 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.081581    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/71885334-70ca-483b-8bd9-fdbfecb77949-sys-fs\") pod \"aws-ebs-csi-driver-node-5ldrp\" (UID: \"71885334-70ca-483b-8bd9-fdbfecb77949\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5ldrp"
May 25 11:17:09.081649 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.081587    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9a250955-746b-416a-b798-283a0ea46864-serviceca\") pod \"node-ca-q8ng6\" (UID: \"9a250955-746b-416a-b798-283a0ea46864\") " pod="openshift-image-registry/node-ca-q8ng6"
May 25 11:17:09.082343 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.081611    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a3d75818-4ca3-4bb1-a3dd-28ae4ce71d09-host-var-lib-cni-bin\") pod \"multus-bxzz6\" (UID: \"a3d75818-4ca3-4bb1-a3dd-28ae4ce71d09\") " pod="openshift-multus/multus-bxzz6"
May 25 11:17:09.082343 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.081616    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b0e8a1a4-c0cb-443e-8c38-9abcf242fc84-host\") pod \"tuned-k2fsn\" (UID: \"b0e8a1a4-c0cb-443e-8c38-9abcf242fc84\") " pod="openshift-cluster-node-tuning-operator/tuned-k2fsn"
May 25 11:17:09.082343 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.081642    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6wpk\" (UniqueName: \"kubernetes.io/projected/7c97daaa-b9c1-4ee0-ade1-e8de392e496f-kube-api-access-g6wpk\") pod \"multus-additional-cni-plugins-dmzql\" (UID: \"7c97daaa-b9c1-4ee0-ade1-e8de392e496f\") " pod="openshift-multus/multus-additional-cni-plugins-dmzql"
May 25 11:17:09.082343 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.081672    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-d68xg\" (UniqueName: \"kubernetes.io/projected/71885334-70ca-483b-8bd9-fdbfecb77949-kube-api-access-d68xg\") pod \"aws-ebs-csi-driver-node-5ldrp\" (UID: \"71885334-70ca-483b-8bd9-fdbfecb77949\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5ldrp"
May 25 11:17:09.082343 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.081698    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8c6r8\" (UniqueName: \"kubernetes.io/projected/a3d75818-4ca3-4bb1-a3dd-28ae4ce71d09-kube-api-access-8c6r8\") pod \"multus-bxzz6\" (UID: \"a3d75818-4ca3-4bb1-a3dd-28ae4ce71d09\") " pod="openshift-multus/multus-bxzz6"
May 25 11:17:09.082343 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.081721    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/7c97daaa-b9c1-4ee0-ade1-e8de392e496f-os-release\") pod \"multus-additional-cni-plugins-dmzql\" (UID: \"7c97daaa-b9c1-4ee0-ade1-e8de392e496f\") " pod="openshift-multus/multus-additional-cni-plugins-dmzql"
May 25 11:17:09.082343 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.081744    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7c97daaa-b9c1-4ee0-ade1-e8de392e496f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-dmzql\" (UID: \"7c97daaa-b9c1-4ee0-ade1-e8de392e496f\") " pod="openshift-multus/multus-additional-cni-plugins-dmzql"
May 25 11:17:09.082343 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.081768    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7c97daaa-b9c1-4ee0-ade1-e8de392e496f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-dmzql\" (UID: \"7c97daaa-b9c1-4ee0-ade1-e8de392e496f\") " pod="openshift-multus/multus-additional-cni-plugins-dmzql"
May 25 11:17:09.082343 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.081827    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5b2bf5ae-a8a4-48e9-9afc-ee576988791e-host-slash\") pod \"ovnkube-node-thvg5\" (UID: \"5b2bf5ae-a8a4-48e9-9afc-ee576988791e\") " pod="openshift-ovn-kubernetes/ovnkube-node-thvg5"
May 25 11:17:09.082343 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.081850    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5b2bf5ae-a8a4-48e9-9afc-ee576988791e-etc-openvswitch\") pod \"ovnkube-node-thvg5\" (UID: \"5b2bf5ae-a8a4-48e9-9afc-ee576988791e\") " pod="openshift-ovn-kubernetes/ovnkube-node-thvg5"
May 25 11:17:09.082343 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.081875    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-6qj5l\" (UniqueName: \"kubernetes.io/projected/5b2bf5ae-a8a4-48e9-9afc-ee576988791e-kube-api-access-6qj5l\") pod \"ovnkube-node-thvg5\" (UID: \"5b2bf5ae-a8a4-48e9-9afc-ee576988791e\") " pod="openshift-ovn-kubernetes/ovnkube-node-thvg5"
May 25 11:17:09.082343 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.081881    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5b2bf5ae-a8a4-48e9-9afc-ee576988791e-host-slash\") pod \"ovnkube-node-thvg5\" (UID: \"5b2bf5ae-a8a4-48e9-9afc-ee576988791e\") " pod="openshift-ovn-kubernetes/ovnkube-node-thvg5"
May 25 11:17:09.082343 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.081899    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-q5nqq\" (UniqueName: \"kubernetes.io/projected/a8a3a45a-3dba-4919-895a-ef467fbd4343-kube-api-access-q5nqq\") pod \"iptables-alerter-4xz76\" (UID: \"a8a3a45a-3dba-4919-895a-ef467fbd4343\") " pod="openshift-network-operator/iptables-alerter-4xz76"
May 25 11:17:09.082343 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.081923    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b0e8a1a4-c0cb-443e-8c38-9abcf242fc84-lib-modules\") pod \"tuned-k2fsn\" (UID: \"b0e8a1a4-c0cb-443e-8c38-9abcf242fc84\") " pod="openshift-cluster-node-tuning-operator/tuned-k2fsn"
May 25 11:17:09.082343 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.081928    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5b2bf5ae-a8a4-48e9-9afc-ee576988791e-etc-openvswitch\") pod \"ovnkube-node-thvg5\" (UID: \"5b2bf5ae-a8a4-48e9-9afc-ee576988791e\") " pod="openshift-ovn-kubernetes/ovnkube-node-thvg5"
May 25 11:17:09.082343 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.081946    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b0e8a1a4-c0cb-443e-8c38-9abcf242fc84-etc-tuned\") pod \"tuned-k2fsn\" (UID: \"b0e8a1a4-c0cb-443e-8c38-9abcf242fc84\") " pod="openshift-cluster-node-tuning-operator/tuned-k2fsn"
May 25 11:17:09.082343 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.081992    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9a250955-746b-416a-b798-283a0ea46864-host\") pod \"node-ca-q8ng6\" (UID: \"9a250955-746b-416a-b798-283a0ea46864\") " pod="openshift-image-registry/node-ca-q8ng6"
May 25 11:17:09.083029 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.082021    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5b2bf5ae-a8a4-48e9-9afc-ee576988791e-var-lib-openvswitch\") pod \"ovnkube-node-thvg5\" (UID: \"5b2bf5ae-a8a4-48e9-9afc-ee576988791e\") " pod="openshift-ovn-kubernetes/ovnkube-node-thvg5"
May 25 11:17:09.083029 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.082032    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9a250955-746b-416a-b798-283a0ea46864-serviceca\") pod \"node-ca-q8ng6\" (UID: \"9a250955-746b-416a-b798-283a0ea46864\") " pod="openshift-image-registry/node-ca-q8ng6"
May 25 11:17:09.083029 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.082047    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/5b2bf5ae-a8a4-48e9-9afc-ee576988791e-node-log\") pod \"ovnkube-node-thvg5\" (UID: \"5b2bf5ae-a8a4-48e9-9afc-ee576988791e\") " pod="openshift-ovn-kubernetes/ovnkube-node-thvg5"
May 25 11:17:09.083029 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.082076    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b55f6df2-1dbc-4fc5-a4e6-e679d289e9f5-hosts-file\") pod \"node-resolver-xvcpl\" (UID: \"b55f6df2-1dbc-4fc5-a4e6-e679d289e9f5\") " pod="openshift-dns/node-resolver-xvcpl"
May 25 11:17:09.083029 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.082076    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9a250955-746b-416a-b798-283a0ea46864-host\") pod \"node-ca-q8ng6\" (UID: \"9a250955-746b-416a-b798-283a0ea46864\") " pod="openshift-image-registry/node-ca-q8ng6"
May 25 11:17:09.083029 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.082091    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b0e8a1a4-c0cb-443e-8c38-9abcf242fc84-lib-modules\") pod \"tuned-k2fsn\" (UID: \"b0e8a1a4-c0cb-443e-8c38-9abcf242fc84\") " pod="openshift-cluster-node-tuning-operator/tuned-k2fsn"
May 25 11:17:09.083029 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.082096    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/5b2bf5ae-a8a4-48e9-9afc-ee576988791e-node-log\") pod \"ovnkube-node-thvg5\" (UID: \"5b2bf5ae-a8a4-48e9-9afc-ee576988791e\") " pod="openshift-ovn-kubernetes/ovnkube-node-thvg5"
May 25 11:17:09.083029 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.082102    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5b2bf5ae-a8a4-48e9-9afc-ee576988791e-var-lib-openvswitch\") pod \"ovnkube-node-thvg5\" (UID: \"5b2bf5ae-a8a4-48e9-9afc-ee576988791e\") " pod="openshift-ovn-kubernetes/ovnkube-node-thvg5"
May 25 11:17:09.083029 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.082112    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a3d75818-4ca3-4bb1-a3dd-28ae4ce71d09-cnibin\") pod \"multus-bxzz6\" (UID: \"a3d75818-4ca3-4bb1-a3dd-28ae4ce71d09\") " pod="openshift-multus/multus-bxzz6"
May 25 11:17:09.083029 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.082142    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6m6wg\" (UniqueName: \"kubernetes.io/projected/ead20bcd-df93-4d45-b5bb-470a8fe373e7-kube-api-access-6m6wg\") pod \"network-metrics-daemon-444mb\" (UID: \"ead20bcd-df93-4d45-b5bb-470a8fe373e7\") " pod="openshift-multus/network-metrics-daemon-444mb"
May 25 11:17:09.083029 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.082165    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7c97daaa-b9c1-4ee0-ade1-e8de392e496f-cni-binary-copy\") pod \"multus-additional-cni-plugins-dmzql\" (UID: \"7c97daaa-b9c1-4ee0-ade1-e8de392e496f\") " pod="openshift-multus/multus-additional-cni-plugins-dmzql"
May 25 11:17:09.083029 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.082195    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5b2bf5ae-a8a4-48e9-9afc-ee576988791e-host-run-ovn-kubernetes\") pod \"ovnkube-node-thvg5\" (UID: \"5b2bf5ae-a8a4-48e9-9afc-ee576988791e\") " pod="openshift-ovn-kubernetes/ovnkube-node-thvg5"
May 25 11:17:09.083029 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.082208    2561 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
May 25 11:17:09.083029 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.082220    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b0e8a1a4-c0cb-443e-8c38-9abcf242fc84-var-lib-kubelet\") pod \"tuned-k2fsn\" (UID: \"b0e8a1a4-c0cb-443e-8c38-9abcf242fc84\") " pod="openshift-cluster-node-tuning-operator/tuned-k2fsn"
May 25 11:17:09.083029 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.082252    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b0e8a1a4-c0cb-443e-8c38-9abcf242fc84-tmp\") pod \"tuned-k2fsn\" (UID: \"b0e8a1a4-c0cb-443e-8c38-9abcf242fc84\") " pod="openshift-cluster-node-tuning-operator/tuned-k2fsn"
May 25 11:17:09.083029 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.082291    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-gg7wx\" (UniqueName: \"kubernetes.io/projected/b0e8a1a4-c0cb-443e-8c38-9abcf242fc84-kube-api-access-gg7wx\") pod \"tuned-k2fsn\" (UID: \"b0e8a1a4-c0cb-443e-8c38-9abcf242fc84\") " pod="openshift-cluster-node-tuning-operator/tuned-k2fsn"
May 25 11:17:09.083029 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.082309    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/5b2bf5ae-a8a4-48e9-9afc-ee576988791e-ovnkube-config\") pod \"ovnkube-node-thvg5\" (UID: \"5b2bf5ae-a8a4-48e9-9afc-ee576988791e\") " pod="openshift-ovn-kubernetes/ovnkube-node-thvg5"
May 25 11:17:09.083029 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.082311    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b0e8a1a4-c0cb-443e-8c38-9abcf242fc84-var-lib-kubelet\") pod \"tuned-k2fsn\" (UID: \"b0e8a1a4-c0cb-443e-8c38-9abcf242fc84\") " pod="openshift-cluster-node-tuning-operator/tuned-k2fsn"
May 25 11:17:09.083932 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.082324    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b0e8a1a4-c0cb-443e-8c38-9abcf242fc84-etc-sysctl-d\") pod \"tuned-k2fsn\" (UID: \"b0e8a1a4-c0cb-443e-8c38-9abcf242fc84\") " pod="openshift-cluster-node-tuning-operator/tuned-k2fsn"
May 25 11:17:09.083932 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.082358    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ae10dbbc-8226-48df-a0f7-39aed3d95c58-konnectivity-ca\") pod \"konnectivity-agent-mr48p\" (UID: \"ae10dbbc-8226-48df-a0f7-39aed3d95c58\") " pod="kube-system/konnectivity-agent-mr48p"
May 25 11:17:09.083932 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.082394    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a3d75818-4ca3-4bb1-a3dd-28ae4ce71d09-multus-socket-dir-parent\") pod \"multus-bxzz6\" (UID: \"a3d75818-4ca3-4bb1-a3dd-28ae4ce71d09\") " pod="openshift-multus/multus-bxzz6"
May 25 11:17:09.083932 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.082411    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b0e8a1a4-c0cb-443e-8c38-9abcf242fc84-etc-sysctl-d\") pod \"tuned-k2fsn\" (UID: \"b0e8a1a4-c0cb-443e-8c38-9abcf242fc84\") " pod="openshift-cluster-node-tuning-operator/tuned-k2fsn"
May 25 11:17:09.083932 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.082420    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b0e8a1a4-c0cb-443e-8c38-9abcf242fc84-etc-sysctl-conf\") pod \"tuned-k2fsn\" (UID: \"b0e8a1a4-c0cb-443e-8c38-9abcf242fc84\") " pod="openshift-cluster-node-tuning-operator/tuned-k2fsn"
May 25 11:17:09.083932 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.082448    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b55f6df2-1dbc-4fc5-a4e6-e679d289e9f5-tmp-dir\") pod \"node-resolver-xvcpl\" (UID: \"b55f6df2-1dbc-4fc5-a4e6-e679d289e9f5\") " pod="openshift-dns/node-resolver-xvcpl"
May 25 11:17:09.083932 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.082474    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/71885334-70ca-483b-8bd9-fdbfecb77949-device-dir\") pod \"aws-ebs-csi-driver-node-5ldrp\" (UID: \"71885334-70ca-483b-8bd9-fdbfecb77949\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5ldrp"
May 25 11:17:09.083932 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.082312    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5b2bf5ae-a8a4-48e9-9afc-ee576988791e-host-run-ovn-kubernetes\") pod \"ovnkube-node-thvg5\" (UID: \"5b2bf5ae-a8a4-48e9-9afc-ee576988791e\") " pod="openshift-ovn-kubernetes/ovnkube-node-thvg5"
May 25 11:17:09.083932 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.082499    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a3d75818-4ca3-4bb1-a3dd-28ae4ce71d09-os-release\") pod \"multus-bxzz6\" (UID: \"a3d75818-4ca3-4bb1-a3dd-28ae4ce71d09\") " pod="openshift-multus/multus-bxzz6"
May 25 11:17:09.083932 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.082596    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/71885334-70ca-483b-8bd9-fdbfecb77949-device-dir\") pod \"aws-ebs-csi-driver-node-5ldrp\" (UID: \"71885334-70ca-483b-8bd9-fdbfecb77949\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5ldrp"
May 25 11:17:09.083932 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.082613    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b0e8a1a4-c0cb-443e-8c38-9abcf242fc84-etc-sysctl-conf\") pod \"tuned-k2fsn\" (UID: \"b0e8a1a4-c0cb-443e-8c38-9abcf242fc84\") " pod="openshift-cluster-node-tuning-operator/tuned-k2fsn"
May 25 11:17:09.083932 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.082622    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a3d75818-4ca3-4bb1-a3dd-28ae4ce71d09-host-var-lib-kubelet\") pod \"multus-bxzz6\" (UID: \"a3d75818-4ca3-4bb1-a3dd-28ae4ce71d09\") " pod="openshift-multus/multus-bxzz6"
May 25 11:17:09.083932 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.082681    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7c97daaa-b9c1-4ee0-ade1-e8de392e496f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-dmzql\" (UID: \"7c97daaa-b9c1-4ee0-ade1-e8de392e496f\") " pod="openshift-multus/multus-additional-cni-plugins-dmzql"
May 25 11:17:09.083932 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.082713    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/5b2bf5ae-a8a4-48e9-9afc-ee576988791e-run-ovn\") pod \"ovnkube-node-thvg5\" (UID: \"5b2bf5ae-a8a4-48e9-9afc-ee576988791e\") " pod="openshift-ovn-kubernetes/ovnkube-node-thvg5"
May 25 11:17:09.083932 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.082738    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5b2bf5ae-a8a4-48e9-9afc-ee576988791e-host-cni-bin\") pod \"ovnkube-node-thvg5\" (UID: \"5b2bf5ae-a8a4-48e9-9afc-ee576988791e\") " pod="openshift-ovn-kubernetes/ovnkube-node-thvg5"
May 25 11:17:09.083932 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.082762    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a8a3a45a-3dba-4919-895a-ef467fbd4343-host-slash\") pod \"iptables-alerter-4xz76\" (UID: \"a8a3a45a-3dba-4919-895a-ef467fbd4343\") " pod="openshift-network-operator/iptables-alerter-4xz76"
May 25 11:17:09.083932 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.082808    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbhnn\" (UniqueName: \"kubernetes.io/projected/2b05b78f-6ab1-4f94-8273-e5fa70b78d15-kube-api-access-nbhnn\") pod \"network-check-target-rl7hd\" (UID: \"2b05b78f-6ab1-4f94-8273-e5fa70b78d15\") " pod="openshift-network-diagnostics/network-check-target-rl7hd"
May 25 11:17:09.084670 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.082817    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/5b2bf5ae-a8a4-48e9-9afc-ee576988791e-run-ovn\") pod \"ovnkube-node-thvg5\" (UID: \"5b2bf5ae-a8a4-48e9-9afc-ee576988791e\") " pod="openshift-ovn-kubernetes/ovnkube-node-thvg5"
May 25 11:17:09.084670 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.082836    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a3d75818-4ca3-4bb1-a3dd-28ae4ce71d09-multus-conf-dir\") pod \"multus-bxzz6\" (UID: \"a3d75818-4ca3-4bb1-a3dd-28ae4ce71d09\") " pod="openshift-multus/multus-bxzz6"
May 25 11:17:09.084670 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.082842    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5b2bf5ae-a8a4-48e9-9afc-ee576988791e-host-cni-bin\") pod \"ovnkube-node-thvg5\" (UID: \"5b2bf5ae-a8a4-48e9-9afc-ee576988791e\") " pod="openshift-ovn-kubernetes/ovnkube-node-thvg5"
May 25 11:17:09.084670 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.082860    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jp8vr\" (UniqueName: \"kubernetes.io/projected/b55f6df2-1dbc-4fc5-a4e6-e679d289e9f5-kube-api-access-jp8vr\") pod \"node-resolver-xvcpl\" (UID: \"b55f6df2-1dbc-4fc5-a4e6-e679d289e9f5\") " pod="openshift-dns/node-resolver-xvcpl"
May 25 11:17:09.084670 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.082867    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a8a3a45a-3dba-4919-895a-ef467fbd4343-host-slash\") pod \"iptables-alerter-4xz76\" (UID: \"a8a3a45a-3dba-4919-895a-ef467fbd4343\") " pod="openshift-network-operator/iptables-alerter-4xz76"
May 25 11:17:09.084670 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.082887    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/5b2bf5ae-a8a4-48e9-9afc-ee576988791e-run-systemd\") pod \"ovnkube-node-thvg5\" (UID: \"5b2bf5ae-a8a4-48e9-9afc-ee576988791e\") " pod="openshift-ovn-kubernetes/ovnkube-node-thvg5"
May 25 11:17:09.084670 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.082913    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/5b2bf5ae-a8a4-48e9-9afc-ee576988791e-env-overrides\") pod \"ovnkube-node-thvg5\" (UID: \"5b2bf5ae-a8a4-48e9-9afc-ee576988791e\") " pod="openshift-ovn-kubernetes/ovnkube-node-thvg5"
May 25 11:17:09.084670 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.082915    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/5b2bf5ae-a8a4-48e9-9afc-ee576988791e-ovnkube-config\") pod \"ovnkube-node-thvg5\" (UID: \"5b2bf5ae-a8a4-48e9-9afc-ee576988791e\") " pod="openshift-ovn-kubernetes/ovnkube-node-thvg5"
May 25 11:17:09.084670 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.082938    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/71885334-70ca-483b-8bd9-fdbfecb77949-socket-dir\") pod \"aws-ebs-csi-driver-node-5ldrp\" (UID: \"71885334-70ca-483b-8bd9-fdbfecb77949\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5ldrp"
May 25 11:17:09.084670 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.082937    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/5b2bf5ae-a8a4-48e9-9afc-ee576988791e-run-systemd\") pod \"ovnkube-node-thvg5\" (UID: \"5b2bf5ae-a8a4-48e9-9afc-ee576988791e\") " pod="openshift-ovn-kubernetes/ovnkube-node-thvg5"
May 25 11:17:09.084670 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.082970    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a3d75818-4ca3-4bb1-a3dd-28ae4ce71d09-host-run-netns\") pod \"multus-bxzz6\" (UID: \"a3d75818-4ca3-4bb1-a3dd-28ae4ce71d09\") " pod="openshift-multus/multus-bxzz6"
May 25 11:17:09.084670 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.083012    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a3d75818-4ca3-4bb1-a3dd-28ae4ce71d09-etc-kubernetes\") pod \"multus-bxzz6\" (UID: \"a3d75818-4ca3-4bb1-a3dd-28ae4ce71d09\") " pod="openshift-multus/multus-bxzz6"
May 25 11:17:09.084670 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.083041    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/71885334-70ca-483b-8bd9-fdbfecb77949-etc-selinux\") pod \"aws-ebs-csi-driver-node-5ldrp\" (UID: \"71885334-70ca-483b-8bd9-fdbfecb77949\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5ldrp"
May 25 11:17:09.084670 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.083068    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b0e8a1a4-c0cb-443e-8c38-9abcf242fc84-etc-systemd\") pod \"tuned-k2fsn\" (UID: \"b0e8a1a4-c0cb-443e-8c38-9abcf242fc84\") " pod="openshift-cluster-node-tuning-operator/tuned-k2fsn"
May 25 11:17:09.084670 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.083094    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b0e8a1a4-c0cb-443e-8c38-9abcf242fc84-run\") pod \"tuned-k2fsn\" (UID: \"b0e8a1a4-c0cb-443e-8c38-9abcf242fc84\") " pod="openshift-cluster-node-tuning-operator/tuned-k2fsn"
May 25 11:17:09.084670 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.083120    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a3d75818-4ca3-4bb1-a3dd-28ae4ce71d09-multus-cni-dir\") pod \"multus-bxzz6\" (UID: \"a3d75818-4ca3-4bb1-a3dd-28ae4ce71d09\") " pod="openshift-multus/multus-bxzz6"
May 25 11:17:09.084670 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.083145    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a3d75818-4ca3-4bb1-a3dd-28ae4ce71d09-host-run-k8s-cni-cncf-io\") pod \"multus-bxzz6\" (UID: \"a3d75818-4ca3-4bb1-a3dd-28ae4ce71d09\") " pod="openshift-multus/multus-bxzz6"
May 25 11:17:09.085308 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.083158    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/71885334-70ca-483b-8bd9-fdbfecb77949-etc-selinux\") pod \"aws-ebs-csi-driver-node-5ldrp\" (UID: \"71885334-70ca-483b-8bd9-fdbfecb77949\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5ldrp"
May 25 11:17:09.085308 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.083172    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/5b2bf5ae-a8a4-48e9-9afc-ee576988791e-ovnkube-script-lib\") pod \"ovnkube-node-thvg5\" (UID: \"5b2bf5ae-a8a4-48e9-9afc-ee576988791e\") " pod="openshift-ovn-kubernetes/ovnkube-node-thvg5"
May 25 11:17:09.085308 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.083199    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a3d75818-4ca3-4bb1-a3dd-28ae4ce71d09-cni-binary-copy\") pod \"multus-bxzz6\" (UID: \"a3d75818-4ca3-4bb1-a3dd-28ae4ce71d09\") " pod="openshift-multus/multus-bxzz6"
May 25 11:17:09.085308 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.083222    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b0e8a1a4-c0cb-443e-8c38-9abcf242fc84-etc-systemd\") pod \"tuned-k2fsn\" (UID: \"b0e8a1a4-c0cb-443e-8c38-9abcf242fc84\") " pod="openshift-cluster-node-tuning-operator/tuned-k2fsn"
May 25 11:17:09.085308 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.083226    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ead20bcd-df93-4d45-b5bb-470a8fe373e7-metrics-certs\") pod \"network-metrics-daemon-444mb\" (UID: \"ead20bcd-df93-4d45-b5bb-470a8fe373e7\") " pod="openshift-multus/network-metrics-daemon-444mb"
May 25 11:17:09.085308 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.083068    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/71885334-70ca-483b-8bd9-fdbfecb77949-socket-dir\") pod \"aws-ebs-csi-driver-node-5ldrp\" (UID: \"71885334-70ca-483b-8bd9-fdbfecb77949\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5ldrp"
May 25 11:17:09.085308 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.083252    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5b2bf5ae-a8a4-48e9-9afc-ee576988791e-run-openvswitch\") pod \"ovnkube-node-thvg5\" (UID: \"5b2bf5ae-a8a4-48e9-9afc-ee576988791e\") " pod="openshift-ovn-kubernetes/ovnkube-node-thvg5"
May 25 11:17:09.085308 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.083309    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b0e8a1a4-c0cb-443e-8c38-9abcf242fc84-etc-modprobe-d\") pod \"tuned-k2fsn\" (UID: \"b0e8a1a4-c0cb-443e-8c38-9abcf242fc84\") " pod="openshift-cluster-node-tuning-operator/tuned-k2fsn"
May 25 11:17:09.085308 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.083334    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b0e8a1a4-c0cb-443e-8c38-9abcf242fc84-run\") pod \"tuned-k2fsn\" (UID: \"b0e8a1a4-c0cb-443e-8c38-9abcf242fc84\") " pod="openshift-cluster-node-tuning-operator/tuned-k2fsn"
May 25 11:17:09.085308 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.083263    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/5b2bf5ae-a8a4-48e9-9afc-ee576988791e-env-overrides\") pod \"ovnkube-node-thvg5\" (UID: \"5b2bf5ae-a8a4-48e9-9afc-ee576988791e\") " pod="openshift-ovn-kubernetes/ovnkube-node-thvg5"
May 25 11:17:09.085308 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.083352    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b0e8a1a4-c0cb-443e-8c38-9abcf242fc84-etc-sysconfig\") pod \"tuned-k2fsn\" (UID: \"b0e8a1a4-c0cb-443e-8c38-9abcf242fc84\") " pod="openshift-cluster-node-tuning-operator/tuned-k2fsn"
May 25 11:17:09.085308 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.083380    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-pq5fc\" (UniqueName: \"kubernetes.io/projected/9a250955-746b-416a-b798-283a0ea46864-kube-api-access-pq5fc\") pod \"node-ca-q8ng6\" (UID: \"9a250955-746b-416a-b798-283a0ea46864\") " pod="openshift-image-registry/node-ca-q8ng6"
May 25 11:17:09.085308 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.083402    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5b2bf5ae-a8a4-48e9-9afc-ee576988791e-run-openvswitch\") pod \"ovnkube-node-thvg5\" (UID: \"5b2bf5ae-a8a4-48e9-9afc-ee576988791e\") " pod="openshift-ovn-kubernetes/ovnkube-node-thvg5"
May 25 11:17:09.085308 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.083408    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ae10dbbc-8226-48df-a0f7-39aed3d95c58-agent-certs\") pod \"konnectivity-agent-mr48p\" (UID: \"ae10dbbc-8226-48df-a0f7-39aed3d95c58\") " pod="kube-system/konnectivity-agent-mr48p"
May 25 11:17:09.085308 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.083432    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a3d75818-4ca3-4bb1-a3dd-28ae4ce71d09-multus-daemon-config\") pod \"multus-bxzz6\" (UID: \"a3d75818-4ca3-4bb1-a3dd-28ae4ce71d09\") " pod="openshift-multus/multus-bxzz6"
May 25 11:17:09.085308 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.083454    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7c97daaa-b9c1-4ee0-ade1-e8de392e496f-cnibin\") pod \"multus-additional-cni-plugins-dmzql\" (UID: \"7c97daaa-b9c1-4ee0-ade1-e8de392e496f\") " pod="openshift-multus/multus-additional-cni-plugins-dmzql"
May 25 11:17:09.085308 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.083479    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/5b2bf5ae-a8a4-48e9-9afc-ee576988791e-systemd-units\") pod \"ovnkube-node-thvg5\" (UID: \"5b2bf5ae-a8a4-48e9-9afc-ee576988791e\") " pod="openshift-ovn-kubernetes/ovnkube-node-thvg5"
May 25 11:17:09.085998 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.083491    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b0e8a1a4-c0cb-443e-8c38-9abcf242fc84-etc-modprobe-d\") pod \"tuned-k2fsn\" (UID: \"b0e8a1a4-c0cb-443e-8c38-9abcf242fc84\") " pod="openshift-cluster-node-tuning-operator/tuned-k2fsn"
May 25 11:17:09.085998 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.083532    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/5b2bf5ae-a8a4-48e9-9afc-ee576988791e-ovn-node-metrics-cert\") pod \"ovnkube-node-thvg5\" (UID: \"5b2bf5ae-a8a4-48e9-9afc-ee576988791e\") " pod="openshift-ovn-kubernetes/ovnkube-node-thvg5"
May 25 11:17:09.085998 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.083546    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/5b2bf5ae-a8a4-48e9-9afc-ee576988791e-systemd-units\") pod \"ovnkube-node-thvg5\" (UID: \"5b2bf5ae-a8a4-48e9-9afc-ee576988791e\") " pod="openshift-ovn-kubernetes/ovnkube-node-thvg5"
May 25 11:17:09.085998 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.083559    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a8a3a45a-3dba-4919-895a-ef467fbd4343-iptables-alerter-script\") pod \"iptables-alerter-4xz76\" (UID: \"a8a3a45a-3dba-4919-895a-ef467fbd4343\") " pod="openshift-network-operator/iptables-alerter-4xz76"
May 25 11:17:09.085998 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.083585    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/71885334-70ca-483b-8bd9-fdbfecb77949-registration-dir\") pod \"aws-ebs-csi-driver-node-5ldrp\" (UID: \"71885334-70ca-483b-8bd9-fdbfecb77949\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5ldrp"
May 25 11:17:09.085998 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.083595    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b0e8a1a4-c0cb-443e-8c38-9abcf242fc84-etc-sysconfig\") pod \"tuned-k2fsn\" (UID: \"b0e8a1a4-c0cb-443e-8c38-9abcf242fc84\") " pod="openshift-cluster-node-tuning-operator/tuned-k2fsn"
May 25 11:17:09.085998 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.083673    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/71885334-70ca-483b-8bd9-fdbfecb77949-registration-dir\") pod \"aws-ebs-csi-driver-node-5ldrp\" (UID: \"71885334-70ca-483b-8bd9-fdbfecb77949\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5ldrp"
May 25 11:17:09.085998 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.083719    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/5b2bf5ae-a8a4-48e9-9afc-ee576988791e-ovnkube-script-lib\") pod \"ovnkube-node-thvg5\" (UID: \"5b2bf5ae-a8a4-48e9-9afc-ee576988791e\") " pod="openshift-ovn-kubernetes/ovnkube-node-thvg5"
May 25 11:17:09.085998 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.084236    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a8a3a45a-3dba-4919-895a-ef467fbd4343-iptables-alerter-script\") pod \"iptables-alerter-4xz76\" (UID: \"a8a3a45a-3dba-4919-895a-ef467fbd4343\") " pod="openshift-network-operator/iptables-alerter-4xz76"
May 25 11:17:09.085998 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.085942    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b0e8a1a4-c0cb-443e-8c38-9abcf242fc84-tmp\") pod \"tuned-k2fsn\" (UID: \"b0e8a1a4-c0cb-443e-8c38-9abcf242fc84\") " pod="openshift-cluster-node-tuning-operator/tuned-k2fsn"
May 25 11:17:09.086343 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.086003    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b0e8a1a4-c0cb-443e-8c38-9abcf242fc84-etc-tuned\") pod \"tuned-k2fsn\" (UID: \"b0e8a1a4-c0cb-443e-8c38-9abcf242fc84\") " pod="openshift-cluster-node-tuning-operator/tuned-k2fsn"
May 25 11:17:09.086343 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.086141    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/5b2bf5ae-a8a4-48e9-9afc-ee576988791e-ovn-node-metrics-cert\") pod \"ovnkube-node-thvg5\" (UID: \"5b2bf5ae-a8a4-48e9-9afc-ee576988791e\") " pod="openshift-ovn-kubernetes/ovnkube-node-thvg5"
May 25 11:17:09.093024 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.092899    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gg7wx\" (UniqueName: \"kubernetes.io/projected/b0e8a1a4-c0cb-443e-8c38-9abcf242fc84-kube-api-access-gg7wx\") pod \"tuned-k2fsn\" (UID: \"b0e8a1a4-c0cb-443e-8c38-9abcf242fc84\") " pod="openshift-cluster-node-tuning-operator/tuned-k2fsn"
May 25 11:17:09.093024 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.092997    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d68xg\" (UniqueName: \"kubernetes.io/projected/71885334-70ca-483b-8bd9-fdbfecb77949-kube-api-access-d68xg\") pod \"aws-ebs-csi-driver-node-5ldrp\" (UID: \"71885334-70ca-483b-8bd9-fdbfecb77949\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5ldrp"
May 25 11:17:09.093024 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.092999    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5nqq\" (UniqueName: \"kubernetes.io/projected/a8a3a45a-3dba-4919-895a-ef467fbd4343-kube-api-access-q5nqq\") pod \"iptables-alerter-4xz76\" (UID: \"a8a3a45a-3dba-4919-895a-ef467fbd4343\") " pod="openshift-network-operator/iptables-alerter-4xz76"
May 25 11:17:09.093490 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.093455    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pq5fc\" (UniqueName: \"kubernetes.io/projected/9a250955-746b-416a-b798-283a0ea46864-kube-api-access-pq5fc\") pod \"node-ca-q8ng6\" (UID: \"9a250955-746b-416a-b798-283a0ea46864\") " pod="openshift-image-registry/node-ca-q8ng6"
May 25 11:17:09.093850 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.093819    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qj5l\" (UniqueName: \"kubernetes.io/projected/5b2bf5ae-a8a4-48e9-9afc-ee576988791e-kube-api-access-6qj5l\") pod \"ovnkube-node-thvg5\" (UID: \"5b2bf5ae-a8a4-48e9-9afc-ee576988791e\") " pod="openshift-ovn-kubernetes/ovnkube-node-thvg5"
May 25 11:17:09.114236 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.114209    2561 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-05-24 11:12:08 +0000 UTC" deadline="2028-01-07 05:07:34.563373928 +0000 UTC"
May 25 11:17:09.114236 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.114232    2561 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14201h50m25.449144416s"
May 25 11:17:09.184170 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.184102    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ae10dbbc-8226-48df-a0f7-39aed3d95c58-agent-certs\") pod \"konnectivity-agent-mr48p\" (UID: \"ae10dbbc-8226-48df-a0f7-39aed3d95c58\") " pod="kube-system/konnectivity-agent-mr48p"
May 25 11:17:09.184170 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.184135    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a3d75818-4ca3-4bb1-a3dd-28ae4ce71d09-multus-daemon-config\") pod \"multus-bxzz6\" (UID: \"a3d75818-4ca3-4bb1-a3dd-28ae4ce71d09\") " pod="openshift-multus/multus-bxzz6"
May 25 11:17:09.184170 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.184159    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7c97daaa-b9c1-4ee0-ade1-e8de392e496f-cnibin\") pod \"multus-additional-cni-plugins-dmzql\" (UID: \"7c97daaa-b9c1-4ee0-ade1-e8de392e496f\") " pod="openshift-multus/multus-additional-cni-plugins-dmzql"
May 25 11:17:09.184441 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.184186    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a3d75818-4ca3-4bb1-a3dd-28ae4ce71d09-system-cni-dir\") pod \"multus-bxzz6\" (UID: \"a3d75818-4ca3-4bb1-a3dd-28ae4ce71d09\") " pod="openshift-multus/multus-bxzz6"
May 25 11:17:09.184441 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.184206    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a3d75818-4ca3-4bb1-a3dd-28ae4ce71d09-host-var-lib-cni-multus\") pod \"multus-bxzz6\" (UID: \"a3d75818-4ca3-4bb1-a3dd-28ae4ce71d09\") " pod="openshift-multus/multus-bxzz6"
May 25 11:17:09.184441 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.184230    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a3d75818-4ca3-4bb1-a3dd-28ae4ce71d09-hostroot\") pod \"multus-bxzz6\" (UID: \"a3d75818-4ca3-4bb1-a3dd-28ae4ce71d09\") " pod="openshift-multus/multus-bxzz6"
May 25 11:17:09.184441 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.184252    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a3d75818-4ca3-4bb1-a3dd-28ae4ce71d09-host-run-multus-certs\") pod \"multus-bxzz6\" (UID: \"a3d75818-4ca3-4bb1-a3dd-28ae4ce71d09\") " pod="openshift-multus/multus-bxzz6"
May 25 11:17:09.184441 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.184289    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7c97daaa-b9c1-4ee0-ade1-e8de392e496f-system-cni-dir\") pod \"multus-additional-cni-plugins-dmzql\" (UID: \"7c97daaa-b9c1-4ee0-ade1-e8de392e496f\") " pod="openshift-multus/multus-additional-cni-plugins-dmzql"
May 25 11:17:09.184441 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.184317    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a3d75818-4ca3-4bb1-a3dd-28ae4ce71d09-host-var-lib-cni-bin\") pod \"multus-bxzz6\" (UID: \"a3d75818-4ca3-4bb1-a3dd-28ae4ce71d09\") " pod="openshift-multus/multus-bxzz6"
May 25 11:17:09.184441 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.184324    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a3d75818-4ca3-4bb1-a3dd-28ae4ce71d09-system-cni-dir\") pod \"multus-bxzz6\" (UID: \"a3d75818-4ca3-4bb1-a3dd-28ae4ce71d09\") " pod="openshift-multus/multus-bxzz6"
May 25 11:17:09.184441 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.184338    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a3d75818-4ca3-4bb1-a3dd-28ae4ce71d09-host-run-multus-certs\") pod \"multus-bxzz6\" (UID: \"a3d75818-4ca3-4bb1-a3dd-28ae4ce71d09\") " pod="openshift-multus/multus-bxzz6"
May 25 11:17:09.184441 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.184314    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a3d75818-4ca3-4bb1-a3dd-28ae4ce71d09-host-var-lib-cni-multus\") pod \"multus-bxzz6\" (UID: \"a3d75818-4ca3-4bb1-a3dd-28ae4ce71d09\") " pod="openshift-multus/multus-bxzz6"
May 25 11:17:09.184441 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.184327    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a3d75818-4ca3-4bb1-a3dd-28ae4ce71d09-hostroot\") pod \"multus-bxzz6\" (UID: \"a3d75818-4ca3-4bb1-a3dd-28ae4ce71d09\") " pod="openshift-multus/multus-bxzz6"
May 25 11:17:09.184441 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.184345    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-g6wpk\" (UniqueName: \"kubernetes.io/projected/7c97daaa-b9c1-4ee0-ade1-e8de392e496f-kube-api-access-g6wpk\") pod \"multus-additional-cni-plugins-dmzql\" (UID: \"7c97daaa-b9c1-4ee0-ade1-e8de392e496f\") " pod="openshift-multus/multus-additional-cni-plugins-dmzql"
May 25 11:17:09.184441 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.184350    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7c97daaa-b9c1-4ee0-ade1-e8de392e496f-system-cni-dir\") pod \"multus-additional-cni-plugins-dmzql\" (UID: \"7c97daaa-b9c1-4ee0-ade1-e8de392e496f\") " pod="openshift-multus/multus-additional-cni-plugins-dmzql"
May 25 11:17:09.184441 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.184365    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a3d75818-4ca3-4bb1-a3dd-28ae4ce71d09-host-var-lib-cni-bin\") pod \"multus-bxzz6\" (UID: \"a3d75818-4ca3-4bb1-a3dd-28ae4ce71d09\") " pod="openshift-multus/multus-bxzz6"
May 25 11:17:09.184441 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.184399    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-8c6r8\" (UniqueName: \"kubernetes.io/projected/a3d75818-4ca3-4bb1-a3dd-28ae4ce71d09-kube-api-access-8c6r8\") pod \"multus-bxzz6\" (UID: \"a3d75818-4ca3-4bb1-a3dd-28ae4ce71d09\") " pod="openshift-multus/multus-bxzz6"
May 25 11:17:09.184441 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.184418    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7c97daaa-b9c1-4ee0-ade1-e8de392e496f-cnibin\") pod \"multus-additional-cni-plugins-dmzql\" (UID: \"7c97daaa-b9c1-4ee0-ade1-e8de392e496f\") " pod="openshift-multus/multus-additional-cni-plugins-dmzql"
May 25 11:17:09.184441 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.184437    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/7c97daaa-b9c1-4ee0-ade1-e8de392e496f-os-release\") pod \"multus-additional-cni-plugins-dmzql\" (UID: \"7c97daaa-b9c1-4ee0-ade1-e8de392e496f\") " pod="openshift-multus/multus-additional-cni-plugins-dmzql"
May 25 11:17:09.185153 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.184459    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7c97daaa-b9c1-4ee0-ade1-e8de392e496f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-dmzql\" (UID: \"7c97daaa-b9c1-4ee0-ade1-e8de392e496f\") " pod="openshift-multus/multus-additional-cni-plugins-dmzql"
May 25 11:17:09.185153 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.184502    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7c97daaa-b9c1-4ee0-ade1-e8de392e496f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-dmzql\" (UID: \"7c97daaa-b9c1-4ee0-ade1-e8de392e496f\") " pod="openshift-multus/multus-additional-cni-plugins-dmzql"
May 25 11:17:09.185153 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.184517    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/7c97daaa-b9c1-4ee0-ade1-e8de392e496f-os-release\") pod \"multus-additional-cni-plugins-dmzql\" (UID: \"7c97daaa-b9c1-4ee0-ade1-e8de392e496f\") " pod="openshift-multus/multus-additional-cni-plugins-dmzql"
May 25 11:17:09.185153 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.184540    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b55f6df2-1dbc-4fc5-a4e6-e679d289e9f5-hosts-file\") pod \"node-resolver-xvcpl\" (UID: \"b55f6df2-1dbc-4fc5-a4e6-e679d289e9f5\") " pod="openshift-dns/node-resolver-xvcpl"
May 25 11:17:09.185153 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.184592    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a3d75818-4ca3-4bb1-a3dd-28ae4ce71d09-cnibin\") pod \"multus-bxzz6\" (UID: \"a3d75818-4ca3-4bb1-a3dd-28ae4ce71d09\") " pod="openshift-multus/multus-bxzz6"
May 25 11:17:09.185153 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.184634    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-6m6wg\" (UniqueName: \"kubernetes.io/projected/ead20bcd-df93-4d45-b5bb-470a8fe373e7-kube-api-access-6m6wg\") pod \"network-metrics-daemon-444mb\" (UID: \"ead20bcd-df93-4d45-b5bb-470a8fe373e7\") " pod="openshift-multus/network-metrics-daemon-444mb"
May 25 11:17:09.185153 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.184660    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7c97daaa-b9c1-4ee0-ade1-e8de392e496f-cni-binary-copy\") pod \"multus-additional-cni-plugins-dmzql\" (UID: \"7c97daaa-b9c1-4ee0-ade1-e8de392e496f\") " pod="openshift-multus/multus-additional-cni-plugins-dmzql"
May 25 11:17:09.185153 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.184686    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7c97daaa-b9c1-4ee0-ade1-e8de392e496f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-dmzql\" (UID: \"7c97daaa-b9c1-4ee0-ade1-e8de392e496f\") " pod="openshift-multus/multus-additional-cni-plugins-dmzql"
May 25 11:17:09.185153 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.184708    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ae10dbbc-8226-48df-a0f7-39aed3d95c58-konnectivity-ca\") pod \"konnectivity-agent-mr48p\" (UID: \"ae10dbbc-8226-48df-a0f7-39aed3d95c58\") " pod="kube-system/konnectivity-agent-mr48p"
May 25 11:17:09.185153 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.184739    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b55f6df2-1dbc-4fc5-a4e6-e679d289e9f5-hosts-file\") pod \"node-resolver-xvcpl\" (UID: \"b55f6df2-1dbc-4fc5-a4e6-e679d289e9f5\") " pod="openshift-dns/node-resolver-xvcpl"
May 25 11:17:09.185153 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.184748    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a3d75818-4ca3-4bb1-a3dd-28ae4ce71d09-multus-socket-dir-parent\") pod \"multus-bxzz6\" (UID: \"a3d75818-4ca3-4bb1-a3dd-28ae4ce71d09\") " pod="openshift-multus/multus-bxzz6"
May 25 11:17:09.185153 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.184776    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b55f6df2-1dbc-4fc5-a4e6-e679d289e9f5-tmp-dir\") pod \"node-resolver-xvcpl\" (UID: \"b55f6df2-1dbc-4fc5-a4e6-e679d289e9f5\") " pod="openshift-dns/node-resolver-xvcpl"
May 25 11:17:09.185153 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.184846    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a3d75818-4ca3-4bb1-a3dd-28ae4ce71d09-multus-daemon-config\") pod \"multus-bxzz6\" (UID: \"a3d75818-4ca3-4bb1-a3dd-28ae4ce71d09\") " pod="openshift-multus/multus-bxzz6"
May 25 11:17:09.185153 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.184960    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a3d75818-4ca3-4bb1-a3dd-28ae4ce71d09-multus-socket-dir-parent\") pod \"multus-bxzz6\" (UID: \"a3d75818-4ca3-4bb1-a3dd-28ae4ce71d09\") " pod="openshift-multus/multus-bxzz6"
May 25 11:17:09.185153 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.185137    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7c97daaa-b9c1-4ee0-ade1-e8de392e496f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-dmzql\" (UID: \"7c97daaa-b9c1-4ee0-ade1-e8de392e496f\") " pod="openshift-multus/multus-additional-cni-plugins-dmzql"
May 25 11:17:09.185860 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.185180    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a3d75818-4ca3-4bb1-a3dd-28ae4ce71d09-cnibin\") pod \"multus-bxzz6\" (UID: \"a3d75818-4ca3-4bb1-a3dd-28ae4ce71d09\") " pod="openshift-multus/multus-bxzz6"
May 25 11:17:09.185860 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.185257    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7c97daaa-b9c1-4ee0-ade1-e8de392e496f-cni-binary-copy\") pod \"multus-additional-cni-plugins-dmzql\" (UID: \"7c97daaa-b9c1-4ee0-ade1-e8de392e496f\") " pod="openshift-multus/multus-additional-cni-plugins-dmzql"
May 25 11:17:09.185860 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.185259    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a3d75818-4ca3-4bb1-a3dd-28ae4ce71d09-os-release\") pod \"multus-bxzz6\" (UID: \"a3d75818-4ca3-4bb1-a3dd-28ae4ce71d09\") " pod="openshift-multus/multus-bxzz6"
May 25 11:17:09.185860 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.185327    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a3d75818-4ca3-4bb1-a3dd-28ae4ce71d09-host-var-lib-kubelet\") pod \"multus-bxzz6\" (UID: \"a3d75818-4ca3-4bb1-a3dd-28ae4ce71d09\") " pod="openshift-multus/multus-bxzz6"
May 25 11:17:09.185860 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.185346    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a3d75818-4ca3-4bb1-a3dd-28ae4ce71d09-os-release\") pod \"multus-bxzz6\" (UID: \"a3d75818-4ca3-4bb1-a3dd-28ae4ce71d09\") " pod="openshift-multus/multus-bxzz6"
May 25 11:17:09.185860 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.185360    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7c97daaa-b9c1-4ee0-ade1-e8de392e496f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-dmzql\" (UID: \"7c97daaa-b9c1-4ee0-ade1-e8de392e496f\") " pod="openshift-multus/multus-additional-cni-plugins-dmzql"
May 25 11:17:09.185860 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.185382    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b55f6df2-1dbc-4fc5-a4e6-e679d289e9f5-tmp-dir\") pod \"node-resolver-xvcpl\" (UID: \"b55f6df2-1dbc-4fc5-a4e6-e679d289e9f5\") " pod="openshift-dns/node-resolver-xvcpl"
May 25 11:17:09.185860 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.185397    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ae10dbbc-8226-48df-a0f7-39aed3d95c58-konnectivity-ca\") pod \"konnectivity-agent-mr48p\" (UID: \"ae10dbbc-8226-48df-a0f7-39aed3d95c58\") " pod="kube-system/konnectivity-agent-mr48p"
May 25 11:17:09.185860 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.185398    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-nbhnn\" (UniqueName: \"kubernetes.io/projected/2b05b78f-6ab1-4f94-8273-e5fa70b78d15-kube-api-access-nbhnn\") pod \"network-check-target-rl7hd\" (UID: \"2b05b78f-6ab1-4f94-8273-e5fa70b78d15\") " pod="openshift-network-diagnostics/network-check-target-rl7hd"
May 25 11:17:09.185860 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.185384    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a3d75818-4ca3-4bb1-a3dd-28ae4ce71d09-host-var-lib-kubelet\") pod \"multus-bxzz6\" (UID: \"a3d75818-4ca3-4bb1-a3dd-28ae4ce71d09\") " pod="openshift-multus/multus-bxzz6"
May 25 11:17:09.185860 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.185445    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a3d75818-4ca3-4bb1-a3dd-28ae4ce71d09-multus-conf-dir\") pod \"multus-bxzz6\" (UID: \"a3d75818-4ca3-4bb1-a3dd-28ae4ce71d09\") " pod="openshift-multus/multus-bxzz6"
May 25 11:17:09.185860 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.185473    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-jp8vr\" (UniqueName: \"kubernetes.io/projected/b55f6df2-1dbc-4fc5-a4e6-e679d289e9f5-kube-api-access-jp8vr\") pod \"node-resolver-xvcpl\" (UID: \"b55f6df2-1dbc-4fc5-a4e6-e679d289e9f5\") " pod="openshift-dns/node-resolver-xvcpl"
May 25 11:17:09.185860 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.185502    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a3d75818-4ca3-4bb1-a3dd-28ae4ce71d09-host-run-netns\") pod \"multus-bxzz6\" (UID: \"a3d75818-4ca3-4bb1-a3dd-28ae4ce71d09\") " pod="openshift-multus/multus-bxzz6"
May 25 11:17:09.185860 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.185505    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a3d75818-4ca3-4bb1-a3dd-28ae4ce71d09-multus-conf-dir\") pod \"multus-bxzz6\" (UID: \"a3d75818-4ca3-4bb1-a3dd-28ae4ce71d09\") " pod="openshift-multus/multus-bxzz6"
May 25 11:17:09.185860 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.185527    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a3d75818-4ca3-4bb1-a3dd-28ae4ce71d09-etc-kubernetes\") pod \"multus-bxzz6\" (UID: \"a3d75818-4ca3-4bb1-a3dd-28ae4ce71d09\") " pod="openshift-multus/multus-bxzz6"
May 25 11:17:09.185860 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.185554    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a3d75818-4ca3-4bb1-a3dd-28ae4ce71d09-multus-cni-dir\") pod \"multus-bxzz6\" (UID: \"a3d75818-4ca3-4bb1-a3dd-28ae4ce71d09\") " pod="openshift-multus/multus-bxzz6"
May 25 11:17:09.185860 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.185563    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a3d75818-4ca3-4bb1-a3dd-28ae4ce71d09-host-run-netns\") pod \"multus-bxzz6\" (UID: \"a3d75818-4ca3-4bb1-a3dd-28ae4ce71d09\") " pod="openshift-multus/multus-bxzz6"
May 25 11:17:09.185860 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.185579    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a3d75818-4ca3-4bb1-a3dd-28ae4ce71d09-host-run-k8s-cni-cncf-io\") pod \"multus-bxzz6\" (UID: \"a3d75818-4ca3-4bb1-a3dd-28ae4ce71d09\") " pod="openshift-multus/multus-bxzz6"
May 25 11:17:09.186424 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.185606    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a3d75818-4ca3-4bb1-a3dd-28ae4ce71d09-cni-binary-copy\") pod \"multus-bxzz6\" (UID: \"a3d75818-4ca3-4bb1-a3dd-28ae4ce71d09\") " pod="openshift-multus/multus-bxzz6"
May 25 11:17:09.186424 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.185618    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a3d75818-4ca3-4bb1-a3dd-28ae4ce71d09-etc-kubernetes\") pod \"multus-bxzz6\" (UID: \"a3d75818-4ca3-4bb1-a3dd-28ae4ce71d09\") " pod="openshift-multus/multus-bxzz6"
May 25 11:17:09.186424 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.185630    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ead20bcd-df93-4d45-b5bb-470a8fe373e7-metrics-certs\") pod \"network-metrics-daemon-444mb\" (UID: \"ead20bcd-df93-4d45-b5bb-470a8fe373e7\") " pod="openshift-multus/network-metrics-daemon-444mb"
May 25 11:17:09.186424 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.185634    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a3d75818-4ca3-4bb1-a3dd-28ae4ce71d09-host-run-k8s-cni-cncf-io\") pod \"multus-bxzz6\" (UID: \"a3d75818-4ca3-4bb1-a3dd-28ae4ce71d09\") " pod="openshift-multus/multus-bxzz6"
May 25 11:17:09.186424 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.185703    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a3d75818-4ca3-4bb1-a3dd-28ae4ce71d09-multus-cni-dir\") pod \"multus-bxzz6\" (UID: \"a3d75818-4ca3-4bb1-a3dd-28ae4ce71d09\") " pod="openshift-multus/multus-bxzz6"
May 25 11:17:09.186424 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.185737    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7c97daaa-b9c1-4ee0-ade1-e8de392e496f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-dmzql\" (UID: \"7c97daaa-b9c1-4ee0-ade1-e8de392e496f\") " pod="openshift-multus/multus-additional-cni-plugins-dmzql"
May 25 11:17:09.186424 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:09.185741    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 11:17:09.186424 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:09.185915    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ead20bcd-df93-4d45-b5bb-470a8fe373e7-metrics-certs podName:ead20bcd-df93-4d45-b5bb-470a8fe373e7 nodeName:}" failed. No retries permitted until 2026-05-25 11:17:09.685872371 +0000 UTC m=+3.231049972 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ead20bcd-df93-4d45-b5bb-470a8fe373e7-metrics-certs") pod "network-metrics-daemon-444mb" (UID: "ead20bcd-df93-4d45-b5bb-470a8fe373e7") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 11:17:09.186424 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.186071    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a3d75818-4ca3-4bb1-a3dd-28ae4ce71d09-cni-binary-copy\") pod \"multus-bxzz6\" (UID: \"a3d75818-4ca3-4bb1-a3dd-28ae4ce71d09\") " pod="openshift-multus/multus-bxzz6"
May 25 11:17:09.186764 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.186751    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ae10dbbc-8226-48df-a0f7-39aed3d95c58-agent-certs\") pod \"konnectivity-agent-mr48p\" (UID: \"ae10dbbc-8226-48df-a0f7-39aed3d95c58\") " pod="kube-system/konnectivity-agent-mr48p"
May 25 11:17:09.197689 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.197660    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8c6r8\" (UniqueName: \"kubernetes.io/projected/a3d75818-4ca3-4bb1-a3dd-28ae4ce71d09-kube-api-access-8c6r8\") pod \"multus-bxzz6\" (UID: \"a3d75818-4ca3-4bb1-a3dd-28ae4ce71d09\") " pod="openshift-multus/multus-bxzz6"
May 25 11:17:09.197842 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.197815    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6wpk\" (UniqueName: \"kubernetes.io/projected/7c97daaa-b9c1-4ee0-ade1-e8de392e496f-kube-api-access-g6wpk\") pod \"multus-additional-cni-plugins-dmzql\" (UID: \"7c97daaa-b9c1-4ee0-ade1-e8de392e496f\") " pod="openshift-multus/multus-additional-cni-plugins-dmzql"
May 25 11:17:09.208535 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:09.208518    2561 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 25 11:17:09.208535 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:09.208536    2561 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 25 11:17:09.208648 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:09.208546    2561 projected.go:194] Error preparing data for projected volume kube-api-access-nbhnn for pod openshift-network-diagnostics/network-check-target-rl7hd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 11:17:09.208648 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:09.208595    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2b05b78f-6ab1-4f94-8273-e5fa70b78d15-kube-api-access-nbhnn podName:2b05b78f-6ab1-4f94-8273-e5fa70b78d15 nodeName:}" failed. No retries permitted until 2026-05-25 11:17:09.708581327 +0000 UTC m=+3.253758924 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-nbhnn" (UniqueName: "kubernetes.io/projected/2b05b78f-6ab1-4f94-8273-e5fa70b78d15-kube-api-access-nbhnn") pod "network-check-target-rl7hd" (UID: "2b05b78f-6ab1-4f94-8273-e5fa70b78d15") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 11:17:09.211405 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.211384    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jp8vr\" (UniqueName: \"kubernetes.io/projected/b55f6df2-1dbc-4fc5-a4e6-e679d289e9f5-kube-api-access-jp8vr\") pod \"node-resolver-xvcpl\" (UID: \"b55f6df2-1dbc-4fc5-a4e6-e679d289e9f5\") " pod="openshift-dns/node-resolver-xvcpl"
May 25 11:17:09.211738 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.211722    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6m6wg\" (UniqueName: \"kubernetes.io/projected/ead20bcd-df93-4d45-b5bb-470a8fe373e7-kube-api-access-6m6wg\") pod \"network-metrics-daemon-444mb\" (UID: \"ead20bcd-df93-4d45-b5bb-470a8fe373e7\") " pod="openshift-multus/network-metrics-daemon-444mb"
May 25 11:17:09.272478 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.272452    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-thvg5"
May 25 11:17:09.280161 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.280137    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-4xz76"
May 25 11:17:09.289443 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.289419    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-k2fsn"
May 25 11:17:09.296070 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.296052    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5ldrp"
May 25 11:17:09.302658 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.302632    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-q8ng6"
May 25 11:17:09.309244 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.309228    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-mr48p"
May 25 11:17:09.317751 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.317735    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-xvcpl"
May 25 11:17:09.323234 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.323203    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-dmzql"
May 25 11:17:09.328768 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.328751    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-bxzz6"
May 25 11:17:09.689764 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.689735    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ead20bcd-df93-4d45-b5bb-470a8fe373e7-metrics-certs\") pod \"network-metrics-daemon-444mb\" (UID: \"ead20bcd-df93-4d45-b5bb-470a8fe373e7\") " pod="openshift-multus/network-metrics-daemon-444mb"
May 25 11:17:09.689907 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:09.689878    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 11:17:09.689976 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:09.689940    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ead20bcd-df93-4d45-b5bb-470a8fe373e7-metrics-certs podName:ead20bcd-df93-4d45-b5bb-470a8fe373e7 nodeName:}" failed. No retries permitted until 2026-05-25 11:17:10.689925115 +0000 UTC m=+4.235102696 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ead20bcd-df93-4d45-b5bb-470a8fe373e7-metrics-certs") pod "network-metrics-daemon-444mb" (UID: "ead20bcd-df93-4d45-b5bb-470a8fe373e7") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 11:17:09.719617 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:09.719591    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b2bf5ae_a8a4_48e9_9afc_ee576988791e.slice/crio-210853d5f436b0aeb1ddfccdc4e047cc5204b86898f671569537ca4ababf5aed WatchSource:0}: Error finding container 210853d5f436b0aeb1ddfccdc4e047cc5204b86898f671569537ca4ababf5aed: Status 404 returned error can't find the container with id 210853d5f436b0aeb1ddfccdc4e047cc5204b86898f671569537ca4ababf5aed
May 25 11:17:09.720793 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:09.720767    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae10dbbc_8226_48df_a0f7_39aed3d95c58.slice/crio-6850d138f6deb0b73c1627544f3a3903b75d5af3624ecfa9af55300c312a2d2f WatchSource:0}: Error finding container 6850d138f6deb0b73c1627544f3a3903b75d5af3624ecfa9af55300c312a2d2f: Status 404 returned error can't find the container with id 6850d138f6deb0b73c1627544f3a3903b75d5af3624ecfa9af55300c312a2d2f
May 25 11:17:09.723811 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:09.723790    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8a3a45a_3dba_4919_895a_ef467fbd4343.slice/crio-615dfe994fbe9daba4848a848e01878bdf297711c08a379154c5cdac620af7b2 WatchSource:0}: Error finding container 615dfe994fbe9daba4848a848e01878bdf297711c08a379154c5cdac620af7b2: Status 404 returned error can't find the container with id 615dfe994fbe9daba4848a848e01878bdf297711c08a379154c5cdac620af7b2
May 25 11:17:09.724305 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:09.724286    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3d75818_4ca3_4bb1_a3dd_28ae4ce71d09.slice/crio-f340dab568dc0523e539fce7396d535a0814da4953c6d3c0f22d44350485edbc WatchSource:0}: Error finding container f340dab568dc0523e539fce7396d535a0814da4953c6d3c0f22d44350485edbc: Status 404 returned error can't find the container with id f340dab568dc0523e539fce7396d535a0814da4953c6d3c0f22d44350485edbc
May 25 11:17:09.725849 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:09.725792    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a250955_746b_416a_b798_283a0ea46864.slice/crio-a98ead4fd88a1211f62c5ebe167bf08e59e630b98443765e561118a84386f7e6 WatchSource:0}: Error finding container a98ead4fd88a1211f62c5ebe167bf08e59e630b98443765e561118a84386f7e6: Status 404 returned error can't find the container with id a98ead4fd88a1211f62c5ebe167bf08e59e630b98443765e561118a84386f7e6
May 25 11:17:09.726326 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:09.726197    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0e8a1a4_c0cb_443e_8c38_9abcf242fc84.slice/crio-e913065937d8e6ddc24302a1a7138d76cb284d9ea4db7e07a9a962a2b8689f86 WatchSource:0}: Error finding container e913065937d8e6ddc24302a1a7138d76cb284d9ea4db7e07a9a962a2b8689f86: Status 404 returned error can't find the container with id e913065937d8e6ddc24302a1a7138d76cb284d9ea4db7e07a9a962a2b8689f86
May 25 11:17:09.727062 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:09.727036    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71885334_70ca_483b_8bd9_fdbfecb77949.slice/crio-ede3691a14840884aa836e96bba13e9a7390b87fb72d966ba28a3ce1d7a907d1 WatchSource:0}: Error finding container ede3691a14840884aa836e96bba13e9a7390b87fb72d966ba28a3ce1d7a907d1: Status 404 returned error can't find the container with id ede3691a14840884aa836e96bba13e9a7390b87fb72d966ba28a3ce1d7a907d1
May 25 11:17:09.790539 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:09.790438    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-nbhnn\" (UniqueName: \"kubernetes.io/projected/2b05b78f-6ab1-4f94-8273-e5fa70b78d15-kube-api-access-nbhnn\") pod \"network-check-target-rl7hd\" (UID: \"2b05b78f-6ab1-4f94-8273-e5fa70b78d15\") " pod="openshift-network-diagnostics/network-check-target-rl7hd"
May 25 11:17:09.790631 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:09.790582    2561 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 25 11:17:09.790631 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:09.790600    2561 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 25 11:17:09.790631 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:09.790610    2561 projected.go:194] Error preparing data for projected volume kube-api-access-nbhnn for pod openshift-network-diagnostics/network-check-target-rl7hd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 11:17:09.790749 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:09.790652    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2b05b78f-6ab1-4f94-8273-e5fa70b78d15-kube-api-access-nbhnn podName:2b05b78f-6ab1-4f94-8273-e5fa70b78d15 nodeName:}" failed. No retries permitted until 2026-05-25 11:17:10.790638981 +0000 UTC m=+4.335816566 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-nbhnn" (UniqueName: "kubernetes.io/projected/2b05b78f-6ab1-4f94-8273-e5fa70b78d15-kube-api-access-nbhnn") pod "network-check-target-rl7hd" (UID: "2b05b78f-6ab1-4f94-8273-e5fa70b78d15") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 11:17:10.114720 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:10.114657    2561 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-05-24 11:12:08 +0000 UTC" deadline="2027-11-01 21:21:02.064939439 +0000 UTC"
May 25 11:17:10.114720 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:10.114693    2561 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12610h3m51.950250551s"
May 25 11:17:10.121352 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:10.121320    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-bxzz6" event={"ID":"a3d75818-4ca3-4bb1-a3dd-28ae4ce71d09","Type":"ContainerStarted","Data":"f340dab568dc0523e539fce7396d535a0814da4953c6d3c0f22d44350485edbc"}
May 25 11:17:10.122381 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:10.122359    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-xvcpl" event={"ID":"b55f6df2-1dbc-4fc5-a4e6-e679d289e9f5","Type":"ContainerStarted","Data":"3ecdfeba9ff8bfa5facab4030b5b3be3c741b19966177ce1207f645519e9800b"}
May 25 11:17:10.124188 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:10.124162    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dmzql" event={"ID":"7c97daaa-b9c1-4ee0-ade1-e8de392e496f","Type":"ContainerStarted","Data":"4c33dcde88b27a5f00e0e6d91e302ee903a64669e420b661d079411e75d2369e"}
May 25 11:17:10.126148 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:10.126125    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-mr48p" event={"ID":"ae10dbbc-8226-48df-a0f7-39aed3d95c58","Type":"ContainerStarted","Data":"6850d138f6deb0b73c1627544f3a3903b75d5af3624ecfa9af55300c312a2d2f"}
May 25 11:17:10.128717 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:10.128694    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-thvg5" event={"ID":"5b2bf5ae-a8a4-48e9-9afc-ee576988791e","Type":"ContainerStarted","Data":"210853d5f436b0aeb1ddfccdc4e047cc5204b86898f671569537ca4ababf5aed"}
May 25 11:17:10.147301 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:10.146568    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-227.ec2.internal" event={"ID":"f96ecd3e8e97719fd70e0de354f8eb4d","Type":"ContainerStarted","Data":"2f6ce776ece8ddfe8a723b6e0eb82e20a0e079d10659414d25d2bd58b9284163"}
May 25 11:17:10.158621 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:10.158573    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5ldrp" event={"ID":"71885334-70ca-483b-8bd9-fdbfecb77949","Type":"ContainerStarted","Data":"ede3691a14840884aa836e96bba13e9a7390b87fb72d966ba28a3ce1d7a907d1"}
May 25 11:17:10.160393 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:10.160350    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-k2fsn" event={"ID":"b0e8a1a4-c0cb-443e-8c38-9abcf242fc84","Type":"ContainerStarted","Data":"e913065937d8e6ddc24302a1a7138d76cb284d9ea4db7e07a9a962a2b8689f86"}
May 25 11:17:10.172644 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:10.172453    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-q8ng6" event={"ID":"9a250955-746b-416a-b798-283a0ea46864","Type":"ContainerStarted","Data":"a98ead4fd88a1211f62c5ebe167bf08e59e630b98443765e561118a84386f7e6"}
May 25 11:17:10.181503 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:10.181482    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-4xz76" event={"ID":"a8a3a45a-3dba-4919-895a-ef467fbd4343","Type":"ContainerStarted","Data":"615dfe994fbe9daba4848a848e01878bdf297711c08a379154c5cdac620af7b2"}
May 25 11:17:10.186279 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:10.186215    2561 generic.go:358] "Generic (PLEG): container finished" podID="17c74a3057617237e14ccfbc812fc82f" containerID="3fd494f9cc50dca7be6586a51a99da5a914622009dae74b396b365ae93576f43" exitCode=0
May 25 11:17:10.186279 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:10.186256    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-227.ec2.internal" event={"ID":"17c74a3057617237e14ccfbc812fc82f","Type":"ContainerDied","Data":"3fd494f9cc50dca7be6586a51a99da5a914622009dae74b396b365ae93576f43"}
May 25 11:17:10.201923 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:10.201873    2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-227.ec2.internal" podStartSLOduration=2.201859164 podStartE2EDuration="2.201859164s" podCreationTimestamp="2026-05-25 11:17:08 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 11:17:10.163191633 +0000 UTC m=+3.708369236" watchObservedRunningTime="2026-05-25 11:17:10.201859164 +0000 UTC m=+3.747036765"
May 25 11:17:10.697332 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:10.697235    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ead20bcd-df93-4d45-b5bb-470a8fe373e7-metrics-certs\") pod \"network-metrics-daemon-444mb\" (UID: \"ead20bcd-df93-4d45-b5bb-470a8fe373e7\") " pod="openshift-multus/network-metrics-daemon-444mb"
May 25 11:17:10.697495 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:10.697409    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 11:17:10.697495 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:10.697473    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ead20bcd-df93-4d45-b5bb-470a8fe373e7-metrics-certs podName:ead20bcd-df93-4d45-b5bb-470a8fe373e7 nodeName:}" failed. No retries permitted until 2026-05-25 11:17:12.697455349 +0000 UTC m=+6.242632945 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ead20bcd-df93-4d45-b5bb-470a8fe373e7-metrics-certs") pod "network-metrics-daemon-444mb" (UID: "ead20bcd-df93-4d45-b5bb-470a8fe373e7") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 11:17:10.798194 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:10.798157    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-nbhnn\" (UniqueName: \"kubernetes.io/projected/2b05b78f-6ab1-4f94-8273-e5fa70b78d15-kube-api-access-nbhnn\") pod \"network-check-target-rl7hd\" (UID: \"2b05b78f-6ab1-4f94-8273-e5fa70b78d15\") " pod="openshift-network-diagnostics/network-check-target-rl7hd"
May 25 11:17:10.798393 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:10.798366    2561 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 25 11:17:10.798393 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:10.798385    2561 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 25 11:17:10.798496 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:10.798397    2561 projected.go:194] Error preparing data for projected volume kube-api-access-nbhnn for pod openshift-network-diagnostics/network-check-target-rl7hd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 11:17:10.798496 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:10.798455    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2b05b78f-6ab1-4f94-8273-e5fa70b78d15-kube-api-access-nbhnn podName:2b05b78f-6ab1-4f94-8273-e5fa70b78d15 nodeName:}" failed. No retries permitted until 2026-05-25 11:17:12.798436662 +0000 UTC m=+6.343614262 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-nbhnn" (UniqueName: "kubernetes.io/projected/2b05b78f-6ab1-4f94-8273-e5fa70b78d15-kube-api-access-nbhnn") pod "network-check-target-rl7hd" (UID: "2b05b78f-6ab1-4f94-8273-e5fa70b78d15") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 11:17:11.112758 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:11.112603    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-444mb"
May 25 11:17:11.112758 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:11.112734    2561 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-444mb" podUID="ead20bcd-df93-4d45-b5bb-470a8fe373e7"
May 25 11:17:11.112969 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:11.112828    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rl7hd"
May 25 11:17:11.112969 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:11.112890    2561 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rl7hd" podUID="2b05b78f-6ab1-4f94-8273-e5fa70b78d15"
May 25 11:17:11.212448 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:11.212415    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-227.ec2.internal" event={"ID":"17c74a3057617237e14ccfbc812fc82f","Type":"ContainerStarted","Data":"6d949f4826a5456b7c3596008f94282096d38a1a4224ef3f230cc23aef27dc98"}
May 25 11:17:12.715969 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:12.715356    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ead20bcd-df93-4d45-b5bb-470a8fe373e7-metrics-certs\") pod \"network-metrics-daemon-444mb\" (UID: \"ead20bcd-df93-4d45-b5bb-470a8fe373e7\") " pod="openshift-multus/network-metrics-daemon-444mb"
May 25 11:17:12.715969 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:12.715500    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 11:17:12.715969 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:12.715559    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ead20bcd-df93-4d45-b5bb-470a8fe373e7-metrics-certs podName:ead20bcd-df93-4d45-b5bb-470a8fe373e7 nodeName:}" failed. No retries permitted until 2026-05-25 11:17:16.715540447 +0000 UTC m=+10.260718034 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ead20bcd-df93-4d45-b5bb-470a8fe373e7-metrics-certs") pod "network-metrics-daemon-444mb" (UID: "ead20bcd-df93-4d45-b5bb-470a8fe373e7") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 11:17:12.816717 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:12.816161    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-nbhnn\" (UniqueName: \"kubernetes.io/projected/2b05b78f-6ab1-4f94-8273-e5fa70b78d15-kube-api-access-nbhnn\") pod \"network-check-target-rl7hd\" (UID: \"2b05b78f-6ab1-4f94-8273-e5fa70b78d15\") " pod="openshift-network-diagnostics/network-check-target-rl7hd"
May 25 11:17:12.816717 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:12.816333    2561 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 25 11:17:12.816717 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:12.816357    2561 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 25 11:17:12.816717 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:12.816370    2561 projected.go:194] Error preparing data for projected volume kube-api-access-nbhnn for pod openshift-network-diagnostics/network-check-target-rl7hd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 11:17:12.816717 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:12.816429    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2b05b78f-6ab1-4f94-8273-e5fa70b78d15-kube-api-access-nbhnn podName:2b05b78f-6ab1-4f94-8273-e5fa70b78d15 nodeName:}" failed. No retries permitted until 2026-05-25 11:17:16.816409622 +0000 UTC m=+10.361587222 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-nbhnn" (UniqueName: "kubernetes.io/projected/2b05b78f-6ab1-4f94-8273-e5fa70b78d15-kube-api-access-nbhnn") pod "network-check-target-rl7hd" (UID: "2b05b78f-6ab1-4f94-8273-e5fa70b78d15") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 11:17:12.874308 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:12.872877    2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-227.ec2.internal" podStartSLOduration=4.872858002 podStartE2EDuration="4.872858002s" podCreationTimestamp="2026-05-25 11:17:08 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 11:17:11.2343305 +0000 UTC m=+4.779508105" watchObservedRunningTime="2026-05-25 11:17:12.872858002 +0000 UTC m=+6.418035607"
May 25 11:17:12.874308 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:12.873625    2561 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-dq8s4"]
May 25 11:17:12.878023 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:12.877657    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dq8s4"
May 25 11:17:12.878023 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:12.877725    2561 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dq8s4" podUID="3e92c2a7-1021-4ee8-aded-8ae5ddc6b81c"
May 25 11:17:12.917283 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:12.917235    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3e92c2a7-1021-4ee8-aded-8ae5ddc6b81c-original-pull-secret\") pod \"global-pull-secret-syncer-dq8s4\" (UID: \"3e92c2a7-1021-4ee8-aded-8ae5ddc6b81c\") " pod="kube-system/global-pull-secret-syncer-dq8s4"
May 25 11:17:12.917408 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:12.917362    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3e92c2a7-1021-4ee8-aded-8ae5ddc6b81c-dbus\") pod \"global-pull-secret-syncer-dq8s4\" (UID: \"3e92c2a7-1021-4ee8-aded-8ae5ddc6b81c\") " pod="kube-system/global-pull-secret-syncer-dq8s4"
May 25 11:17:12.917478 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:12.917421    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3e92c2a7-1021-4ee8-aded-8ae5ddc6b81c-kubelet-config\") pod \"global-pull-secret-syncer-dq8s4\" (UID: \"3e92c2a7-1021-4ee8-aded-8ae5ddc6b81c\") " pod="kube-system/global-pull-secret-syncer-dq8s4"
May 25 11:17:13.018315 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:13.018216    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3e92c2a7-1021-4ee8-aded-8ae5ddc6b81c-dbus\") pod \"global-pull-secret-syncer-dq8s4\" (UID: \"3e92c2a7-1021-4ee8-aded-8ae5ddc6b81c\") " pod="kube-system/global-pull-secret-syncer-dq8s4"
May 25 11:17:13.018315 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:13.018300    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3e92c2a7-1021-4ee8-aded-8ae5ddc6b81c-kubelet-config\") pod \"global-pull-secret-syncer-dq8s4\" (UID: \"3e92c2a7-1021-4ee8-aded-8ae5ddc6b81c\") " pod="kube-system/global-pull-secret-syncer-dq8s4"
May 25 11:17:13.018474 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:13.018349    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3e92c2a7-1021-4ee8-aded-8ae5ddc6b81c-original-pull-secret\") pod \"global-pull-secret-syncer-dq8s4\" (UID: \"3e92c2a7-1021-4ee8-aded-8ae5ddc6b81c\") " pod="kube-system/global-pull-secret-syncer-dq8s4"
May 25 11:17:13.018474 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:13.018457    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
May 25 11:17:13.018547 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:13.018515    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3e92c2a7-1021-4ee8-aded-8ae5ddc6b81c-original-pull-secret podName:3e92c2a7-1021-4ee8-aded-8ae5ddc6b81c nodeName:}" failed. No retries permitted until 2026-05-25 11:17:13.518496558 +0000 UTC m=+7.063674154 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3e92c2a7-1021-4ee8-aded-8ae5ddc6b81c-original-pull-secret") pod "global-pull-secret-syncer-dq8s4" (UID: "3e92c2a7-1021-4ee8-aded-8ae5ddc6b81c") : object "kube-system"/"original-pull-secret" not registered
May 25 11:17:13.018982 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:13.018821    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3e92c2a7-1021-4ee8-aded-8ae5ddc6b81c-dbus\") pod \"global-pull-secret-syncer-dq8s4\" (UID: \"3e92c2a7-1021-4ee8-aded-8ae5ddc6b81c\") " pod="kube-system/global-pull-secret-syncer-dq8s4"
May 25 11:17:13.018982 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:13.018887    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3e92c2a7-1021-4ee8-aded-8ae5ddc6b81c-kubelet-config\") pod \"global-pull-secret-syncer-dq8s4\" (UID: \"3e92c2a7-1021-4ee8-aded-8ae5ddc6b81c\") " pod="kube-system/global-pull-secret-syncer-dq8s4"
May 25 11:17:13.116715 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:13.116689    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-444mb"
May 25 11:17:13.116864 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:13.116817    2561 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-444mb" podUID="ead20bcd-df93-4d45-b5bb-470a8fe373e7"
May 25 11:17:13.116946 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:13.116898    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rl7hd"
May 25 11:17:13.116996 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:13.116969    2561 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rl7hd" podUID="2b05b78f-6ab1-4f94-8273-e5fa70b78d15"
May 25 11:17:13.522439 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:13.522405    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3e92c2a7-1021-4ee8-aded-8ae5ddc6b81c-original-pull-secret\") pod \"global-pull-secret-syncer-dq8s4\" (UID: \"3e92c2a7-1021-4ee8-aded-8ae5ddc6b81c\") " pod="kube-system/global-pull-secret-syncer-dq8s4"
May 25 11:17:13.522606 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:13.522545    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
May 25 11:17:13.522677 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:13.522621    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3e92c2a7-1021-4ee8-aded-8ae5ddc6b81c-original-pull-secret podName:3e92c2a7-1021-4ee8-aded-8ae5ddc6b81c nodeName:}" failed. No retries permitted until 2026-05-25 11:17:14.522601207 +0000 UTC m=+8.067778793 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3e92c2a7-1021-4ee8-aded-8ae5ddc6b81c-original-pull-secret") pod "global-pull-secret-syncer-dq8s4" (UID: "3e92c2a7-1021-4ee8-aded-8ae5ddc6b81c") : object "kube-system"/"original-pull-secret" not registered
May 25 11:17:14.112995 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:14.112962    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dq8s4"
May 25 11:17:14.113515 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:14.113104    2561 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dq8s4" podUID="3e92c2a7-1021-4ee8-aded-8ae5ddc6b81c"
May 25 11:17:14.532067 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:14.531988    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3e92c2a7-1021-4ee8-aded-8ae5ddc6b81c-original-pull-secret\") pod \"global-pull-secret-syncer-dq8s4\" (UID: \"3e92c2a7-1021-4ee8-aded-8ae5ddc6b81c\") " pod="kube-system/global-pull-secret-syncer-dq8s4"
May 25 11:17:14.532213 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:14.532130    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
May 25 11:17:14.532213 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:14.532192    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3e92c2a7-1021-4ee8-aded-8ae5ddc6b81c-original-pull-secret podName:3e92c2a7-1021-4ee8-aded-8ae5ddc6b81c nodeName:}" failed. No retries permitted until 2026-05-25 11:17:16.53217365 +0000 UTC m=+10.077351231 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3e92c2a7-1021-4ee8-aded-8ae5ddc6b81c-original-pull-secret") pod "global-pull-secret-syncer-dq8s4" (UID: "3e92c2a7-1021-4ee8-aded-8ae5ddc6b81c") : object "kube-system"/"original-pull-secret" not registered
May 25 11:17:15.115785 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:15.115755    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rl7hd"
May 25 11:17:15.116203 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:15.115798    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-444mb"
May 25 11:17:15.116203 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:15.115873    2561 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rl7hd" podUID="2b05b78f-6ab1-4f94-8273-e5fa70b78d15"
May 25 11:17:15.116335 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:15.116263    2561 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-444mb" podUID="ead20bcd-df93-4d45-b5bb-470a8fe373e7"
May 25 11:17:16.112400 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:16.112367    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dq8s4"
May 25 11:17:16.112586 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:16.112501    2561 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dq8s4" podUID="3e92c2a7-1021-4ee8-aded-8ae5ddc6b81c"
May 25 11:17:16.547055 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:16.546967    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3e92c2a7-1021-4ee8-aded-8ae5ddc6b81c-original-pull-secret\") pod \"global-pull-secret-syncer-dq8s4\" (UID: \"3e92c2a7-1021-4ee8-aded-8ae5ddc6b81c\") " pod="kube-system/global-pull-secret-syncer-dq8s4"
May 25 11:17:16.547496 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:16.547103    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
May 25 11:17:16.547496 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:16.547161    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3e92c2a7-1021-4ee8-aded-8ae5ddc6b81c-original-pull-secret podName:3e92c2a7-1021-4ee8-aded-8ae5ddc6b81c nodeName:}" failed. No retries permitted until 2026-05-25 11:17:20.547143486 +0000 UTC m=+14.092321073 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3e92c2a7-1021-4ee8-aded-8ae5ddc6b81c-original-pull-secret") pod "global-pull-secret-syncer-dq8s4" (UID: "3e92c2a7-1021-4ee8-aded-8ae5ddc6b81c") : object "kube-system"/"original-pull-secret" not registered
May 25 11:17:16.749406 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:16.749301    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ead20bcd-df93-4d45-b5bb-470a8fe373e7-metrics-certs\") pod \"network-metrics-daemon-444mb\" (UID: \"ead20bcd-df93-4d45-b5bb-470a8fe373e7\") " pod="openshift-multus/network-metrics-daemon-444mb"
May 25 11:17:16.749568 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:16.749498    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 11:17:16.749568 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:16.749565    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ead20bcd-df93-4d45-b5bb-470a8fe373e7-metrics-certs podName:ead20bcd-df93-4d45-b5bb-470a8fe373e7 nodeName:}" failed. No retries permitted until 2026-05-25 11:17:24.749546805 +0000 UTC m=+18.294724401 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ead20bcd-df93-4d45-b5bb-470a8fe373e7-metrics-certs") pod "network-metrics-daemon-444mb" (UID: "ead20bcd-df93-4d45-b5bb-470a8fe373e7") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 11:17:16.850674 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:16.850591    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-nbhnn\" (UniqueName: \"kubernetes.io/projected/2b05b78f-6ab1-4f94-8273-e5fa70b78d15-kube-api-access-nbhnn\") pod \"network-check-target-rl7hd\" (UID: \"2b05b78f-6ab1-4f94-8273-e5fa70b78d15\") " pod="openshift-network-diagnostics/network-check-target-rl7hd"
May 25 11:17:16.850826 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:16.850768    2561 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 25 11:17:16.850826 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:16.850790    2561 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 25 11:17:16.850826 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:16.850802    2561 projected.go:194] Error preparing data for projected volume kube-api-access-nbhnn for pod openshift-network-diagnostics/network-check-target-rl7hd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 11:17:16.850989 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:16.850867    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2b05b78f-6ab1-4f94-8273-e5fa70b78d15-kube-api-access-nbhnn podName:2b05b78f-6ab1-4f94-8273-e5fa70b78d15 nodeName:}" failed. No retries permitted until 2026-05-25 11:17:24.850846962 +0000 UTC m=+18.396024561 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-nbhnn" (UniqueName: "kubernetes.io/projected/2b05b78f-6ab1-4f94-8273-e5fa70b78d15-kube-api-access-nbhnn") pod "network-check-target-rl7hd" (UID: "2b05b78f-6ab1-4f94-8273-e5fa70b78d15") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 11:17:17.116054 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:17.115973    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-444mb"
May 25 11:17:17.116240 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:17.115977    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rl7hd"
May 25 11:17:17.116240 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:17.116105    2561 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-444mb" podUID="ead20bcd-df93-4d45-b5bb-470a8fe373e7"
May 25 11:17:17.116240 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:17.116206    2561 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rl7hd" podUID="2b05b78f-6ab1-4f94-8273-e5fa70b78d15"
May 25 11:17:18.112881 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:18.112849    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dq8s4"
May 25 11:17:18.113351 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:18.112971    2561 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dq8s4" podUID="3e92c2a7-1021-4ee8-aded-8ae5ddc6b81c"
May 25 11:17:19.112633 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:19.112580    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-444mb"
May 25 11:17:19.112810 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:19.112719    2561 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-444mb" podUID="ead20bcd-df93-4d45-b5bb-470a8fe373e7"
May 25 11:17:19.112810 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:19.112763    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rl7hd"
May 25 11:17:19.112919 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:19.112863    2561 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rl7hd" podUID="2b05b78f-6ab1-4f94-8273-e5fa70b78d15"
May 25 11:17:20.112973 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:20.112887    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dq8s4"
May 25 11:17:20.113374 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:20.112994    2561 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dq8s4" podUID="3e92c2a7-1021-4ee8-aded-8ae5ddc6b81c"
May 25 11:17:20.579577 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:20.579536    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3e92c2a7-1021-4ee8-aded-8ae5ddc6b81c-original-pull-secret\") pod \"global-pull-secret-syncer-dq8s4\" (UID: \"3e92c2a7-1021-4ee8-aded-8ae5ddc6b81c\") " pod="kube-system/global-pull-secret-syncer-dq8s4"
May 25 11:17:20.579733 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:20.579689    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
May 25 11:17:20.579805 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:20.579772    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3e92c2a7-1021-4ee8-aded-8ae5ddc6b81c-original-pull-secret podName:3e92c2a7-1021-4ee8-aded-8ae5ddc6b81c nodeName:}" failed. No retries permitted until 2026-05-25 11:17:28.579750711 +0000 UTC m=+22.124928300 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3e92c2a7-1021-4ee8-aded-8ae5ddc6b81c-original-pull-secret") pod "global-pull-secret-syncer-dq8s4" (UID: "3e92c2a7-1021-4ee8-aded-8ae5ddc6b81c") : object "kube-system"/"original-pull-secret" not registered
May 25 11:17:21.115417 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:21.115389    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-444mb"
May 25 11:17:21.115857 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:21.115388    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rl7hd"
May 25 11:17:21.115857 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:21.115526    2561 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-444mb" podUID="ead20bcd-df93-4d45-b5bb-470a8fe373e7"
May 25 11:17:21.115857 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:21.115572    2561 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rl7hd" podUID="2b05b78f-6ab1-4f94-8273-e5fa70b78d15"
May 25 11:17:22.112489 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:22.112456    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dq8s4"
May 25 11:17:22.112676 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:22.112569    2561 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dq8s4" podUID="3e92c2a7-1021-4ee8-aded-8ae5ddc6b81c"
May 25 11:17:23.116261 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:23.116232    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-444mb"
May 25 11:17:23.116261 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:23.116232    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rl7hd"
May 25 11:17:23.116773 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:23.116393    2561 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-444mb" podUID="ead20bcd-df93-4d45-b5bb-470a8fe373e7"
May 25 11:17:23.116773 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:23.116473    2561 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rl7hd" podUID="2b05b78f-6ab1-4f94-8273-e5fa70b78d15"
May 25 11:17:24.112405 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:24.112371    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dq8s4"
May 25 11:17:24.112572 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:24.112474    2561 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dq8s4" podUID="3e92c2a7-1021-4ee8-aded-8ae5ddc6b81c"
May 25 11:17:24.812334 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:24.812247    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ead20bcd-df93-4d45-b5bb-470a8fe373e7-metrics-certs\") pod \"network-metrics-daemon-444mb\" (UID: \"ead20bcd-df93-4d45-b5bb-470a8fe373e7\") " pod="openshift-multus/network-metrics-daemon-444mb"
May 25 11:17:24.812775 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:24.812434    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 11:17:24.812775 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:24.812515    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ead20bcd-df93-4d45-b5bb-470a8fe373e7-metrics-certs podName:ead20bcd-df93-4d45-b5bb-470a8fe373e7 nodeName:}" failed. No retries permitted until 2026-05-25 11:17:40.812492182 +0000 UTC m=+34.357669764 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ead20bcd-df93-4d45-b5bb-470a8fe373e7-metrics-certs") pod "network-metrics-daemon-444mb" (UID: "ead20bcd-df93-4d45-b5bb-470a8fe373e7") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 11:17:24.912855 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:24.912814    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-nbhnn\" (UniqueName: \"kubernetes.io/projected/2b05b78f-6ab1-4f94-8273-e5fa70b78d15-kube-api-access-nbhnn\") pod \"network-check-target-rl7hd\" (UID: \"2b05b78f-6ab1-4f94-8273-e5fa70b78d15\") " pod="openshift-network-diagnostics/network-check-target-rl7hd"
May 25 11:17:24.913030 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:24.912957    2561 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 25 11:17:24.913030 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:24.912976    2561 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 25 11:17:24.913030 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:24.912988    2561 projected.go:194] Error preparing data for projected volume kube-api-access-nbhnn for pod openshift-network-diagnostics/network-check-target-rl7hd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 11:17:24.913185 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:24.913052    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2b05b78f-6ab1-4f94-8273-e5fa70b78d15-kube-api-access-nbhnn podName:2b05b78f-6ab1-4f94-8273-e5fa70b78d15 nodeName:}" failed. No retries permitted until 2026-05-25 11:17:40.913034006 +0000 UTC m=+34.458211606 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-nbhnn" (UniqueName: "kubernetes.io/projected/2b05b78f-6ab1-4f94-8273-e5fa70b78d15-kube-api-access-nbhnn") pod "network-check-target-rl7hd" (UID: "2b05b78f-6ab1-4f94-8273-e5fa70b78d15") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 11:17:25.112397 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:25.112361    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-444mb"
May 25 11:17:25.112397 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:25.112382    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rl7hd"
May 25 11:17:25.112637 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:25.112538    2561 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-444mb" podUID="ead20bcd-df93-4d45-b5bb-470a8fe373e7"
May 25 11:17:25.112723 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:25.112678    2561 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rl7hd" podUID="2b05b78f-6ab1-4f94-8273-e5fa70b78d15"
May 25 11:17:26.112794 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:26.112764    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dq8s4"
May 25 11:17:26.113167 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:26.112868    2561 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dq8s4" podUID="3e92c2a7-1021-4ee8-aded-8ae5ddc6b81c"
May 25 11:17:27.113665 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:27.113634    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rl7hd"
May 25 11:17:27.114105 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:27.113731    2561 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rl7hd" podUID="2b05b78f-6ab1-4f94-8273-e5fa70b78d15"
May 25 11:17:27.114105 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:27.113825    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-444mb"
May 25 11:17:27.114105 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:27.113979    2561 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-444mb" podUID="ead20bcd-df93-4d45-b5bb-470a8fe373e7"
May 25 11:17:28.113724 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:28.113109    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dq8s4"
May 25 11:17:28.113724 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:28.113233    2561 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dq8s4" podUID="3e92c2a7-1021-4ee8-aded-8ae5ddc6b81c"
May 25 11:17:28.244352 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:28.244315    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5ldrp" event={"ID":"71885334-70ca-483b-8bd9-fdbfecb77949","Type":"ContainerStarted","Data":"fd22e3763b245bcf4e6954e94d66ba1ecf131ed858ec1d9ffcd602fa3766eed1"}
May 25 11:17:28.245941 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:28.245917    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-k2fsn" event={"ID":"b0e8a1a4-c0cb-443e-8c38-9abcf242fc84","Type":"ContainerStarted","Data":"38c0375096057c2a0158426ada982b27e6c0005b9c3858323ef92df5dd29f9a0"}
May 25 11:17:28.247632 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:28.247611    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-q8ng6" event={"ID":"9a250955-746b-416a-b798-283a0ea46864","Type":"ContainerStarted","Data":"986ff0f28430c4e87d375f6ca0ea804bc45d41f2402ba1b15e8f03172a0d9089"}
May 25 11:17:28.249073 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:28.249053    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-bxzz6" event={"ID":"a3d75818-4ca3-4bb1-a3dd-28ae4ce71d09","Type":"ContainerStarted","Data":"0209dee9b65fb28d881e4fd70b2c282222e22727a8caa1c4a67f17c694e22e92"}
May 25 11:17:28.251187 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:28.251144    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-xvcpl" event={"ID":"b55f6df2-1dbc-4fc5-a4e6-e679d289e9f5","Type":"ContainerStarted","Data":"60d8b7eda09ab5ba37b3fe7eab9d49ac19d4e9a20f4657edbcd0c9f18190ae0b"}
May 25 11:17:28.252933 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:28.252910    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dmzql" event={"ID":"7c97daaa-b9c1-4ee0-ade1-e8de392e496f","Type":"ContainerStarted","Data":"586fc1556236748ee0fee889a240e2f910f6b6d6db13f822396a4aa5afbfedd6"}
May 25 11:17:28.254651 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:28.254629    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-mr48p" event={"ID":"ae10dbbc-8226-48df-a0f7-39aed3d95c58","Type":"ContainerStarted","Data":"aa547cac2cb4197a6be65b6f69ffb56bd77edd5cefa4109c0a99c377ee28955a"}
May 25 11:17:28.256055 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:28.256034    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-thvg5" event={"ID":"5b2bf5ae-a8a4-48e9-9afc-ee576988791e","Type":"ContainerStarted","Data":"75371a5b992a8b5e80713f1824726f6cf59ddc7d40ea6ff91555c2c285b3fec2"}
May 25 11:17:28.256055 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:28.256057    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-thvg5" event={"ID":"5b2bf5ae-a8a4-48e9-9afc-ee576988791e","Type":"ContainerStarted","Data":"898352bc7b62212d11425bd44dbce7b67e2e5ea9c592dcd4fddda18ee030d681"}
May 25 11:17:28.260568 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:28.260532    2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-k2fsn" podStartSLOduration=3.006909703 podStartE2EDuration="21.260521715s" podCreationTimestamp="2026-05-25 11:17:07 +0000 UTC" firstStartedPulling="2026-05-25 11:17:09.728735568 +0000 UTC m=+3.273913154" lastFinishedPulling="2026-05-25 11:17:27.982347568 +0000 UTC m=+21.527525166" observedRunningTime="2026-05-25 11:17:28.260256831 +0000 UTC m=+21.805434434" watchObservedRunningTime="2026-05-25 11:17:28.260521715 +0000 UTC m=+21.805699317"
May 25 11:17:28.272830 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:28.272738    2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/konnectivity-agent-mr48p" podStartSLOduration=3.014793909 podStartE2EDuration="21.272727057s" podCreationTimestamp="2026-05-25 11:17:07 +0000 UTC" firstStartedPulling="2026-05-25 11:17:09.722635215 +0000 UTC m=+3.267812797" lastFinishedPulling="2026-05-25 11:17:27.980568353 +0000 UTC m=+21.525745945" observedRunningTime="2026-05-25 11:17:28.272345634 +0000 UTC m=+21.817523239" watchObservedRunningTime="2026-05-25 11:17:28.272727057 +0000 UTC m=+21.817904717"
May 25 11:17:28.290592 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:28.290546    2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/multus-bxzz6" podStartSLOduration=2.992778423 podStartE2EDuration="21.290533538s" podCreationTimestamp="2026-05-25 11:17:07 +0000 UTC" firstStartedPulling="2026-05-25 11:17:09.725960942 +0000 UTC m=+3.271138527" lastFinishedPulling="2026-05-25 11:17:28.023716044 +0000 UTC m=+21.568893642" observedRunningTime="2026-05-25 11:17:28.290125895 +0000 UTC m=+21.835303499" watchObservedRunningTime="2026-05-25 11:17:28.290533538 +0000 UTC m=+21.835711177"
May 25 11:17:28.314388 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:28.314346    2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-dns/node-resolver-xvcpl" podStartSLOduration=3.066705742 podStartE2EDuration="21.314334275s" podCreationTimestamp="2026-05-25 11:17:07 +0000 UTC" firstStartedPulling="2026-05-25 11:17:09.733044658 +0000 UTC m=+3.278222243" lastFinishedPulling="2026-05-25 11:17:27.980673179 +0000 UTC m=+21.525850776" observedRunningTime="2026-05-25 11:17:28.314083453 +0000 UTC m=+21.859261057" watchObservedRunningTime="2026-05-25 11:17:28.314334275 +0000 UTC m=+21.859511878"
May 25 11:17:28.345455 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:28.345399    2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-image-registry/node-ca-q8ng6" podStartSLOduration=11.365298919 podStartE2EDuration="21.345379237s" podCreationTimestamp="2026-05-25 11:17:07 +0000 UTC" firstStartedPulling="2026-05-25 11:17:09.727640555 +0000 UTC m=+3.272818140" lastFinishedPulling="2026-05-25 11:17:19.707720861 +0000 UTC m=+13.252898458" observedRunningTime="2026-05-25 11:17:28.34501956 +0000 UTC m=+21.890197164" watchObservedRunningTime="2026-05-25 11:17:28.345379237 +0000 UTC m=+21.890556900"
May 25 11:17:28.643433 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:28.643398    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3e92c2a7-1021-4ee8-aded-8ae5ddc6b81c-original-pull-secret\") pod \"global-pull-secret-syncer-dq8s4\" (UID: \"3e92c2a7-1021-4ee8-aded-8ae5ddc6b81c\") " pod="kube-system/global-pull-secret-syncer-dq8s4"
May 25 11:17:28.643695 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:28.643517    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
May 25 11:17:28.643695 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:28.643601    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3e92c2a7-1021-4ee8-aded-8ae5ddc6b81c-original-pull-secret podName:3e92c2a7-1021-4ee8-aded-8ae5ddc6b81c nodeName:}" failed. No retries permitted until 2026-05-25 11:17:44.643578215 +0000 UTC m=+38.188755812 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3e92c2a7-1021-4ee8-aded-8ae5ddc6b81c-original-pull-secret") pod "global-pull-secret-syncer-dq8s4" (UID: "3e92c2a7-1021-4ee8-aded-8ae5ddc6b81c") : object "kube-system"/"original-pull-secret" not registered
May 25 11:17:29.113319 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:29.113289    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-444mb"
May 25 11:17:29.113469 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:29.113304    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rl7hd"
May 25 11:17:29.113469 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:29.113409    2561 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-444mb" podUID="ead20bcd-df93-4d45-b5bb-470a8fe373e7"
May 25 11:17:29.113558 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:29.113468    2561 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rl7hd" podUID="2b05b78f-6ab1-4f94-8273-e5fa70b78d15"
May 25 11:17:29.258490 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:29.258462    2561 generic.go:358] "Generic (PLEG): container finished" podID="7c97daaa-b9c1-4ee0-ade1-e8de392e496f" containerID="586fc1556236748ee0fee889a240e2f910f6b6d6db13f822396a4aa5afbfedd6" exitCode=0
May 25 11:17:29.259091 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:29.258539    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dmzql" event={"ID":"7c97daaa-b9c1-4ee0-ade1-e8de392e496f","Type":"ContainerDied","Data":"586fc1556236748ee0fee889a240e2f910f6b6d6db13f822396a4aa5afbfedd6"}
May 25 11:17:29.261051 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:29.261029    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-thvg5" event={"ID":"5b2bf5ae-a8a4-48e9-9afc-ee576988791e","Type":"ContainerStarted","Data":"94c00b09c811d0501287aa4b7de7ec56cf28742bffb1fc59d03a7ae8ff8104fd"}
May 25 11:17:29.261121 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:29.261059    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-thvg5" event={"ID":"5b2bf5ae-a8a4-48e9-9afc-ee576988791e","Type":"ContainerStarted","Data":"c97b886b7b15f66e951b210a53aaa580fbe6eb572576676d3d3bacc5e4a93e00"}
May 25 11:17:29.261121 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:29.261073    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-thvg5" event={"ID":"5b2bf5ae-a8a4-48e9-9afc-ee576988791e","Type":"ContainerStarted","Data":"a4ed2ecca2d2482b0f7773141da6b58306633a2abc34f14b42f91e96be36e50b"}
May 25 11:17:29.261121 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:29.261084    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-thvg5" event={"ID":"5b2bf5ae-a8a4-48e9-9afc-ee576988791e","Type":"ContainerStarted","Data":"7be2d6489dd5f193f03c9513cafd1ac4222ef8f05c9876c3a97734848e2d1de6"}
May 25 11:17:29.392422 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:29.392398    2561 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
May 25 11:17:30.053218 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:30.053110    2561 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-05-25T11:17:29.392419865Z","UUID":"d7c1d37a-dc3f-40d7-9fb0-8906289f095c","Handler":null,"Name":"","Endpoint":""}
May 25 11:17:30.056334 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:30.056308    2561 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
May 25 11:17:30.056469 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:30.056341    2561 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
May 25 11:17:30.112687 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:30.112661    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dq8s4"
May 25 11:17:30.112818 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:30.112773    2561 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dq8s4" podUID="3e92c2a7-1021-4ee8-aded-8ae5ddc6b81c"
May 25 11:17:30.265119 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:30.265084    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5ldrp" event={"ID":"71885334-70ca-483b-8bd9-fdbfecb77949","Type":"ContainerStarted","Data":"5b4893df5e9b789fd53006031d827c7a7566a50ba34d5ea504ba6a27c52c32d9"}
May 25 11:17:30.266640 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:30.266613    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-4xz76" event={"ID":"a8a3a45a-3dba-4919-895a-ef467fbd4343","Type":"ContainerStarted","Data":"7e13846bac726a5f29eb678bf773e32191f78e8a8b1a9d9a89e4f395beb8a027"}
May 25 11:17:30.283005 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:30.282962    2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-4xz76" podStartSLOduration=5.027871952 podStartE2EDuration="23.282949885s" podCreationTimestamp="2026-05-25 11:17:07 +0000 UTC" firstStartedPulling="2026-05-25 11:17:09.725587895 +0000 UTC m=+3.270765483" lastFinishedPulling="2026-05-25 11:17:27.980665819 +0000 UTC m=+21.525843416" observedRunningTime="2026-05-25 11:17:30.282535411 +0000 UTC m=+23.827713014" watchObservedRunningTime="2026-05-25 11:17:30.282949885 +0000 UTC m=+23.828127545"
May 25 11:17:31.113167 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:31.113136    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rl7hd"
May 25 11:17:31.113390 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:31.113143    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-444mb"
May 25 11:17:31.113390 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:31.113256    2561 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rl7hd" podUID="2b05b78f-6ab1-4f94-8273-e5fa70b78d15"
May 25 11:17:31.113390 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:31.113368    2561 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-444mb" podUID="ead20bcd-df93-4d45-b5bb-470a8fe373e7"
May 25 11:17:31.271817 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:31.271780    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-thvg5" event={"ID":"5b2bf5ae-a8a4-48e9-9afc-ee576988791e","Type":"ContainerStarted","Data":"3debbacf971b1de909ba9a01edb014133cff50050c26f730869ca1d4fec8c30a"}
May 25 11:17:31.273995 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:31.273964    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5ldrp" event={"ID":"71885334-70ca-483b-8bd9-fdbfecb77949","Type":"ContainerStarted","Data":"62c6e18bbe2727003265c8ebd35b44a451c837facc0d81684b9f24a87aec9deb"}
May 25 11:17:31.293050 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:31.293003    2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5ldrp" podStartSLOduration=3.725203265 podStartE2EDuration="24.292987303s" podCreationTimestamp="2026-05-25 11:17:07 +0000 UTC" firstStartedPulling="2026-05-25 11:17:09.732280904 +0000 UTC m=+3.277458499" lastFinishedPulling="2026-05-25 11:17:30.300064956 +0000 UTC m=+23.845242537" observedRunningTime="2026-05-25 11:17:31.292863838 +0000 UTC m=+24.838041442" watchObservedRunningTime="2026-05-25 11:17:31.292987303 +0000 UTC m=+24.838164909"
May 25 11:17:32.112614 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:32.112580    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dq8s4"
May 25 11:17:32.112790 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:32.112705    2561 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dq8s4" podUID="3e92c2a7-1021-4ee8-aded-8ae5ddc6b81c"
May 25 11:17:33.112284 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:33.112239    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rl7hd"
May 25 11:17:33.112766 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:33.112285    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-444mb"
May 25 11:17:33.112766 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:33.112370    2561 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rl7hd" podUID="2b05b78f-6ab1-4f94-8273-e5fa70b78d15"
May 25 11:17:33.112766 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:33.112470    2561 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-444mb" podUID="ead20bcd-df93-4d45-b5bb-470a8fe373e7"
May 25 11:17:33.214551 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:33.214526    2561 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-mr48p"
May 25 11:17:33.215076 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:33.215030    2561 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-mr48p"
May 25 11:17:33.280094 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:33.280068    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-thvg5" event={"ID":"5b2bf5ae-a8a4-48e9-9afc-ee576988791e","Type":"ContainerStarted","Data":"fb24c8f95f0cc8dc444fe146178c328686c44ef97ddc06cafa9b074f7bb0cad7"}
May 25 11:17:33.280591 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:33.280517    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-mr48p"
May 25 11:17:33.280591 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:33.280548    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-thvg5"
May 25 11:17:33.280591 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:33.280564    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-thvg5"
May 25 11:17:33.281004 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:33.280977    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-thvg5"
May 25 11:17:33.281095 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:33.281039    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-mr48p"
May 25 11:17:33.296003 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:33.295982    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-thvg5"
May 25 11:17:33.296099 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:33.296065    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-thvg5"
May 25 11:17:33.311288 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:33.311232    2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-thvg5" podStartSLOduration=7.961140885 podStartE2EDuration="26.311222759s" podCreationTimestamp="2026-05-25 11:17:07 +0000 UTC" firstStartedPulling="2026-05-25 11:17:09.721841945 +0000 UTC m=+3.267019539" lastFinishedPulling="2026-05-25 11:17:28.07192383 +0000 UTC m=+21.617101413" observedRunningTime="2026-05-25 11:17:33.310764628 +0000 UTC m=+26.855942231" watchObservedRunningTime="2026-05-25 11:17:33.311222759 +0000 UTC m=+26.856400362"
May 25 11:17:34.112761 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:34.112619    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dq8s4"
May 25 11:17:34.113207 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:34.112835    2561 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dq8s4" podUID="3e92c2a7-1021-4ee8-aded-8ae5ddc6b81c"
May 25 11:17:34.283816 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:34.283790    2561 generic.go:358] "Generic (PLEG): container finished" podID="7c97daaa-b9c1-4ee0-ade1-e8de392e496f" containerID="8a60ab2defe27ba95877c28fb678bdbfa38ae3534be300de41e4f2e6b59480bd" exitCode=0
May 25 11:17:34.283965 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:34.283879    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dmzql" event={"ID":"7c97daaa-b9c1-4ee0-ade1-e8de392e496f","Type":"ContainerDied","Data":"8a60ab2defe27ba95877c28fb678bdbfa38ae3534be300de41e4f2e6b59480bd"}
May 25 11:17:35.112971 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:35.112945    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rl7hd"
May 25 11:17:35.113340 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:35.113035    2561 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rl7hd" podUID="2b05b78f-6ab1-4f94-8273-e5fa70b78d15"
May 25 11:17:35.113340 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:35.113119    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-444mb"
May 25 11:17:35.113340 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:35.113209    2561 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-444mb" podUID="ead20bcd-df93-4d45-b5bb-470a8fe373e7"
May 25 11:17:35.130982 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:35.130956    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-dq8s4"]
May 25 11:17:35.131120 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:35.131034    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dq8s4"
May 25 11:17:35.131120 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:35.131105    2561 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dq8s4" podUID="3e92c2a7-1021-4ee8-aded-8ae5ddc6b81c"
May 25 11:17:35.137750 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:35.137717    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-444mb"]
May 25 11:17:35.138315 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:35.138291    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-rl7hd"]
May 25 11:17:35.285457 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:35.285430    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-444mb"
May 25 11:17:35.285585 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:35.285485    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rl7hd"
May 25 11:17:35.285625 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:35.285574    2561 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-444mb" podUID="ead20bcd-df93-4d45-b5bb-470a8fe373e7"
May 25 11:17:35.285676 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:35.285648    2561 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rl7hd" podUID="2b05b78f-6ab1-4f94-8273-e5fa70b78d15"
May 25 11:17:36.288603 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:36.288571    2561 generic.go:358] "Generic (PLEG): container finished" podID="7c97daaa-b9c1-4ee0-ade1-e8de392e496f" containerID="f002e7d9c33dbd8a96b82bf21e7f16a6ecddc75f0c57f1cc713cd800217aedf3" exitCode=0
May 25 11:17:36.288953 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:36.288620    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dmzql" event={"ID":"7c97daaa-b9c1-4ee0-ade1-e8de392e496f","Type":"ContainerDied","Data":"f002e7d9c33dbd8a96b82bf21e7f16a6ecddc75f0c57f1cc713cd800217aedf3"}
May 25 11:17:37.113174 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:37.113147    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rl7hd"
May 25 11:17:37.113331 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:37.113234    2561 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rl7hd" podUID="2b05b78f-6ab1-4f94-8273-e5fa70b78d15"
May 25 11:17:37.113331 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:37.113311    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dq8s4"
May 25 11:17:37.113418 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:37.113397    2561 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dq8s4" podUID="3e92c2a7-1021-4ee8-aded-8ae5ddc6b81c"
May 25 11:17:37.113474 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:37.113436    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-444mb"
May 25 11:17:37.113531 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:37.113517    2561 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-444mb" podUID="ead20bcd-df93-4d45-b5bb-470a8fe373e7"
May 25 11:17:37.293325 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:37.293126    2561 generic.go:358] "Generic (PLEG): container finished" podID="7c97daaa-b9c1-4ee0-ade1-e8de392e496f" containerID="de0732ac1e2daa7d21a5a2da59b9c933b9674b2031e81b1fffcbb04b414cc363" exitCode=0
May 25 11:17:37.293634 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:37.293209    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dmzql" event={"ID":"7c97daaa-b9c1-4ee0-ade1-e8de392e496f","Type":"ContainerDied","Data":"de0732ac1e2daa7d21a5a2da59b9c933b9674b2031e81b1fffcbb04b414cc363"}
May 25 11:17:39.113012 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:39.112982    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dq8s4"
May 25 11:17:39.113419 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:39.112985    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-444mb"
May 25 11:17:39.113419 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:39.113077    2561 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dq8s4" podUID="3e92c2a7-1021-4ee8-aded-8ae5ddc6b81c"
May 25 11:17:39.113419 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:39.112985    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rl7hd"
May 25 11:17:39.113419 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:39.113194    2561 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-444mb" podUID="ead20bcd-df93-4d45-b5bb-470a8fe373e7"
May 25 11:17:39.113419 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:39.113246    2561 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rl7hd" podUID="2b05b78f-6ab1-4f94-8273-e5fa70b78d15"
May 25 11:17:40.788868 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:40.788804    2561 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-136-227.ec2.internal" event="NodeReady"
May 25 11:17:40.789443 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:40.788935    2561 kubelet_node_status.go:550] "Fast updating node status as it just became ready"
May 25 11:17:40.836744 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:40.836717    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ead20bcd-df93-4d45-b5bb-470a8fe373e7-metrics-certs\") pod \"network-metrics-daemon-444mb\" (UID: \"ead20bcd-df93-4d45-b5bb-470a8fe373e7\") " pod="openshift-multus/network-metrics-daemon-444mb"
May 25 11:17:40.836905 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:40.836805    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 11:17:40.836905 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:40.836865    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ead20bcd-df93-4d45-b5bb-470a8fe373e7-metrics-certs podName:ead20bcd-df93-4d45-b5bb-470a8fe373e7 nodeName:}" failed. No retries permitted until 2026-05-25 11:18:12.836847642 +0000 UTC m=+66.382025236 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ead20bcd-df93-4d45-b5bb-470a8fe373e7-metrics-certs") pod "network-metrics-daemon-444mb" (UID: "ead20bcd-df93-4d45-b5bb-470a8fe373e7") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 11:17:40.865162 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:40.865127    2561 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-fwgjs"]
May 25 11:17:40.935247 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:40.935206    2561 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7b9fd9b9cb-fpswn"]
May 25 11:17:40.935421 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:40.935367    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-fwgjs"
May 25 11:17:40.937115 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:40.937090    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-nbhnn\" (UniqueName: \"kubernetes.io/projected/2b05b78f-6ab1-4f94-8273-e5fa70b78d15-kube-api-access-nbhnn\") pod \"network-check-target-rl7hd\" (UID: \"2b05b78f-6ab1-4f94-8273-e5fa70b78d15\") " pod="openshift-network-diagnostics/network-check-target-rl7hd"
May 25 11:17:40.937278 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:40.937250    2561 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 25 11:17:40.937333 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:40.937286    2561 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 25 11:17:40.937333 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:40.937297    2561 projected.go:194] Error preparing data for projected volume kube-api-access-nbhnn for pod openshift-network-diagnostics/network-check-target-rl7hd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 11:17:40.937443 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:40.937350    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2b05b78f-6ab1-4f94-8273-e5fa70b78d15-kube-api-access-nbhnn podName:2b05b78f-6ab1-4f94-8273-e5fa70b78d15 nodeName:}" failed. No retries permitted until 2026-05-25 11:18:12.93733756 +0000 UTC m=+66.482515141 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-nbhnn" (UniqueName: "kubernetes.io/projected/2b05b78f-6ab1-4f94-8273-e5fa70b78d15-kube-api-access-nbhnn") pod "network-check-target-rl7hd" (UID: "2b05b78f-6ab1-4f94-8273-e5fa70b78d15") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 11:17:40.939298 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:40.939263    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
May 25 11:17:40.939725 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:40.939707    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
May 25 11:17:40.939822 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:40.939702    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-l7927\""
May 25 11:17:40.959585 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:40.959563    2561 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-846bb98c44-b4vb2"]
May 25 11:17:40.959748 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:40.959727    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7b9fd9b9cb-fpswn"
May 25 11:17:40.964497 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:40.964303    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-gzhs6\""
May 25 11:17:40.964497 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:40.964345    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
May 25 11:17:40.964673 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:40.964660    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
May 25 11:17:40.965083 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:40.964901    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
May 25 11:17:40.970567 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:40.970538    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
May 25 11:17:40.984155 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:40.984131    2561 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-76dff48d59-9lj95"]
May 25 11:17:40.984340 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:40.984318    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-846bb98c44-b4vb2"
May 25 11:17:40.986981 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:40.986962    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-dmhkh\""
May 25 11:17:40.987077 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:40.986972    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
May 25 11:17:40.987077 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:40.987020    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
May 25 11:17:40.987191 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:40.987112    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
May 25 11:17:40.987314 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:40.987297    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
May 25 11:17:40.987380 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:40.987300    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
May 25 11:17:40.987581 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:40.987550    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
May 25 11:17:40.999739 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:40.999719    2561 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6c75959455-tfzq9"]
May 25 11:17:40.999929 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:40.999911    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-76dff48d59-9lj95"
May 25 11:17:41.003220 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.003193    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-cc5vw\""
May 25 11:17:41.003868 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.003393    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
May 25 11:17:41.003868 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.003475    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
May 25 11:17:41.003868 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.003839    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
May 25 11:17:41.004263 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.003847    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
May 25 11:17:41.011380 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.011363    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
May 25 11:17:41.015933 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.015913    2561 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-88l9x"]
May 25 11:17:41.016089 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.016069    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-tfzq9"
May 25 11:17:41.020663 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.020647    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
May 25 11:17:41.021421 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.021384    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-9rfft\""
May 25 11:17:41.021515 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.021432    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
May 25 11:17:41.021700 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.021683    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
May 25 11:17:41.023491 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.023474    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
May 25 11:17:41.032013 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.031993    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-fwgjs"]
May 25 11:17:41.032117 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.032022    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6c75959455-tfzq9"]
May 25 11:17:41.032181 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.032120    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-76dff48d59-9lj95"]
May 25 11:17:41.032181 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.032141    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7b9fd9b9cb-fpswn"]
May 25 11:17:41.032181 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.032143    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-88l9x"
May 25 11:17:41.032181 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.032154    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-846bb98c44-b4vb2"]
May 25 11:17:41.032181 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.032168    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-88l9x"]
May 25 11:17:41.032386 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.032198    2561 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-697587599-6vz7w"]
May 25 11:17:41.035591 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.035570    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
May 25 11:17:41.035681 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.035583    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
May 25 11:17:41.035863 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.035849    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
May 25 11:17:41.036017 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.035999    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
May 25 11:17:41.036122 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.036012    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-82962\""
May 25 11:17:41.037412 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.037391    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/21792ece-de04-4eb0-990c-e1adec1c79c5-ca-trust-extracted\") pod \"image-registry-7b9fd9b9cb-fpswn\" (UID: \"21792ece-de04-4eb0-990c-e1adec1c79c5\") " pod="openshift-image-registry/image-registry-7b9fd9b9cb-fpswn"
May 25 11:17:41.037497 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.037427    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5zjj\" (UniqueName: \"kubernetes.io/projected/21792ece-de04-4eb0-990c-e1adec1c79c5-kube-api-access-x5zjj\") pod \"image-registry-7b9fd9b9cb-fpswn\" (UID: \"21792ece-de04-4eb0-990c-e1adec1c79c5\") " pod="openshift-image-registry/image-registry-7b9fd9b9cb-fpswn"
May 25 11:17:41.037497 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.037470    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/21792ece-de04-4eb0-990c-e1adec1c79c5-registry-certificates\") pod \"image-registry-7b9fd9b9cb-fpswn\" (UID: \"21792ece-de04-4eb0-990c-e1adec1c79c5\") " pod="openshift-image-registry/image-registry-7b9fd9b9cb-fpswn"
May 25 11:17:41.037592 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.037577    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqrf7\" (UniqueName: \"kubernetes.io/projected/aa041c38-250b-4342-9951-7f04c9daf30c-kube-api-access-qqrf7\") pod \"volume-data-source-validator-5cf8c89d64-fwgjs\" (UID: \"aa041c38-250b-4342-9951-7f04c9daf30c\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-fwgjs"
May 25 11:17:41.037650 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.037612    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/21792ece-de04-4eb0-990c-e1adec1c79c5-installation-pull-secrets\") pod \"image-registry-7b9fd9b9cb-fpswn\" (UID: \"21792ece-de04-4eb0-990c-e1adec1c79c5\") " pod="openshift-image-registry/image-registry-7b9fd9b9cb-fpswn"
May 25 11:17:41.037706 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.037680    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/21792ece-de04-4eb0-990c-e1adec1c79c5-trusted-ca\") pod \"image-registry-7b9fd9b9cb-fpswn\" (UID: \"21792ece-de04-4eb0-990c-e1adec1c79c5\") " pod="openshift-image-registry/image-registry-7b9fd9b9cb-fpswn"
May 25 11:17:41.037761 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.037704    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/21792ece-de04-4eb0-990c-e1adec1c79c5-bound-sa-token\") pod \"image-registry-7b9fd9b9cb-fpswn\" (UID: \"21792ece-de04-4eb0-990c-e1adec1c79c5\") " pod="openshift-image-registry/image-registry-7b9fd9b9cb-fpswn"
May 25 11:17:41.037761 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.037734    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/21792ece-de04-4eb0-990c-e1adec1c79c5-image-registry-private-configuration\") pod \"image-registry-7b9fd9b9cb-fpswn\" (UID: \"21792ece-de04-4eb0-990c-e1adec1c79c5\") " pod="openshift-image-registry/image-registry-7b9fd9b9cb-fpswn"
May 25 11:17:41.037904 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.037760    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/21792ece-de04-4eb0-990c-e1adec1c79c5-registry-tls\") pod \"image-registry-7b9fd9b9cb-fpswn\" (UID: \"21792ece-de04-4eb0-990c-e1adec1c79c5\") " pod="openshift-image-registry/image-registry-7b9fd9b9cb-fpswn"
May 25 11:17:41.048015 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.047973    2561 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-78bdfb888c-nd5f5"]
May 25 11:17:41.048156 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.048124    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-6vz7w"
May 25 11:17:41.052796 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.052776    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-t8vmz\""
May 25 11:17:41.052891 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.052853    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
May 25 11:17:41.053136 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.053112    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
May 25 11:17:41.053219 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.053179    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
May 25 11:17:41.062336 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.062020    2561 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-d4d8f94f9-8ktvq"]
May 25 11:17:41.062502 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.062483    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-78bdfb888c-nd5f5"
May 25 11:17:41.067437 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.067408    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
May 25 11:17:41.067564 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.067470    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
May 25 11:17:41.067564 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.067497    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
May 25 11:17:41.067748 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.067602    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
May 25 11:17:41.067819 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.067764    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-w6j6d\""
May 25 11:17:41.074512 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.074495    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
May 25 11:17:41.083130 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.083109    2561 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7f44b6dd87-7b6rg"]
May 25 11:17:41.083257 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.083242    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-d4d8f94f9-8ktvq"
May 25 11:17:41.087153 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.087137    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
May 25 11:17:41.087252 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.087157    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
May 25 11:17:41.087336 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.087317    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-6n45b\""
May 25 11:17:41.107353 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.107332    2561 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-66tdv"]
May 25 11:17:41.107493 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.107478    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7f44b6dd87-7b6rg"
May 25 11:17:41.110097 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.110082    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
May 25 11:17:41.110215 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.110153    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
May 25 11:17:41.110215 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.110202    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-ptms6\""
May 25 11:17:41.122560 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.122540    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-66tdv"
May 25 11:17:41.125328 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.125307    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
May 25 11:17:41.125454 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.125426    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
May 25 11:17:41.125556 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.125511    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-kttpg\""
May 25 11:17:41.125696 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.125678    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
May 25 11:17:41.126009 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.125987    2561 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57b5b68b8-clx9s"]
May 25 11:17:41.138827 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.138806    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/21792ece-de04-4eb0-990c-e1adec1c79c5-registry-certificates\") pod \"image-registry-7b9fd9b9cb-fpswn\" (UID: \"21792ece-de04-4eb0-990c-e1adec1c79c5\") " pod="openshift-image-registry/image-registry-7b9fd9b9cb-fpswn"
May 25 11:17:41.138927 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.138840    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9b9cd68b-6d33-40ae-b547-43f4ef7a2f5c-config\") pod \"kube-storage-version-migrator-operator-565644f49-88l9x\" (UID: \"9b9cd68b-6d33-40ae-b547-43f4ef7a2f5c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-88l9x"
May 25 11:17:41.138927 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.138878    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqd8w\" (UniqueName: \"kubernetes.io/projected/8bf76398-20ba-499c-9103-c4017cdad681-kube-api-access-lqd8w\") pod \"cluster-samples-operator-697587599-6vz7w\" (UID: \"8bf76398-20ba-499c-9103-c4017cdad681\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-6vz7w"
May 25 11:17:41.138927 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.138904    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lsrfh\" (UniqueName: \"kubernetes.io/projected/c6cac71b-58b3-43ba-8a3a-efb15e35c401-kube-api-access-lsrfh\") pod \"router-default-846bb98c44-b4vb2\" (UID: \"c6cac71b-58b3-43ba-8a3a-efb15e35c401\") " pod="openshift-ingress/router-default-846bb98c44-b4vb2"
May 25 11:17:41.138927 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.138924    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/10a66344-3304-49a1-bbc1-b885800f5acf-config\") pod \"console-operator-76dff48d59-9lj95\" (UID: \"10a66344-3304-49a1-bbc1-b885800f5acf\") " pod="openshift-console-operator/console-operator-76dff48d59-9lj95"
May 25 11:17:41.139131 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.138989    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-qqrf7\" (UniqueName: \"kubernetes.io/projected/aa041c38-250b-4342-9951-7f04c9daf30c-kube-api-access-qqrf7\") pod \"volume-data-source-validator-5cf8c89d64-fwgjs\" (UID: \"aa041c38-250b-4342-9951-7f04c9daf30c\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-fwgjs"
May 25 11:17:41.139131 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.139039    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/21792ece-de04-4eb0-990c-e1adec1c79c5-installation-pull-secrets\") pod \"image-registry-7b9fd9b9cb-fpswn\" (UID: \"21792ece-de04-4eb0-990c-e1adec1c79c5\") " pod="openshift-image-registry/image-registry-7b9fd9b9cb-fpswn"
May 25 11:17:41.139231 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.139205    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/21792ece-de04-4eb0-990c-e1adec1c79c5-trusted-ca\") pod \"image-registry-7b9fd9b9cb-fpswn\" (UID: \"21792ece-de04-4eb0-990c-e1adec1c79c5\") " pod="openshift-image-registry/image-registry-7b9fd9b9cb-fpswn"
May 25 11:17:41.139294 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.139236    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/21792ece-de04-4eb0-990c-e1adec1c79c5-bound-sa-token\") pod \"image-registry-7b9fd9b9cb-fpswn\" (UID: \"21792ece-de04-4eb0-990c-e1adec1c79c5\") " pod="openshift-image-registry/image-registry-7b9fd9b9cb-fpswn"
May 25 11:17:41.139294 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.139282    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/21792ece-de04-4eb0-990c-e1adec1c79c5-image-registry-private-configuration\") pod \"image-registry-7b9fd9b9cb-fpswn\" (UID: \"21792ece-de04-4eb0-990c-e1adec1c79c5\") " pod="openshift-image-registry/image-registry-7b9fd9b9cb-fpswn"
May 25 11:17:41.139408 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.139311    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/21792ece-de04-4eb0-990c-e1adec1c79c5-registry-tls\") pod \"image-registry-7b9fd9b9cb-fpswn\" (UID: \"21792ece-de04-4eb0-990c-e1adec1c79c5\") " pod="openshift-image-registry/image-registry-7b9fd9b9cb-fpswn"
May 25 11:17:41.139408 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.139342    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjd5b\" (UniqueName: \"kubernetes.io/projected/82fb3b0a-798a-406b-99b4-548782582352-kube-api-access-wjd5b\") pod \"cluster-monitoring-operator-6c75959455-tfzq9\" (UID: \"82fb3b0a-798a-406b-99b4-548782582352\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-tfzq9"
May 25 11:17:41.139408 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.139370    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52lrs\" (UniqueName: \"kubernetes.io/projected/10a66344-3304-49a1-bbc1-b885800f5acf-kube-api-access-52lrs\") pod \"console-operator-76dff48d59-9lj95\" (UID: \"10a66344-3304-49a1-bbc1-b885800f5acf\") " pod="openshift-console-operator/console-operator-76dff48d59-9lj95"
May 25 11:17:41.139408 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.139398    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rlgjm\" (UniqueName: \"kubernetes.io/projected/9b9cd68b-6d33-40ae-b547-43f4ef7a2f5c-kube-api-access-rlgjm\") pod \"kube-storage-version-migrator-operator-565644f49-88l9x\" (UID: \"9b9cd68b-6d33-40ae-b547-43f4ef7a2f5c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-88l9x"
May 25 11:17:41.139611 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.139419    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/10a66344-3304-49a1-bbc1-b885800f5acf-serving-cert\") pod \"console-operator-76dff48d59-9lj95\" (UID: \"10a66344-3304-49a1-bbc1-b885800f5acf\") " pod="openshift-console-operator/console-operator-76dff48d59-9lj95"
May 25 11:17:41.139611 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.139442    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/10a66344-3304-49a1-bbc1-b885800f5acf-trusted-ca\") pod \"console-operator-76dff48d59-9lj95\" (UID: \"10a66344-3304-49a1-bbc1-b885800f5acf\") " pod="openshift-console-operator/console-operator-76dff48d59-9lj95"
May 25 11:17:41.139611 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.139463    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/c6cac71b-58b3-43ba-8a3a-efb15e35c401-default-certificate\") pod \"router-default-846bb98c44-b4vb2\" (UID: \"c6cac71b-58b3-43ba-8a3a-efb15e35c401\") " pod="openshift-ingress/router-default-846bb98c44-b4vb2"
May 25 11:17:41.139611 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.139487    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/82fb3b0a-798a-406b-99b4-548782582352-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6c75959455-tfzq9\" (UID: \"82fb3b0a-798a-406b-99b4-548782582352\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-tfzq9"
May 25 11:17:41.139611 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.139502    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/21792ece-de04-4eb0-990c-e1adec1c79c5-registry-certificates\") pod \"image-registry-7b9fd9b9cb-fpswn\" (UID: \"21792ece-de04-4eb0-990c-e1adec1c79c5\") " pod="openshift-image-registry/image-registry-7b9fd9b9cb-fpswn"
May 25 11:17:41.139611 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.139508    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/8bf76398-20ba-499c-9103-c4017cdad681-samples-operator-tls\") pod \"cluster-samples-operator-697587599-6vz7w\" (UID: \"8bf76398-20ba-499c-9103-c4017cdad681\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-6vz7w"
May 25 11:17:41.139611 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:41.139519    2561 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 25 11:17:41.139611 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:41.139546    2561 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7b9fd9b9cb-fpswn: secret "image-registry-tls" not found
May 25 11:17:41.139611 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.139569    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/21792ece-de04-4eb0-990c-e1adec1c79c5-ca-trust-extracted\") pod \"image-registry-7b9fd9b9cb-fpswn\" (UID: \"21792ece-de04-4eb0-990c-e1adec1c79c5\") " pod="openshift-image-registry/image-registry-7b9fd9b9cb-fpswn"
May 25 11:17:41.140048 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.139605    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-x5zjj\" (UniqueName: \"kubernetes.io/projected/21792ece-de04-4eb0-990c-e1adec1c79c5-kube-api-access-x5zjj\") pod \"image-registry-7b9fd9b9cb-fpswn\" (UID: \"21792ece-de04-4eb0-990c-e1adec1c79c5\") " pod="openshift-image-registry/image-registry-7b9fd9b9cb-fpswn"
May 25 11:17:41.140048 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:41.139627    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/21792ece-de04-4eb0-990c-e1adec1c79c5-registry-tls podName:21792ece-de04-4eb0-990c-e1adec1c79c5 nodeName:}" failed. No retries permitted until 2026-05-25 11:17:41.639608303 +0000 UTC m=+35.184785901 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/21792ece-de04-4eb0-990c-e1adec1c79c5-registry-tls") pod "image-registry-7b9fd9b9cb-fpswn" (UID: "21792ece-de04-4eb0-990c-e1adec1c79c5") : secret "image-registry-tls" not found
May 25 11:17:41.140048 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.139649    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9b9cd68b-6d33-40ae-b547-43f4ef7a2f5c-serving-cert\") pod \"kube-storage-version-migrator-operator-565644f49-88l9x\" (UID: \"9b9cd68b-6d33-40ae-b547-43f4ef7a2f5c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-88l9x"
May 25 11:17:41.140048 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.139817    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/82fb3b0a-798a-406b-99b4-548782582352-telemetry-config\") pod \"cluster-monitoring-operator-6c75959455-tfzq9\" (UID: \"82fb3b0a-798a-406b-99b4-548782582352\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-tfzq9"
May 25 11:17:41.140048 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.139857    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6cac71b-58b3-43ba-8a3a-efb15e35c401-service-ca-bundle\") pod \"router-default-846bb98c44-b4vb2\" (UID: \"c6cac71b-58b3-43ba-8a3a-efb15e35c401\") " pod="openshift-ingress/router-default-846bb98c44-b4vb2"
May 25 11:17:41.140048 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.139886    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c6cac71b-58b3-43ba-8a3a-efb15e35c401-metrics-certs\") pod \"router-default-846bb98c44-b4vb2\" (UID: \"c6cac71b-58b3-43ba-8a3a-efb15e35c401\") " pod="openshift-ingress/router-default-846bb98c44-b4vb2"
May 25 11:17:41.140048 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.139886    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/21792ece-de04-4eb0-990c-e1adec1c79c5-ca-trust-extracted\") pod \"image-registry-7b9fd9b9cb-fpswn\" (UID: \"21792ece-de04-4eb0-990c-e1adec1c79c5\") " pod="openshift-image-registry/image-registry-7b9fd9b9cb-fpswn"
May 25 11:17:41.140048 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.139921    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/c6cac71b-58b3-43ba-8a3a-efb15e35c401-stats-auth\") pod \"router-default-846bb98c44-b4vb2\" (UID: \"c6cac71b-58b3-43ba-8a3a-efb15e35c401\") " pod="openshift-ingress/router-default-846bb98c44-b4vb2"
May 25 11:17:41.140423 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.140130    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/21792ece-de04-4eb0-990c-e1adec1c79c5-trusted-ca\") pod \"image-registry-7b9fd9b9cb-fpswn\" (UID: \"21792ece-de04-4eb0-990c-e1adec1c79c5\") " pod="openshift-image-registry/image-registry-7b9fd9b9cb-fpswn"
May 25 11:17:41.143615 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.143594    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/21792ece-de04-4eb0-990c-e1adec1c79c5-image-registry-private-configuration\") pod \"image-registry-7b9fd9b9cb-fpswn\" (UID: \"21792ece-de04-4eb0-990c-e1adec1c79c5\") " pod="openshift-image-registry/image-registry-7b9fd9b9cb-fpswn"
May 25 11:17:41.143615 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.143604    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/21792ece-de04-4eb0-990c-e1adec1c79c5-installation-pull-secrets\") pod \"image-registry-7b9fd9b9cb-fpswn\" (UID: \"21792ece-de04-4eb0-990c-e1adec1c79c5\") " pod="openshift-image-registry/image-registry-7b9fd9b9cb-fpswn"
May 25 11:17:41.148452 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.148428    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-78bdfb888c-nd5f5"]
May 25 11:17:41.148584 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.148459    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57b5b68b8-clx9s"]
May 25 11:17:41.148584 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.148472    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-697587599-6vz7w"]
May 25 11:17:41.148584 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.148485    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7f44b6dd87-7b6rg"]
May 25 11:17:41.148584 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.148499    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-d4d8f94f9-8ktvq"]
May 25 11:17:41.148584 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.148522    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqrf7\" (UniqueName: \"kubernetes.io/projected/aa041c38-250b-4342-9951-7f04c9daf30c-kube-api-access-qqrf7\") pod \"volume-data-source-validator-5cf8c89d64-fwgjs\" (UID: \"aa041c38-250b-4342-9951-7f04c9daf30c\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-fwgjs"
May 25 11:17:41.148584 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.148541    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-66tdv"]
May 25 11:17:41.148842 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.148575    2561 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-d6xph"]
May 25 11:17:41.148842 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.148761    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-444mb"
May 25 11:17:41.149225 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.148978    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-clx9s"
May 25 11:17:41.149225 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.148979    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rl7hd"
May 25 11:17:41.149453 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.149329    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dq8s4"
May 25 11:17:41.150036 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.150015    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5zjj\" (UniqueName: \"kubernetes.io/projected/21792ece-de04-4eb0-990c-e1adec1c79c5-kube-api-access-x5zjj\") pod \"image-registry-7b9fd9b9cb-fpswn\" (UID: \"21792ece-de04-4eb0-990c-e1adec1c79c5\") " pod="openshift-image-registry/image-registry-7b9fd9b9cb-fpswn"
May 25 11:17:41.150134 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.150119    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/21792ece-de04-4eb0-990c-e1adec1c79c5-bound-sa-token\") pod \"image-registry-7b9fd9b9cb-fpswn\" (UID: \"21792ece-de04-4eb0-990c-e1adec1c79c5\") " pod="openshift-image-registry/image-registry-7b9fd9b9cb-fpswn"
May 25 11:17:41.151182 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.151167    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
May 25 11:17:41.151257 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.151230    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
May 25 11:17:41.151257 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.151244    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-x64h9\""
May 25 11:17:41.151527 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.151506    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
May 25 11:17:41.151527 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.151517    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
May 25 11:17:41.151662 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.151553    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
May 25 11:17:41.151759 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.151744    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
May 25 11:17:41.152090 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.152072    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-lf8xr\""
May 25 11:17:41.152174 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.152122    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-gn9nw\""
May 25 11:17:41.174912 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.174892    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-d6xph"]
May 25 11:17:41.175044 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.175028    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-d6xph"
May 25 11:17:41.177706 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.177540    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
May 25 11:17:41.178084 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.178051    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
May 25 11:17:41.178190 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.178118    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-nlbp4\""
May 25 11:17:41.240249 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.240219    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/c6cac71b-58b3-43ba-8a3a-efb15e35c401-stats-auth\") pod \"router-default-846bb98c44-b4vb2\" (UID: \"c6cac71b-58b3-43ba-8a3a-efb15e35c401\") " pod="openshift-ingress/router-default-846bb98c44-b4vb2"
May 25 11:17:41.240413 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.240262    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/1aff1fe9-36dd-4aed-8407-73bf7f9c439e-nginx-conf\") pod \"networking-console-plugin-d4d8f94f9-8ktvq\" (UID: \"1aff1fe9-36dd-4aed-8407-73bf7f9c439e\") " pod="openshift-network-console/networking-console-plugin-d4d8f94f9-8ktvq"
May 25 11:17:41.240413 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.240320    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-lqd8w\" (UniqueName: \"kubernetes.io/projected/8bf76398-20ba-499c-9103-c4017cdad681-kube-api-access-lqd8w\") pod \"cluster-samples-operator-697587599-6vz7w\" (UID: \"8bf76398-20ba-499c-9103-c4017cdad681\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-6vz7w"
May 25 11:17:41.240413 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.240375    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-lsrfh\" (UniqueName: \"kubernetes.io/projected/c6cac71b-58b3-43ba-8a3a-efb15e35c401-kube-api-access-lsrfh\") pod \"router-default-846bb98c44-b4vb2\" (UID: \"c6cac71b-58b3-43ba-8a3a-efb15e35c401\") " pod="openshift-ingress/router-default-846bb98c44-b4vb2"
May 25 11:17:41.240610 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.240582    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1aff1fe9-36dd-4aed-8407-73bf7f9c439e-networking-console-plugin-cert\") pod \"networking-console-plugin-d4d8f94f9-8ktvq\" (UID: \"1aff1fe9-36dd-4aed-8407-73bf7f9c439e\") " pod="openshift-network-console/networking-console-plugin-d4d8f94f9-8ktvq"
May 25 11:17:41.240726 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.240625    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkgrc\" (UniqueName: \"kubernetes.io/projected/13894946-8385-4b6f-a82a-4188f46d576c-kube-api-access-rkgrc\") pod \"insights-operator-78bdfb888c-nd5f5\" (UID: \"13894946-8385-4b6f-a82a-4188f46d576c\") " pod="openshift-insights/insights-operator-78bdfb888c-nd5f5"
May 25 11:17:41.240726 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.240648    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-52lrs\" (UniqueName: \"kubernetes.io/projected/10a66344-3304-49a1-bbc1-b885800f5acf-kube-api-access-52lrs\") pod \"console-operator-76dff48d59-9lj95\" (UID: \"10a66344-3304-49a1-bbc1-b885800f5acf\") " pod="openshift-console-operator/console-operator-76dff48d59-9lj95"
May 25 11:17:41.240726 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.240698    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a9377ad6-eac5-4df9-b7a2-b479c9c86277-config\") pod \"service-ca-operator-57b5b68b8-clx9s\" (UID: \"a9377ad6-eac5-4df9-b7a2-b479c9c86277\") " pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-clx9s"
May 25 11:17:41.240892 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.240770    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7s6sd\" (UniqueName: \"kubernetes.io/projected/039e2ad5-6518-454a-99b9-33221aaef30a-kube-api-access-7s6sd\") pod \"ingress-canary-66tdv\" (UID: \"039e2ad5-6518-454a-99b9-33221aaef30a\") " pod="openshift-ingress-canary/ingress-canary-66tdv"
May 25 11:17:41.240892 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.240823    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/10a66344-3304-49a1-bbc1-b885800f5acf-serving-cert\") pod \"console-operator-76dff48d59-9lj95\" (UID: \"10a66344-3304-49a1-bbc1-b885800f5acf\") " pod="openshift-console-operator/console-operator-76dff48d59-9lj95"
May 25 11:17:41.240973 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.240947    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/13894946-8385-4b6f-a82a-4188f46d576c-snapshots\") pod \"insights-operator-78bdfb888c-nd5f5\" (UID: \"13894946-8385-4b6f-a82a-4188f46d576c\") " pod="openshift-insights/insights-operator-78bdfb888c-nd5f5"
May 25 11:17:41.241023 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.240979    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqphd\" (UniqueName: \"kubernetes.io/projected/bf1cc7fa-a86f-43ee-ab68-3b6b57050611-kube-api-access-kqphd\") pod \"network-check-source-7f44b6dd87-7b6rg\" (UID: \"bf1cc7fa-a86f-43ee-ab68-3b6b57050611\") " pod="openshift-network-diagnostics/network-check-source-7f44b6dd87-7b6rg"
May 25 11:17:41.241023 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.241015    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/10a66344-3304-49a1-bbc1-b885800f5acf-trusted-ca\") pod \"console-operator-76dff48d59-9lj95\" (UID: \"10a66344-3304-49a1-bbc1-b885800f5acf\") " pod="openshift-console-operator/console-operator-76dff48d59-9lj95"
May 25 11:17:41.241121 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.241043    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/82fb3b0a-798a-406b-99b4-548782582352-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6c75959455-tfzq9\" (UID: \"82fb3b0a-798a-406b-99b4-548782582352\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-tfzq9"
May 25 11:17:41.241121 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.241086    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/13894946-8385-4b6f-a82a-4188f46d576c-tmp\") pod \"insights-operator-78bdfb888c-nd5f5\" (UID: \"13894946-8385-4b6f-a82a-4188f46d576c\") " pod="openshift-insights/insights-operator-78bdfb888c-nd5f5"
May 25 11:17:41.241121 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:41.241113    2561 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
May 25 11:17:41.241257 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.241135    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/82fb3b0a-798a-406b-99b4-548782582352-telemetry-config\") pod \"cluster-monitoring-operator-6c75959455-tfzq9\" (UID: \"82fb3b0a-798a-406b-99b4-548782582352\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-tfzq9"
May 25 11:17:41.241257 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:41.241170    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82fb3b0a-798a-406b-99b4-548782582352-cluster-monitoring-operator-tls podName:82fb3b0a-798a-406b-99b4-548782582352 nodeName:}" failed. No retries permitted until 2026-05-25 11:17:41.741150818 +0000 UTC m=+35.286328399 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/82fb3b0a-798a-406b-99b4-548782582352-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6c75959455-tfzq9" (UID: "82fb3b0a-798a-406b-99b4-548782582352") : secret "cluster-monitoring-operator-tls" not found
May 25 11:17:41.241257 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.241186    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6cac71b-58b3-43ba-8a3a-efb15e35c401-service-ca-bundle\") pod \"router-default-846bb98c44-b4vb2\" (UID: \"c6cac71b-58b3-43ba-8a3a-efb15e35c401\") " pod="openshift-ingress/router-default-846bb98c44-b4vb2"
May 25 11:17:41.241257 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.241218    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9b9cd68b-6d33-40ae-b547-43f4ef7a2f5c-config\") pod \"kube-storage-version-migrator-operator-565644f49-88l9x\" (UID: \"9b9cd68b-6d33-40ae-b547-43f4ef7a2f5c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-88l9x"
May 25 11:17:41.241257 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.241254    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a9377ad6-eac5-4df9-b7a2-b479c9c86277-serving-cert\") pod \"service-ca-operator-57b5b68b8-clx9s\" (UID: \"a9377ad6-eac5-4df9-b7a2-b479c9c86277\") " pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-clx9s"
May 25 11:17:41.242636 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.241302    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/10a66344-3304-49a1-bbc1-b885800f5acf-config\") pod \"console-operator-76dff48d59-9lj95\" (UID: \"10a66344-3304-49a1-bbc1-b885800f5acf\") " pod="openshift-console-operator/console-operator-76dff48d59-9lj95"
May 25 11:17:41.242636 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.241334    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/13894946-8385-4b6f-a82a-4188f46d576c-service-ca-bundle\") pod \"insights-operator-78bdfb888c-nd5f5\" (UID: \"13894946-8385-4b6f-a82a-4188f46d576c\") " pod="openshift-insights/insights-operator-78bdfb888c-nd5f5"
May 25 11:17:41.242636 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.241358    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/13894946-8385-4b6f-a82a-4188f46d576c-serving-cert\") pod \"insights-operator-78bdfb888c-nd5f5\" (UID: \"13894946-8385-4b6f-a82a-4188f46d576c\") " pod="openshift-insights/insights-operator-78bdfb888c-nd5f5"
May 25 11:17:41.242636 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.241395    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/8bf76398-20ba-499c-9103-c4017cdad681-samples-operator-tls\") pod \"cluster-samples-operator-697587599-6vz7w\" (UID: \"8bf76398-20ba-499c-9103-c4017cdad681\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-6vz7w"
May 25 11:17:41.242636 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.241424    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9g54c\" (UniqueName: \"kubernetes.io/projected/a9377ad6-eac5-4df9-b7a2-b479c9c86277-kube-api-access-9g54c\") pod \"service-ca-operator-57b5b68b8-clx9s\" (UID: \"a9377ad6-eac5-4df9-b7a2-b479c9c86277\") " pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-clx9s"
May 25 11:17:41.242636 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.241468    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-wjd5b\" (UniqueName: \"kubernetes.io/projected/82fb3b0a-798a-406b-99b4-548782582352-kube-api-access-wjd5b\") pod \"cluster-monitoring-operator-6c75959455-tfzq9\" (UID: \"82fb3b0a-798a-406b-99b4-548782582352\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-tfzq9"
May 25 11:17:41.242636 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.241496    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-rlgjm\" (UniqueName: \"kubernetes.io/projected/9b9cd68b-6d33-40ae-b547-43f4ef7a2f5c-kube-api-access-rlgjm\") pod \"kube-storage-version-migrator-operator-565644f49-88l9x\" (UID: \"9b9cd68b-6d33-40ae-b547-43f4ef7a2f5c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-88l9x"
May 25 11:17:41.242636 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.241529    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/c6cac71b-58b3-43ba-8a3a-efb15e35c401-default-certificate\") pod \"router-default-846bb98c44-b4vb2\" (UID: \"c6cac71b-58b3-43ba-8a3a-efb15e35c401\") " pod="openshift-ingress/router-default-846bb98c44-b4vb2"
May 25 11:17:41.242636 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.241555    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/039e2ad5-6518-454a-99b9-33221aaef30a-cert\") pod \"ingress-canary-66tdv\" (UID: \"039e2ad5-6518-454a-99b9-33221aaef30a\") " pod="openshift-ingress-canary/ingress-canary-66tdv"
May 25 11:17:41.242636 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.241581    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/13894946-8385-4b6f-a82a-4188f46d576c-trusted-ca-bundle\") pod \"insights-operator-78bdfb888c-nd5f5\" (UID: \"13894946-8385-4b6f-a82a-4188f46d576c\") " pod="openshift-insights/insights-operator-78bdfb888c-nd5f5"
May 25 11:17:41.242636 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.241616    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9b9cd68b-6d33-40ae-b547-43f4ef7a2f5c-serving-cert\") pod \"kube-storage-version-migrator-operator-565644f49-88l9x\" (UID: \"9b9cd68b-6d33-40ae-b547-43f4ef7a2f5c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-88l9x"
May 25 11:17:41.242636 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.241650    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c6cac71b-58b3-43ba-8a3a-efb15e35c401-metrics-certs\") pod \"router-default-846bb98c44-b4vb2\" (UID: \"c6cac71b-58b3-43ba-8a3a-efb15e35c401\") " pod="openshift-ingress/router-default-846bb98c44-b4vb2"
May 25 11:17:41.242636 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:41.241783    2561 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
May 25 11:17:41.242636 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:41.241838    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c6cac71b-58b3-43ba-8a3a-efb15e35c401-metrics-certs podName:c6cac71b-58b3-43ba-8a3a-efb15e35c401 nodeName:}" failed. No retries permitted until 2026-05-25 11:17:41.741822031 +0000 UTC m=+35.286999618 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c6cac71b-58b3-43ba-8a3a-efb15e35c401-metrics-certs") pod "router-default-846bb98c44-b4vb2" (UID: "c6cac71b-58b3-43ba-8a3a-efb15e35c401") : secret "router-metrics-certs-default" not found
May 25 11:17:41.242636 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.241879    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9b9cd68b-6d33-40ae-b547-43f4ef7a2f5c-config\") pod \"kube-storage-version-migrator-operator-565644f49-88l9x\" (UID: \"9b9cd68b-6d33-40ae-b547-43f4ef7a2f5c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-88l9x"
May 25 11:17:41.243391 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.241961    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/10a66344-3304-49a1-bbc1-b885800f5acf-trusted-ca\") pod \"console-operator-76dff48d59-9lj95\" (UID: \"10a66344-3304-49a1-bbc1-b885800f5acf\") " pod="openshift-console-operator/console-operator-76dff48d59-9lj95"
May 25 11:17:41.243391 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:41.241986    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c6cac71b-58b3-43ba-8a3a-efb15e35c401-service-ca-bundle podName:c6cac71b-58b3-43ba-8a3a-efb15e35c401 nodeName:}" failed. No retries permitted until 2026-05-25 11:17:41.741971619 +0000 UTC m=+35.287149216 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/c6cac71b-58b3-43ba-8a3a-efb15e35c401-service-ca-bundle") pod "router-default-846bb98c44-b4vb2" (UID: "c6cac71b-58b3-43ba-8a3a-efb15e35c401") : configmap references non-existent config key: service-ca.crt
May 25 11:17:41.243391 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:41.242305    2561 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
May 25 11:17:41.243391 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:41.242350    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8bf76398-20ba-499c-9103-c4017cdad681-samples-operator-tls podName:8bf76398-20ba-499c-9103-c4017cdad681 nodeName:}" failed. No retries permitted until 2026-05-25 11:17:41.742335709 +0000 UTC m=+35.287513305 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/8bf76398-20ba-499c-9103-c4017cdad681-samples-operator-tls") pod "cluster-samples-operator-697587599-6vz7w" (UID: "8bf76398-20ba-499c-9103-c4017cdad681") : secret "samples-operator-tls" not found
May 25 11:17:41.243391 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.242417    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/10a66344-3304-49a1-bbc1-b885800f5acf-config\") pod \"console-operator-76dff48d59-9lj95\" (UID: \"10a66344-3304-49a1-bbc1-b885800f5acf\") " pod="openshift-console-operator/console-operator-76dff48d59-9lj95"
May 25 11:17:41.243391 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.242580    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/82fb3b0a-798a-406b-99b4-548782582352-telemetry-config\") pod \"cluster-monitoring-operator-6c75959455-tfzq9\" (UID: \"82fb3b0a-798a-406b-99b4-548782582352\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-tfzq9"
May 25 11:17:41.243391 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.243263    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/c6cac71b-58b3-43ba-8a3a-efb15e35c401-stats-auth\") pod \"router-default-846bb98c44-b4vb2\" (UID: \"c6cac71b-58b3-43ba-8a3a-efb15e35c401\") " pod="openshift-ingress/router-default-846bb98c44-b4vb2"
May 25 11:17:41.244932 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.244897    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/10a66344-3304-49a1-bbc1-b885800f5acf-serving-cert\") pod \"console-operator-76dff48d59-9lj95\" (UID: \"10a66344-3304-49a1-bbc1-b885800f5acf\") " pod="openshift-console-operator/console-operator-76dff48d59-9lj95"
May 25 11:17:41.245024 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.244932    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9b9cd68b-6d33-40ae-b547-43f4ef7a2f5c-serving-cert\") pod \"kube-storage-version-migrator-operator-565644f49-88l9x\" (UID: \"9b9cd68b-6d33-40ae-b547-43f4ef7a2f5c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-88l9x"
May 25 11:17:41.245024 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.244966    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/c6cac71b-58b3-43ba-8a3a-efb15e35c401-default-certificate\") pod \"router-default-846bb98c44-b4vb2\" (UID: \"c6cac71b-58b3-43ba-8a3a-efb15e35c401\") " pod="openshift-ingress/router-default-846bb98c44-b4vb2"
May 25 11:17:41.245106 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.245027    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-fwgjs"
May 25 11:17:41.253417 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.253356    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-52lrs\" (UniqueName: \"kubernetes.io/projected/10a66344-3304-49a1-bbc1-b885800f5acf-kube-api-access-52lrs\") pod \"console-operator-76dff48d59-9lj95\" (UID: \"10a66344-3304-49a1-bbc1-b885800f5acf\") " pod="openshift-console-operator/console-operator-76dff48d59-9lj95"
May 25 11:17:41.254113 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.253879    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lsrfh\" (UniqueName: \"kubernetes.io/projected/c6cac71b-58b3-43ba-8a3a-efb15e35c401-kube-api-access-lsrfh\") pod \"router-default-846bb98c44-b4vb2\" (UID: \"c6cac71b-58b3-43ba-8a3a-efb15e35c401\") " pod="openshift-ingress/router-default-846bb98c44-b4vb2"
May 25 11:17:41.254113 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.254039    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjd5b\" (UniqueName: \"kubernetes.io/projected/82fb3b0a-798a-406b-99b4-548782582352-kube-api-access-wjd5b\") pod \"cluster-monitoring-operator-6c75959455-tfzq9\" (UID: \"82fb3b0a-798a-406b-99b4-548782582352\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-tfzq9"
May 25 11:17:41.254468 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.254431    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqd8w\" (UniqueName: \"kubernetes.io/projected/8bf76398-20ba-499c-9103-c4017cdad681-kube-api-access-lqd8w\") pod \"cluster-samples-operator-697587599-6vz7w\" (UID: \"8bf76398-20ba-499c-9103-c4017cdad681\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-6vz7w"
May 25 11:17:41.262923 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.262905    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rlgjm\" (UniqueName: \"kubernetes.io/projected/9b9cd68b-6d33-40ae-b547-43f4ef7a2f5c-kube-api-access-rlgjm\") pod \"kube-storage-version-migrator-operator-565644f49-88l9x\" (UID: \"9b9cd68b-6d33-40ae-b547-43f4ef7a2f5c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-88l9x"
May 25 11:17:41.312445 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.312370    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-76dff48d59-9lj95"
May 25 11:17:41.345058 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.341127    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-88l9x"
May 25 11:17:41.345058 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.343709    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ffe11530-9960-4b99-862e-093a4377bd45-config-volume\") pod \"dns-default-d6xph\" (UID: \"ffe11530-9960-4b99-862e-093a4377bd45\") " pod="openshift-dns/dns-default-d6xph"
May 25 11:17:41.345058 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.343747    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ffe11530-9960-4b99-862e-093a4377bd45-tmp-dir\") pod \"dns-default-d6xph\" (UID: \"ffe11530-9960-4b99-862e-093a4377bd45\") " pod="openshift-dns/dns-default-d6xph"
May 25 11:17:41.345058 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.343786    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-7s6sd\" (UniqueName: \"kubernetes.io/projected/039e2ad5-6518-454a-99b9-33221aaef30a-kube-api-access-7s6sd\") pod \"ingress-canary-66tdv\" (UID: \"039e2ad5-6518-454a-99b9-33221aaef30a\") " pod="openshift-ingress-canary/ingress-canary-66tdv"
May 25 11:17:41.345058 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.343837    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/13894946-8385-4b6f-a82a-4188f46d576c-snapshots\") pod \"insights-operator-78bdfb888c-nd5f5\" (UID: \"13894946-8385-4b6f-a82a-4188f46d576c\") " pod="openshift-insights/insights-operator-78bdfb888c-nd5f5"
May 25 11:17:41.345058 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.343866    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-kqphd\" (UniqueName: \"kubernetes.io/projected/bf1cc7fa-a86f-43ee-ab68-3b6b57050611-kube-api-access-kqphd\") pod \"network-check-source-7f44b6dd87-7b6rg\" (UID: \"bf1cc7fa-a86f-43ee-ab68-3b6b57050611\") " pod="openshift-network-diagnostics/network-check-source-7f44b6dd87-7b6rg"
May 25 11:17:41.345058 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.343909    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/13894946-8385-4b6f-a82a-4188f46d576c-tmp\") pod \"insights-operator-78bdfb888c-nd5f5\" (UID: \"13894946-8385-4b6f-a82a-4188f46d576c\") " pod="openshift-insights/insights-operator-78bdfb888c-nd5f5"
May 25 11:17:41.345058 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.343965    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ff4ss\" (UniqueName: \"kubernetes.io/projected/ffe11530-9960-4b99-862e-093a4377bd45-kube-api-access-ff4ss\") pod \"dns-default-d6xph\" (UID: \"ffe11530-9960-4b99-862e-093a4377bd45\") " pod="openshift-dns/dns-default-d6xph"
May 25 11:17:41.345058 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.343994    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a9377ad6-eac5-4df9-b7a2-b479c9c86277-serving-cert\") pod \"service-ca-operator-57b5b68b8-clx9s\" (UID: \"a9377ad6-eac5-4df9-b7a2-b479c9c86277\") " pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-clx9s"
May 25 11:17:41.345058 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.344028    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/13894946-8385-4b6f-a82a-4188f46d576c-service-ca-bundle\") pod \"insights-operator-78bdfb888c-nd5f5\" (UID: \"13894946-8385-4b6f-a82a-4188f46d576c\") " pod="openshift-insights/insights-operator-78bdfb888c-nd5f5"
May 25 11:17:41.345058 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.344057    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/13894946-8385-4b6f-a82a-4188f46d576c-serving-cert\") pod \"insights-operator-78bdfb888c-nd5f5\" (UID: \"13894946-8385-4b6f-a82a-4188f46d576c\") " pod="openshift-insights/insights-operator-78bdfb888c-nd5f5"
May 25 11:17:41.345058 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.344103    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-9g54c\" (UniqueName: \"kubernetes.io/projected/a9377ad6-eac5-4df9-b7a2-b479c9c86277-kube-api-access-9g54c\") pod \"service-ca-operator-57b5b68b8-clx9s\" (UID: \"a9377ad6-eac5-4df9-b7a2-b479c9c86277\") " pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-clx9s"
May 25 11:17:41.345058 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.344169    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/039e2ad5-6518-454a-99b9-33221aaef30a-cert\") pod \"ingress-canary-66tdv\" (UID: \"039e2ad5-6518-454a-99b9-33221aaef30a\") " pod="openshift-ingress-canary/ingress-canary-66tdv"
May 25 11:17:41.345058 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.344198    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/13894946-8385-4b6f-a82a-4188f46d576c-trusted-ca-bundle\") pod \"insights-operator-78bdfb888c-nd5f5\" (UID: \"13894946-8385-4b6f-a82a-4188f46d576c\") " pod="openshift-insights/insights-operator-78bdfb888c-nd5f5"
May 25 11:17:41.345058 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.344254    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ffe11530-9960-4b99-862e-093a4377bd45-metrics-tls\") pod \"dns-default-d6xph\" (UID: \"ffe11530-9960-4b99-862e-093a4377bd45\") " pod="openshift-dns/dns-default-d6xph"
May 25 11:17:41.345058 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.344307    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/1aff1fe9-36dd-4aed-8407-73bf7f9c439e-nginx-conf\") pod \"networking-console-plugin-d4d8f94f9-8ktvq\" (UID: \"1aff1fe9-36dd-4aed-8407-73bf7f9c439e\") " pod="openshift-network-console/networking-console-plugin-d4d8f94f9-8ktvq"
May 25 11:17:41.345058 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.344353    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1aff1fe9-36dd-4aed-8407-73bf7f9c439e-networking-console-plugin-cert\") pod \"networking-console-plugin-d4d8f94f9-8ktvq\" (UID: \"1aff1fe9-36dd-4aed-8407-73bf7f9c439e\") " pod="openshift-network-console/networking-console-plugin-d4d8f94f9-8ktvq"
May 25 11:17:41.346444 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.344388    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-rkgrc\" (UniqueName: \"kubernetes.io/projected/13894946-8385-4b6f-a82a-4188f46d576c-kube-api-access-rkgrc\") pod \"insights-operator-78bdfb888c-nd5f5\" (UID: \"13894946-8385-4b6f-a82a-4188f46d576c\") " pod="openshift-insights/insights-operator-78bdfb888c-nd5f5"
May 25 11:17:41.346444 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.344425    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a9377ad6-eac5-4df9-b7a2-b479c9c86277-config\") pod \"service-ca-operator-57b5b68b8-clx9s\" (UID: \"a9377ad6-eac5-4df9-b7a2-b479c9c86277\") " pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-clx9s"
May 25 11:17:41.346444 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.345077    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a9377ad6-eac5-4df9-b7a2-b479c9c86277-config\") pod \"service-ca-operator-57b5b68b8-clx9s\" (UID: \"a9377ad6-eac5-4df9-b7a2-b479c9c86277\") " pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-clx9s"
May 25 11:17:41.346444 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:41.345898    2561 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 25 11:17:41.346444 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:41.345971    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/039e2ad5-6518-454a-99b9-33221aaef30a-cert podName:039e2ad5-6518-454a-99b9-33221aaef30a nodeName:}" failed. No retries permitted until 2026-05-25 11:17:41.845952165 +0000 UTC m=+35.391129749 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/039e2ad5-6518-454a-99b9-33221aaef30a-cert") pod "ingress-canary-66tdv" (UID: "039e2ad5-6518-454a-99b9-33221aaef30a") : secret "canary-serving-cert" not found
May 25 11:17:41.346823 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.346752    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/1aff1fe9-36dd-4aed-8407-73bf7f9c439e-nginx-conf\") pod \"networking-console-plugin-d4d8f94f9-8ktvq\" (UID: \"1aff1fe9-36dd-4aed-8407-73bf7f9c439e\") " pod="openshift-network-console/networking-console-plugin-d4d8f94f9-8ktvq"
May 25 11:17:41.346878 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.346841    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/13894946-8385-4b6f-a82a-4188f46d576c-tmp\") pod \"insights-operator-78bdfb888c-nd5f5\" (UID: \"13894946-8385-4b6f-a82a-4188f46d576c\") " pod="openshift-insights/insights-operator-78bdfb888c-nd5f5"
May 25 11:17:41.347608 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:41.347024    2561 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
May 25 11:17:41.347608 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:41.347087    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1aff1fe9-36dd-4aed-8407-73bf7f9c439e-networking-console-plugin-cert podName:1aff1fe9-36dd-4aed-8407-73bf7f9c439e nodeName:}" failed. No retries permitted until 2026-05-25 11:17:41.84707075 +0000 UTC m=+35.392248332 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/1aff1fe9-36dd-4aed-8407-73bf7f9c439e-networking-console-plugin-cert") pod "networking-console-plugin-d4d8f94f9-8ktvq" (UID: "1aff1fe9-36dd-4aed-8407-73bf7f9c439e") : secret "networking-console-plugin-cert" not found
May 25 11:17:41.348310 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.348287    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/13894946-8385-4b6f-a82a-4188f46d576c-trusted-ca-bundle\") pod \"insights-operator-78bdfb888c-nd5f5\" (UID: \"13894946-8385-4b6f-a82a-4188f46d576c\") " pod="openshift-insights/insights-operator-78bdfb888c-nd5f5"
May 25 11:17:41.348417 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.348312    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/13894946-8385-4b6f-a82a-4188f46d576c-service-ca-bundle\") pod \"insights-operator-78bdfb888c-nd5f5\" (UID: \"13894946-8385-4b6f-a82a-4188f46d576c\") " pod="openshift-insights/insights-operator-78bdfb888c-nd5f5"
May 25 11:17:41.348647 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.348613    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/13894946-8385-4b6f-a82a-4188f46d576c-snapshots\") pod \"insights-operator-78bdfb888c-nd5f5\" (UID: \"13894946-8385-4b6f-a82a-4188f46d576c\") " pod="openshift-insights/insights-operator-78bdfb888c-nd5f5"
May 25 11:17:41.350304 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.350247    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a9377ad6-eac5-4df9-b7a2-b479c9c86277-serving-cert\") pod \"service-ca-operator-57b5b68b8-clx9s\" (UID: \"a9377ad6-eac5-4df9-b7a2-b479c9c86277\") " pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-clx9s"
May 25 11:17:41.352100 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.352063    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/13894946-8385-4b6f-a82a-4188f46d576c-serving-cert\") pod \"insights-operator-78bdfb888c-nd5f5\" (UID: \"13894946-8385-4b6f-a82a-4188f46d576c\") " pod="openshift-insights/insights-operator-78bdfb888c-nd5f5"
May 25 11:17:41.360623 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.360574    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9g54c\" (UniqueName: \"kubernetes.io/projected/a9377ad6-eac5-4df9-b7a2-b479c9c86277-kube-api-access-9g54c\") pod \"service-ca-operator-57b5b68b8-clx9s\" (UID: \"a9377ad6-eac5-4df9-b7a2-b479c9c86277\") " pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-clx9s"
May 25 11:17:41.361102 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.361049    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7s6sd\" (UniqueName: \"kubernetes.io/projected/039e2ad5-6518-454a-99b9-33221aaef30a-kube-api-access-7s6sd\") pod \"ingress-canary-66tdv\" (UID: \"039e2ad5-6518-454a-99b9-33221aaef30a\") " pod="openshift-ingress-canary/ingress-canary-66tdv"
May 25 11:17:41.369180 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.369082    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqphd\" (UniqueName: \"kubernetes.io/projected/bf1cc7fa-a86f-43ee-ab68-3b6b57050611-kube-api-access-kqphd\") pod \"network-check-source-7f44b6dd87-7b6rg\" (UID: \"bf1cc7fa-a86f-43ee-ab68-3b6b57050611\") " pod="openshift-network-diagnostics/network-check-source-7f44b6dd87-7b6rg"
May 25 11:17:41.370667 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.370575    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkgrc\" (UniqueName: \"kubernetes.io/projected/13894946-8385-4b6f-a82a-4188f46d576c-kube-api-access-rkgrc\") pod \"insights-operator-78bdfb888c-nd5f5\" (UID: \"13894946-8385-4b6f-a82a-4188f46d576c\") " pod="openshift-insights/insights-operator-78bdfb888c-nd5f5"
May 25 11:17:41.372537 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.372050    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-78bdfb888c-nd5f5"
May 25 11:17:41.418378 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.418135    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7f44b6dd87-7b6rg"
May 25 11:17:41.446522 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.445469    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ffe11530-9960-4b99-862e-093a4377bd45-config-volume\") pod \"dns-default-d6xph\" (UID: \"ffe11530-9960-4b99-862e-093a4377bd45\") " pod="openshift-dns/dns-default-d6xph"
May 25 11:17:41.446522 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.445515    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ffe11530-9960-4b99-862e-093a4377bd45-tmp-dir\") pod \"dns-default-d6xph\" (UID: \"ffe11530-9960-4b99-862e-093a4377bd45\") " pod="openshift-dns/dns-default-d6xph"
May 25 11:17:41.446522 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.445630    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-ff4ss\" (UniqueName: \"kubernetes.io/projected/ffe11530-9960-4b99-862e-093a4377bd45-kube-api-access-ff4ss\") pod \"dns-default-d6xph\" (UID: \"ffe11530-9960-4b99-862e-093a4377bd45\") " pod="openshift-dns/dns-default-d6xph"
May 25 11:17:41.446522 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.445743    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ffe11530-9960-4b99-862e-093a4377bd45-metrics-tls\") pod \"dns-default-d6xph\" (UID: \"ffe11530-9960-4b99-862e-093a4377bd45\") " pod="openshift-dns/dns-default-d6xph"
May 25 11:17:41.446522 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:41.445906    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 25 11:17:41.446522 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:41.445969    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ffe11530-9960-4b99-862e-093a4377bd45-metrics-tls podName:ffe11530-9960-4b99-862e-093a4377bd45 nodeName:}" failed. No retries permitted until 2026-05-25 11:17:41.945951178 +0000 UTC m=+35.491128782 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ffe11530-9960-4b99-862e-093a4377bd45-metrics-tls") pod "dns-default-d6xph" (UID: "ffe11530-9960-4b99-862e-093a4377bd45") : secret "dns-default-metrics-tls" not found
May 25 11:17:41.446522 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.446157    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ffe11530-9960-4b99-862e-093a4377bd45-config-volume\") pod \"dns-default-d6xph\" (UID: \"ffe11530-9960-4b99-862e-093a4377bd45\") " pod="openshift-dns/dns-default-d6xph"
May 25 11:17:41.446522 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.446477    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ffe11530-9960-4b99-862e-093a4377bd45-tmp-dir\") pod \"dns-default-d6xph\" (UID: \"ffe11530-9960-4b99-862e-093a4377bd45\") " pod="openshift-dns/dns-default-d6xph"
May 25 11:17:41.472579 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.470308    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-clx9s"
May 25 11:17:41.472579 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.471903    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-fwgjs"]
May 25 11:17:41.486576 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.486518    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ff4ss\" (UniqueName: \"kubernetes.io/projected/ffe11530-9960-4b99-862e-093a4377bd45-kube-api-access-ff4ss\") pod \"dns-default-d6xph\" (UID: \"ffe11530-9960-4b99-862e-093a4377bd45\") " pod="openshift-dns/dns-default-d6xph"
May 25 11:17:41.515991 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.515829    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-76dff48d59-9lj95"]
May 25 11:17:41.549445 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.549285    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-88l9x"]
May 25 11:17:41.554175 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:41.554142    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b9cd68b_6d33_40ae_b547_43f4ef7a2f5c.slice/crio-717756dfb2b0f8c45d2105987cf11de51440df6240928c2b1d7f77946b59d1c2 WatchSource:0}: Error finding container 717756dfb2b0f8c45d2105987cf11de51440df6240928c2b1d7f77946b59d1c2: Status 404 returned error can't find the container with id 717756dfb2b0f8c45d2105987cf11de51440df6240928c2b1d7f77946b59d1c2
May 25 11:17:41.596376 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.596346    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-78bdfb888c-nd5f5"]
May 25 11:17:41.598696 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:41.598665    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13894946_8385_4b6f_a82a_4188f46d576c.slice/crio-dd044be384914eef8990d43c4afb4bdf7adc8bd687737f8bb2cc9651a3b57614 WatchSource:0}: Error finding container dd044be384914eef8990d43c4afb4bdf7adc8bd687737f8bb2cc9651a3b57614: Status 404 returned error can't find the container with id dd044be384914eef8990d43c4afb4bdf7adc8bd687737f8bb2cc9651a3b57614
May 25 11:17:41.605431 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.605402    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7f44b6dd87-7b6rg"]
May 25 11:17:41.608962 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:41.608925    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf1cc7fa_a86f_43ee_ab68_3b6b57050611.slice/crio-cb88794b82f4b061b044b986a1ae5fae83019a5c1bf182eb2155c79bcf30aafe WatchSource:0}: Error finding container cb88794b82f4b061b044b986a1ae5fae83019a5c1bf182eb2155c79bcf30aafe: Status 404 returned error can't find the container with id cb88794b82f4b061b044b986a1ae5fae83019a5c1bf182eb2155c79bcf30aafe
May 25 11:17:41.647738 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.647707    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/21792ece-de04-4eb0-990c-e1adec1c79c5-registry-tls\") pod \"image-registry-7b9fd9b9cb-fpswn\" (UID: \"21792ece-de04-4eb0-990c-e1adec1c79c5\") " pod="openshift-image-registry/image-registry-7b9fd9b9cb-fpswn"
May 25 11:17:41.647890 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:41.647872    2561 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 25 11:17:41.647951 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:41.647894    2561 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7b9fd9b9cb-fpswn: secret "image-registry-tls" not found
May 25 11:17:41.647998 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:41.647963    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/21792ece-de04-4eb0-990c-e1adec1c79c5-registry-tls podName:21792ece-de04-4eb0-990c-e1adec1c79c5 nodeName:}" failed. No retries permitted until 2026-05-25 11:17:42.647940701 +0000 UTC m=+36.193118297 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/21792ece-de04-4eb0-990c-e1adec1c79c5-registry-tls") pod "image-registry-7b9fd9b9cb-fpswn" (UID: "21792ece-de04-4eb0-990c-e1adec1c79c5") : secret "image-registry-tls" not found
May 25 11:17:41.654954 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.654931    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57b5b68b8-clx9s"]
May 25 11:17:41.658415 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:17:41.658389    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9377ad6_eac5_4df9_b7a2_b479c9c86277.slice/crio-960d15f65fa540e8d3423ad90ff21cf9c56cf236a298cde203829ed074a1c2ba WatchSource:0}: Error finding container 960d15f65fa540e8d3423ad90ff21cf9c56cf236a298cde203829ed074a1c2ba: Status 404 returned error can't find the container with id 960d15f65fa540e8d3423ad90ff21cf9c56cf236a298cde203829ed074a1c2ba
May 25 11:17:41.748993 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.748971    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/82fb3b0a-798a-406b-99b4-548782582352-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6c75959455-tfzq9\" (UID: \"82fb3b0a-798a-406b-99b4-548782582352\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-tfzq9"
May 25 11:17:41.749124 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.749007    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6cac71b-58b3-43ba-8a3a-efb15e35c401-service-ca-bundle\") pod \"router-default-846bb98c44-b4vb2\" (UID: \"c6cac71b-58b3-43ba-8a3a-efb15e35c401\") " pod="openshift-ingress/router-default-846bb98c44-b4vb2"
May 25 11:17:41.749124 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.749036    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/8bf76398-20ba-499c-9103-c4017cdad681-samples-operator-tls\") pod \"cluster-samples-operator-697587599-6vz7w\" (UID: \"8bf76398-20ba-499c-9103-c4017cdad681\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-6vz7w"
May 25 11:17:41.749124 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.749077    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c6cac71b-58b3-43ba-8a3a-efb15e35c401-metrics-certs\") pod \"router-default-846bb98c44-b4vb2\" (UID: \"c6cac71b-58b3-43ba-8a3a-efb15e35c401\") " pod="openshift-ingress/router-default-846bb98c44-b4vb2"
May 25 11:17:41.749124 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:41.749114    2561 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
May 25 11:17:41.749254 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:41.749139    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c6cac71b-58b3-43ba-8a3a-efb15e35c401-service-ca-bundle podName:c6cac71b-58b3-43ba-8a3a-efb15e35c401 nodeName:}" failed. No retries permitted until 2026-05-25 11:17:42.749118431 +0000 UTC m=+36.294296028 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/c6cac71b-58b3-43ba-8a3a-efb15e35c401-service-ca-bundle") pod "router-default-846bb98c44-b4vb2" (UID: "c6cac71b-58b3-43ba-8a3a-efb15e35c401") : configmap references non-existent config key: service-ca.crt
May 25 11:17:41.749254 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:41.749158    2561 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
May 25 11:17:41.749254 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:41.749164    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82fb3b0a-798a-406b-99b4-548782582352-cluster-monitoring-operator-tls podName:82fb3b0a-798a-406b-99b4-548782582352 nodeName:}" failed. No retries permitted until 2026-05-25 11:17:42.749152278 +0000 UTC m=+36.294329860 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/82fb3b0a-798a-406b-99b4-548782582352-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6c75959455-tfzq9" (UID: "82fb3b0a-798a-406b-99b4-548782582352") : secret "cluster-monitoring-operator-tls" not found
May 25 11:17:41.749254 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:41.749177    2561 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
May 25 11:17:41.749254 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:41.749189    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c6cac71b-58b3-43ba-8a3a-efb15e35c401-metrics-certs podName:c6cac71b-58b3-43ba-8a3a-efb15e35c401 nodeName:}" failed. No retries permitted until 2026-05-25 11:17:42.749179802 +0000 UTC m=+36.294357383 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c6cac71b-58b3-43ba-8a3a-efb15e35c401-metrics-certs") pod "router-default-846bb98c44-b4vb2" (UID: "c6cac71b-58b3-43ba-8a3a-efb15e35c401") : secret "router-metrics-certs-default" not found
May 25 11:17:41.749254 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:41.749219    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8bf76398-20ba-499c-9103-c4017cdad681-samples-operator-tls podName:8bf76398-20ba-499c-9103-c4017cdad681 nodeName:}" failed. No retries permitted until 2026-05-25 11:17:42.749207151 +0000 UTC m=+36.294384738 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/8bf76398-20ba-499c-9103-c4017cdad681-samples-operator-tls") pod "cluster-samples-operator-697587599-6vz7w" (UID: "8bf76398-20ba-499c-9103-c4017cdad681") : secret "samples-operator-tls" not found
May 25 11:17:41.850326 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.850218    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/039e2ad5-6518-454a-99b9-33221aaef30a-cert\") pod \"ingress-canary-66tdv\" (UID: \"039e2ad5-6518-454a-99b9-33221aaef30a\") " pod="openshift-ingress-canary/ingress-canary-66tdv"
May 25 11:17:41.850923 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.850339    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1aff1fe9-36dd-4aed-8407-73bf7f9c439e-networking-console-plugin-cert\") pod \"networking-console-plugin-d4d8f94f9-8ktvq\" (UID: \"1aff1fe9-36dd-4aed-8407-73bf7f9c439e\") " pod="openshift-network-console/networking-console-plugin-d4d8f94f9-8ktvq"
May 25 11:17:41.850923 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:41.850388    2561 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 25 11:17:41.850923 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:41.850457    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/039e2ad5-6518-454a-99b9-33221aaef30a-cert podName:039e2ad5-6518-454a-99b9-33221aaef30a nodeName:}" failed. No retries permitted until 2026-05-25 11:17:42.850440956 +0000 UTC m=+36.395618551 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/039e2ad5-6518-454a-99b9-33221aaef30a-cert") pod "ingress-canary-66tdv" (UID: "039e2ad5-6518-454a-99b9-33221aaef30a") : secret "canary-serving-cert" not found
May 25 11:17:41.850923 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:41.850477    2561 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
May 25 11:17:41.850923 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:41.850534    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1aff1fe9-36dd-4aed-8407-73bf7f9c439e-networking-console-plugin-cert podName:1aff1fe9-36dd-4aed-8407-73bf7f9c439e nodeName:}" failed. No retries permitted until 2026-05-25 11:17:42.850520044 +0000 UTC m=+36.395697625 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/1aff1fe9-36dd-4aed-8407-73bf7f9c439e-networking-console-plugin-cert") pod "networking-console-plugin-d4d8f94f9-8ktvq" (UID: "1aff1fe9-36dd-4aed-8407-73bf7f9c439e") : secret "networking-console-plugin-cert" not found
May 25 11:17:41.951202 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:41.951172    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ffe11530-9960-4b99-862e-093a4377bd45-metrics-tls\") pod \"dns-default-d6xph\" (UID: \"ffe11530-9960-4b99-862e-093a4377bd45\") " pod="openshift-dns/dns-default-d6xph"
May 25 11:17:41.951364 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:41.951320    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 25 11:17:41.951415 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:41.951383    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ffe11530-9960-4b99-862e-093a4377bd45-metrics-tls podName:ffe11530-9960-4b99-862e-093a4377bd45 nodeName:}" failed. No retries permitted until 2026-05-25 11:17:42.951366088 +0000 UTC m=+36.496543669 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ffe11530-9960-4b99-862e-093a4377bd45-metrics-tls") pod "dns-default-d6xph" (UID: "ffe11530-9960-4b99-862e-093a4377bd45") : secret "dns-default-metrics-tls" not found
May 25 11:17:42.305078 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:42.304999    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-clx9s" event={"ID":"a9377ad6-eac5-4df9-b7a2-b479c9c86277","Type":"ContainerStarted","Data":"960d15f65fa540e8d3423ad90ff21cf9c56cf236a298cde203829ed074a1c2ba"}
May 25 11:17:42.306112 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:42.306074    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7f44b6dd87-7b6rg" event={"ID":"bf1cc7fa-a86f-43ee-ab68-3b6b57050611","Type":"ContainerStarted","Data":"cb88794b82f4b061b044b986a1ae5fae83019a5c1bf182eb2155c79bcf30aafe"}
May 25 11:17:42.307192 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:42.307164    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-78bdfb888c-nd5f5" event={"ID":"13894946-8385-4b6f-a82a-4188f46d576c","Type":"ContainerStarted","Data":"dd044be384914eef8990d43c4afb4bdf7adc8bd687737f8bb2cc9651a3b57614"}
May 25 11:17:42.308237 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:42.308215    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-88l9x" event={"ID":"9b9cd68b-6d33-40ae-b547-43f4ef7a2f5c","Type":"ContainerStarted","Data":"717756dfb2b0f8c45d2105987cf11de51440df6240928c2b1d7f77946b59d1c2"}
May 25 11:17:42.309303 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:42.309280    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-76dff48d59-9lj95" event={"ID":"10a66344-3304-49a1-bbc1-b885800f5acf","Type":"ContainerStarted","Data":"6f0d0882bac835199cb7b5290ffa1ab683a48bebb58e1317a17a7ac563b650b6"}
May 25 11:17:42.310300 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:42.310255    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-fwgjs" event={"ID":"aa041c38-250b-4342-9951-7f04c9daf30c","Type":"ContainerStarted","Data":"38f07e874150f6ab4729b52058eaaa945caac9687a91eeabe4f5a67e9d565029"}
May 25 11:17:42.658857 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:42.658815    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/21792ece-de04-4eb0-990c-e1adec1c79c5-registry-tls\") pod \"image-registry-7b9fd9b9cb-fpswn\" (UID: \"21792ece-de04-4eb0-990c-e1adec1c79c5\") " pod="openshift-image-registry/image-registry-7b9fd9b9cb-fpswn"
May 25 11:17:42.659016 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:42.658991    2561 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 25 11:17:42.659076 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:42.659017    2561 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7b9fd9b9cb-fpswn: secret "image-registry-tls" not found
May 25 11:17:42.659117 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:42.659081    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/21792ece-de04-4eb0-990c-e1adec1c79c5-registry-tls podName:21792ece-de04-4eb0-990c-e1adec1c79c5 nodeName:}" failed. No retries permitted until 2026-05-25 11:17:44.659061454 +0000 UTC m=+38.204239039 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/21792ece-de04-4eb0-990c-e1adec1c79c5-registry-tls") pod "image-registry-7b9fd9b9cb-fpswn" (UID: "21792ece-de04-4eb0-990c-e1adec1c79c5") : secret "image-registry-tls" not found
May 25 11:17:42.759862 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:42.759821    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/82fb3b0a-798a-406b-99b4-548782582352-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6c75959455-tfzq9\" (UID: \"82fb3b0a-798a-406b-99b4-548782582352\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-tfzq9"
May 25 11:17:42.760006 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:42.759886    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6cac71b-58b3-43ba-8a3a-efb15e35c401-service-ca-bundle\") pod \"router-default-846bb98c44-b4vb2\" (UID: \"c6cac71b-58b3-43ba-8a3a-efb15e35c401\") " pod="openshift-ingress/router-default-846bb98c44-b4vb2"
May 25 11:17:42.760006 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:42.759934    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/8bf76398-20ba-499c-9103-c4017cdad681-samples-operator-tls\") pod \"cluster-samples-operator-697587599-6vz7w\" (UID: \"8bf76398-20ba-499c-9103-c4017cdad681\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-6vz7w"
May 25 11:17:42.760006 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:42.759981    2561 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
May 25 11:17:42.760156 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:42.760050    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82fb3b0a-798a-406b-99b4-548782582352-cluster-monitoring-operator-tls podName:82fb3b0a-798a-406b-99b4-548782582352 nodeName:}" failed. No retries permitted until 2026-05-25 11:17:44.760031134 +0000 UTC m=+38.305208725 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/82fb3b0a-798a-406b-99b4-548782582352-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6c75959455-tfzq9" (UID: "82fb3b0a-798a-406b-99b4-548782582352") : secret "cluster-monitoring-operator-tls" not found
May 25 11:17:42.760156 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:42.760055    2561 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
May 25 11:17:42.760156 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:42.760067    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c6cac71b-58b3-43ba-8a3a-efb15e35c401-service-ca-bundle podName:c6cac71b-58b3-43ba-8a3a-efb15e35c401 nodeName:}" failed. No retries permitted until 2026-05-25 11:17:44.760058947 +0000 UTC m=+38.305236528 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/c6cac71b-58b3-43ba-8a3a-efb15e35c401-service-ca-bundle") pod "router-default-846bb98c44-b4vb2" (UID: "c6cac71b-58b3-43ba-8a3a-efb15e35c401") : configmap references non-existent config key: service-ca.crt
May 25 11:17:42.760156 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:42.760114    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8bf76398-20ba-499c-9103-c4017cdad681-samples-operator-tls podName:8bf76398-20ba-499c-9103-c4017cdad681 nodeName:}" failed. No retries permitted until 2026-05-25 11:17:44.760096478 +0000 UTC m=+38.305274061 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/8bf76398-20ba-499c-9103-c4017cdad681-samples-operator-tls") pod "cluster-samples-operator-697587599-6vz7w" (UID: "8bf76398-20ba-499c-9103-c4017cdad681") : secret "samples-operator-tls" not found
May 25 11:17:42.760156 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:42.760133    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c6cac71b-58b3-43ba-8a3a-efb15e35c401-metrics-certs\") pod \"router-default-846bb98c44-b4vb2\" (UID: \"c6cac71b-58b3-43ba-8a3a-efb15e35c401\") " pod="openshift-ingress/router-default-846bb98c44-b4vb2"
May 25 11:17:42.760384 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:42.760255    2561 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
May 25 11:17:42.760384 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:42.760309    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c6cac71b-58b3-43ba-8a3a-efb15e35c401-metrics-certs podName:c6cac71b-58b3-43ba-8a3a-efb15e35c401 nodeName:}" failed. No retries permitted until 2026-05-25 11:17:44.760297925 +0000 UTC m=+38.305475509 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c6cac71b-58b3-43ba-8a3a-efb15e35c401-metrics-certs") pod "router-default-846bb98c44-b4vb2" (UID: "c6cac71b-58b3-43ba-8a3a-efb15e35c401") : secret "router-metrics-certs-default" not found
May 25 11:17:42.860795 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:42.860764    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/039e2ad5-6518-454a-99b9-33221aaef30a-cert\") pod \"ingress-canary-66tdv\" (UID: \"039e2ad5-6518-454a-99b9-33221aaef30a\") " pod="openshift-ingress-canary/ingress-canary-66tdv"
May 25 11:17:42.861193 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:42.860902    2561 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 25 11:17:42.861193 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:42.860934    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1aff1fe9-36dd-4aed-8407-73bf7f9c439e-networking-console-plugin-cert\") pod \"networking-console-plugin-d4d8f94f9-8ktvq\" (UID: \"1aff1fe9-36dd-4aed-8407-73bf7f9c439e\") " pod="openshift-network-console/networking-console-plugin-d4d8f94f9-8ktvq"
May 25 11:17:42.861193 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:42.860965    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/039e2ad5-6518-454a-99b9-33221aaef30a-cert podName:039e2ad5-6518-454a-99b9-33221aaef30a nodeName:}" failed. No retries permitted until 2026-05-25 11:17:44.860947066 +0000 UTC m=+38.406124653 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/039e2ad5-6518-454a-99b9-33221aaef30a-cert") pod "ingress-canary-66tdv" (UID: "039e2ad5-6518-454a-99b9-33221aaef30a") : secret "canary-serving-cert" not found
May 25 11:17:42.861193 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:42.861024    2561 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
May 25 11:17:42.861193 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:42.861084    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1aff1fe9-36dd-4aed-8407-73bf7f9c439e-networking-console-plugin-cert podName:1aff1fe9-36dd-4aed-8407-73bf7f9c439e nodeName:}" failed. No retries permitted until 2026-05-25 11:17:44.86106642 +0000 UTC m=+38.406244014 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/1aff1fe9-36dd-4aed-8407-73bf7f9c439e-networking-console-plugin-cert") pod "networking-console-plugin-d4d8f94f9-8ktvq" (UID: "1aff1fe9-36dd-4aed-8407-73bf7f9c439e") : secret "networking-console-plugin-cert" not found
May 25 11:17:42.961963 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:42.961884    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ffe11530-9960-4b99-862e-093a4377bd45-metrics-tls\") pod \"dns-default-d6xph\" (UID: \"ffe11530-9960-4b99-862e-093a4377bd45\") " pod="openshift-dns/dns-default-d6xph"
May 25 11:17:42.962082 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:42.961991    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 25 11:17:42.962082 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:42.962053    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ffe11530-9960-4b99-862e-093a4377bd45-metrics-tls podName:ffe11530-9960-4b99-862e-093a4377bd45 nodeName:}" failed. No retries permitted until 2026-05-25 11:17:44.962035874 +0000 UTC m=+38.507213455 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ffe11530-9960-4b99-862e-093a4377bd45-metrics-tls") pod "dns-default-d6xph" (UID: "ffe11530-9960-4b99-862e-093a4377bd45") : secret "dns-default-metrics-tls" not found
May 25 11:17:44.327585 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:44.326586    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dmzql" event={"ID":"7c97daaa-b9c1-4ee0-ade1-e8de392e496f","Type":"ContainerStarted","Data":"dc4fcf995fd8f5831402115a5a0935cd4750fab8a1f711eeba865ef7545374be"}
May 25 11:17:44.679692 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:44.679649    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3e92c2a7-1021-4ee8-aded-8ae5ddc6b81c-original-pull-secret\") pod \"global-pull-secret-syncer-dq8s4\" (UID: \"3e92c2a7-1021-4ee8-aded-8ae5ddc6b81c\") " pod="kube-system/global-pull-secret-syncer-dq8s4"
May 25 11:17:44.679867 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:44.679787    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/21792ece-de04-4eb0-990c-e1adec1c79c5-registry-tls\") pod \"image-registry-7b9fd9b9cb-fpswn\" (UID: \"21792ece-de04-4eb0-990c-e1adec1c79c5\") " pod="openshift-image-registry/image-registry-7b9fd9b9cb-fpswn"
May 25 11:17:44.679981 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:44.679964    2561 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 25 11:17:44.680041 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:44.679985    2561 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7b9fd9b9cb-fpswn: secret "image-registry-tls" not found
May 25 11:17:44.680097 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:44.680042    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/21792ece-de04-4eb0-990c-e1adec1c79c5-registry-tls podName:21792ece-de04-4eb0-990c-e1adec1c79c5 nodeName:}" failed. No retries permitted until 2026-05-25 11:17:48.680022408 +0000 UTC m=+42.225200016 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/21792ece-de04-4eb0-990c-e1adec1c79c5-registry-tls") pod "image-registry-7b9fd9b9cb-fpswn" (UID: "21792ece-de04-4eb0-990c-e1adec1c79c5") : secret "image-registry-tls" not found
May 25 11:17:44.689169 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:44.688935    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3e92c2a7-1021-4ee8-aded-8ae5ddc6b81c-original-pull-secret\") pod \"global-pull-secret-syncer-dq8s4\" (UID: \"3e92c2a7-1021-4ee8-aded-8ae5ddc6b81c\") " pod="kube-system/global-pull-secret-syncer-dq8s4"
May 25 11:17:44.781926 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:44.780827    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c6cac71b-58b3-43ba-8a3a-efb15e35c401-metrics-certs\") pod \"router-default-846bb98c44-b4vb2\" (UID: \"c6cac71b-58b3-43ba-8a3a-efb15e35c401\") " pod="openshift-ingress/router-default-846bb98c44-b4vb2"
May 25 11:17:44.781926 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:44.780939    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/82fb3b0a-798a-406b-99b4-548782582352-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6c75959455-tfzq9\" (UID: \"82fb3b0a-798a-406b-99b4-548782582352\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-tfzq9"
May 25 11:17:44.781926 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:44.780981    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6cac71b-58b3-43ba-8a3a-efb15e35c401-service-ca-bundle\") pod \"router-default-846bb98c44-b4vb2\" (UID: \"c6cac71b-58b3-43ba-8a3a-efb15e35c401\") " pod="openshift-ingress/router-default-846bb98c44-b4vb2"
May 25 11:17:44.781926 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:44.781029    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/8bf76398-20ba-499c-9103-c4017cdad681-samples-operator-tls\") pod \"cluster-samples-operator-697587599-6vz7w\" (UID: \"8bf76398-20ba-499c-9103-c4017cdad681\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-6vz7w"
May 25 11:17:44.781926 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:44.781248    2561 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
May 25 11:17:44.781926 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:44.781331    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8bf76398-20ba-499c-9103-c4017cdad681-samples-operator-tls podName:8bf76398-20ba-499c-9103-c4017cdad681 nodeName:}" failed. No retries permitted until 2026-05-25 11:17:48.781310308 +0000 UTC m=+42.326487904 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/8bf76398-20ba-499c-9103-c4017cdad681-samples-operator-tls") pod "cluster-samples-operator-697587599-6vz7w" (UID: "8bf76398-20ba-499c-9103-c4017cdad681") : secret "samples-operator-tls" not found
May 25 11:17:44.781926 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:44.781698    2561 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
May 25 11:17:44.781926 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:44.781746    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c6cac71b-58b3-43ba-8a3a-efb15e35c401-metrics-certs podName:c6cac71b-58b3-43ba-8a3a-efb15e35c401 nodeName:}" failed. No retries permitted until 2026-05-25 11:17:48.781732249 +0000 UTC m=+42.326909850 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c6cac71b-58b3-43ba-8a3a-efb15e35c401-metrics-certs") pod "router-default-846bb98c44-b4vb2" (UID: "c6cac71b-58b3-43ba-8a3a-efb15e35c401") : secret "router-metrics-certs-default" not found
May 25 11:17:44.781926 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:44.781802    2561 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
May 25 11:17:44.781926 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:44.781835    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82fb3b0a-798a-406b-99b4-548782582352-cluster-monitoring-operator-tls podName:82fb3b0a-798a-406b-99b4-548782582352 nodeName:}" failed. No retries permitted until 2026-05-25 11:17:48.781823881 +0000 UTC m=+42.327001479 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/82fb3b0a-798a-406b-99b4-548782582352-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6c75959455-tfzq9" (UID: "82fb3b0a-798a-406b-99b4-548782582352") : secret "cluster-monitoring-operator-tls" not found
May 25 11:17:44.781926 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:44.781896    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c6cac71b-58b3-43ba-8a3a-efb15e35c401-service-ca-bundle podName:c6cac71b-58b3-43ba-8a3a-efb15e35c401 nodeName:}" failed. No retries permitted until 2026-05-25 11:17:48.781886874 +0000 UTC m=+42.327064473 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/c6cac71b-58b3-43ba-8a3a-efb15e35c401-service-ca-bundle") pod "router-default-846bb98c44-b4vb2" (UID: "c6cac71b-58b3-43ba-8a3a-efb15e35c401") : configmap references non-existent config key: service-ca.crt
May 25 11:17:44.783000 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:44.782666    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dq8s4"
May 25 11:17:44.883020 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:44.882766    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/039e2ad5-6518-454a-99b9-33221aaef30a-cert\") pod \"ingress-canary-66tdv\" (UID: \"039e2ad5-6518-454a-99b9-33221aaef30a\") " pod="openshift-ingress-canary/ingress-canary-66tdv"
May 25 11:17:44.883020 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:44.882863    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1aff1fe9-36dd-4aed-8407-73bf7f9c439e-networking-console-plugin-cert\") pod \"networking-console-plugin-d4d8f94f9-8ktvq\" (UID: \"1aff1fe9-36dd-4aed-8407-73bf7f9c439e\") " pod="openshift-network-console/networking-console-plugin-d4d8f94f9-8ktvq"
May 25 11:17:44.883233 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:44.883131    2561 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
May 25 11:17:44.883390 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:44.883316    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1aff1fe9-36dd-4aed-8407-73bf7f9c439e-networking-console-plugin-cert podName:1aff1fe9-36dd-4aed-8407-73bf7f9c439e nodeName:}" failed. No retries permitted until 2026-05-25 11:17:48.883293904 +0000 UTC m=+42.428471502 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/1aff1fe9-36dd-4aed-8407-73bf7f9c439e-networking-console-plugin-cert") pod "networking-console-plugin-d4d8f94f9-8ktvq" (UID: "1aff1fe9-36dd-4aed-8407-73bf7f9c439e") : secret "networking-console-plugin-cert" not found
May 25 11:17:44.883780 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:44.883763    2561 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 25 11:17:44.883868 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:44.883821    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/039e2ad5-6518-454a-99b9-33221aaef30a-cert podName:039e2ad5-6518-454a-99b9-33221aaef30a nodeName:}" failed. No retries permitted until 2026-05-25 11:17:48.883803623 +0000 UTC m=+42.428981205 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/039e2ad5-6518-454a-99b9-33221aaef30a-cert") pod "ingress-canary-66tdv" (UID: "039e2ad5-6518-454a-99b9-33221aaef30a") : secret "canary-serving-cert" not found
May 25 11:17:44.985151 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:44.984561    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ffe11530-9960-4b99-862e-093a4377bd45-metrics-tls\") pod \"dns-default-d6xph\" (UID: \"ffe11530-9960-4b99-862e-093a4377bd45\") " pod="openshift-dns/dns-default-d6xph"
May 25 11:17:44.985151 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:44.984720    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 25 11:17:44.985151 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:44.984780    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ffe11530-9960-4b99-862e-093a4377bd45-metrics-tls podName:ffe11530-9960-4b99-862e-093a4377bd45 nodeName:}" failed. No retries permitted until 2026-05-25 11:17:48.984761517 +0000 UTC m=+42.529939121 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ffe11530-9960-4b99-862e-093a4377bd45-metrics-tls") pod "dns-default-d6xph" (UID: "ffe11530-9960-4b99-862e-093a4377bd45") : secret "dns-default-metrics-tls" not found
May 25 11:17:44.985641 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:44.985573    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-dq8s4"]
May 25 11:17:45.334447 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:45.334371    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-dq8s4" event={"ID":"3e92c2a7-1021-4ee8-aded-8ae5ddc6b81c","Type":"ContainerStarted","Data":"9ef4f85e9164ef806c1f448a729b513316feaa39602d5d28e9d97b576e8930de"}
May 25 11:17:45.345734 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:45.345699    2561 generic.go:358] "Generic (PLEG): container finished" podID="7c97daaa-b9c1-4ee0-ade1-e8de392e496f" containerID="dc4fcf995fd8f5831402115a5a0935cd4750fab8a1f711eeba865ef7545374be" exitCode=0
May 25 11:17:45.345888 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:45.345747    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dmzql" event={"ID":"7c97daaa-b9c1-4ee0-ade1-e8de392e496f","Type":"ContainerDied","Data":"dc4fcf995fd8f5831402115a5a0935cd4750fab8a1f711eeba865ef7545374be"}
May 25 11:17:48.720993 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:48.720953    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/21792ece-de04-4eb0-990c-e1adec1c79c5-registry-tls\") pod \"image-registry-7b9fd9b9cb-fpswn\" (UID: \"21792ece-de04-4eb0-990c-e1adec1c79c5\") " pod="openshift-image-registry/image-registry-7b9fd9b9cb-fpswn"
May 25 11:17:48.721485 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:48.721103    2561 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 25 11:17:48.721485 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:48.721122    2561 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7b9fd9b9cb-fpswn: secret "image-registry-tls" not found
May 25 11:17:48.721485 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:48.721187    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/21792ece-de04-4eb0-990c-e1adec1c79c5-registry-tls podName:21792ece-de04-4eb0-990c-e1adec1c79c5 nodeName:}" failed. No retries permitted until 2026-05-25 11:17:56.721172268 +0000 UTC m=+50.266349852 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/21792ece-de04-4eb0-990c-e1adec1c79c5-registry-tls") pod "image-registry-7b9fd9b9cb-fpswn" (UID: "21792ece-de04-4eb0-990c-e1adec1c79c5") : secret "image-registry-tls" not found
May 25 11:17:48.821838 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:48.821799    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/8bf76398-20ba-499c-9103-c4017cdad681-samples-operator-tls\") pod \"cluster-samples-operator-697587599-6vz7w\" (UID: \"8bf76398-20ba-499c-9103-c4017cdad681\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-6vz7w"
May 25 11:17:48.821985 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:48.821891    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c6cac71b-58b3-43ba-8a3a-efb15e35c401-metrics-certs\") pod \"router-default-846bb98c44-b4vb2\" (UID: \"c6cac71b-58b3-43ba-8a3a-efb15e35c401\") " pod="openshift-ingress/router-default-846bb98c44-b4vb2"
May 25 11:17:48.821985 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:48.821944    2561 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
May 25 11:17:48.822106 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:48.822000    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/82fb3b0a-798a-406b-99b4-548782582352-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6c75959455-tfzq9\" (UID: \"82fb3b0a-798a-406b-99b4-548782582352\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-tfzq9"
May 25 11:17:48.822106 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:48.822023    2561 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
May 25 11:17:48.822106 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:48.822043    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8bf76398-20ba-499c-9103-c4017cdad681-samples-operator-tls podName:8bf76398-20ba-499c-9103-c4017cdad681 nodeName:}" failed. No retries permitted until 2026-05-25 11:17:56.822021009 +0000 UTC m=+50.367198610 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/8bf76398-20ba-499c-9103-c4017cdad681-samples-operator-tls") pod "cluster-samples-operator-697587599-6vz7w" (UID: "8bf76398-20ba-499c-9103-c4017cdad681") : secret "samples-operator-tls" not found
May 25 11:17:48.822106 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:48.822074    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c6cac71b-58b3-43ba-8a3a-efb15e35c401-metrics-certs podName:c6cac71b-58b3-43ba-8a3a-efb15e35c401 nodeName:}" failed. No retries permitted until 2026-05-25 11:17:56.822057991 +0000 UTC m=+50.367235642 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c6cac71b-58b3-43ba-8a3a-efb15e35c401-metrics-certs") pod "router-default-846bb98c44-b4vb2" (UID: "c6cac71b-58b3-43ba-8a3a-efb15e35c401") : secret "router-metrics-certs-default" not found
May 25 11:17:48.822106 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:48.822086    2561 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
May 25 11:17:48.822292 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:48.822109    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6cac71b-58b3-43ba-8a3a-efb15e35c401-service-ca-bundle\") pod \"router-default-846bb98c44-b4vb2\" (UID: \"c6cac71b-58b3-43ba-8a3a-efb15e35c401\") " pod="openshift-ingress/router-default-846bb98c44-b4vb2"
May 25 11:17:48.822292 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:48.822131    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82fb3b0a-798a-406b-99b4-548782582352-cluster-monitoring-operator-tls podName:82fb3b0a-798a-406b-99b4-548782582352 nodeName:}" failed. No retries permitted until 2026-05-25 11:17:56.822117843 +0000 UTC m=+50.367295427 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/82fb3b0a-798a-406b-99b4-548782582352-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6c75959455-tfzq9" (UID: "82fb3b0a-798a-406b-99b4-548782582352") : secret "cluster-monitoring-operator-tls" not found
May 25 11:17:48.822292 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:48.822166    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c6cac71b-58b3-43ba-8a3a-efb15e35c401-service-ca-bundle podName:c6cac71b-58b3-43ba-8a3a-efb15e35c401 nodeName:}" failed. No retries permitted until 2026-05-25 11:17:56.822156939 +0000 UTC m=+50.367334540 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/c6cac71b-58b3-43ba-8a3a-efb15e35c401-service-ca-bundle") pod "router-default-846bb98c44-b4vb2" (UID: "c6cac71b-58b3-43ba-8a3a-efb15e35c401") : configmap references non-existent config key: service-ca.crt
May 25 11:17:48.923236 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:48.923205    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1aff1fe9-36dd-4aed-8407-73bf7f9c439e-networking-console-plugin-cert\") pod \"networking-console-plugin-d4d8f94f9-8ktvq\" (UID: \"1aff1fe9-36dd-4aed-8407-73bf7f9c439e\") " pod="openshift-network-console/networking-console-plugin-d4d8f94f9-8ktvq"
May 25 11:17:48.923419 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:48.923365    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/039e2ad5-6518-454a-99b9-33221aaef30a-cert\") pod \"ingress-canary-66tdv\" (UID: \"039e2ad5-6518-454a-99b9-33221aaef30a\") " pod="openshift-ingress-canary/ingress-canary-66tdv"
May 25 11:17:48.923419 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:48.923385    2561 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
May 25 11:17:48.923530 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:48.923463    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1aff1fe9-36dd-4aed-8407-73bf7f9c439e-networking-console-plugin-cert podName:1aff1fe9-36dd-4aed-8407-73bf7f9c439e nodeName:}" failed. No retries permitted until 2026-05-25 11:17:56.923442283 +0000 UTC m=+50.468619883 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/1aff1fe9-36dd-4aed-8407-73bf7f9c439e-networking-console-plugin-cert") pod "networking-console-plugin-d4d8f94f9-8ktvq" (UID: "1aff1fe9-36dd-4aed-8407-73bf7f9c439e") : secret "networking-console-plugin-cert" not found
May 25 11:17:48.923530 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:48.923484    2561 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 25 11:17:48.923635 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:48.923531    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/039e2ad5-6518-454a-99b9-33221aaef30a-cert podName:039e2ad5-6518-454a-99b9-33221aaef30a nodeName:}" failed. No retries permitted until 2026-05-25 11:17:56.923516375 +0000 UTC m=+50.468693957 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/039e2ad5-6518-454a-99b9-33221aaef30a-cert") pod "ingress-canary-66tdv" (UID: "039e2ad5-6518-454a-99b9-33221aaef30a") : secret "canary-serving-cert" not found
May 25 11:17:49.024228 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:49.024147    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ffe11530-9960-4b99-862e-093a4377bd45-metrics-tls\") pod \"dns-default-d6xph\" (UID: \"ffe11530-9960-4b99-862e-093a4377bd45\") " pod="openshift-dns/dns-default-d6xph"
May 25 11:17:49.024407 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:49.024294    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 25 11:17:49.024407 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:49.024363    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ffe11530-9960-4b99-862e-093a4377bd45-metrics-tls podName:ffe11530-9960-4b99-862e-093a4377bd45 nodeName:}" failed. No retries permitted until 2026-05-25 11:17:57.024344833 +0000 UTC m=+50.569522431 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ffe11530-9960-4b99-862e-093a4377bd45-metrics-tls") pod "dns-default-d6xph" (UID: "ffe11530-9960-4b99-862e-093a4377bd45") : secret "dns-default-metrics-tls" not found
May 25 11:17:51.359178 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:51.359142    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-clx9s" event={"ID":"a9377ad6-eac5-4df9-b7a2-b479c9c86277","Type":"ContainerStarted","Data":"6b10e7c715738a0e361b7f1cf4eff3fa9fd6ff4ae718cdbc8026558e52be04d3"}
May 25 11:17:51.361304 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:51.361248    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7f44b6dd87-7b6rg" event={"ID":"bf1cc7fa-a86f-43ee-ab68-3b6b57050611","Type":"ContainerStarted","Data":"1f6e7cfee41449dde109d7b84d7fda967dfa5df375669fb1fca7355327634d97"}
May 25 11:17:51.363792 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:51.363367    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-78bdfb888c-nd5f5" event={"ID":"13894946-8385-4b6f-a82a-4188f46d576c","Type":"ContainerStarted","Data":"7c70edf00194ca405e70f794cf546ab437cf0174098cce7cdd9ee77cc031db66"}
May 25 11:17:51.366366 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:51.366208    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-88l9x" event={"ID":"9b9cd68b-6d33-40ae-b547-43f4ef7a2f5c","Type":"ContainerStarted","Data":"0b177d42e419ed65bec60d24bb23ebe6798a9b55ef59d1f9735337811bcbc530"}
May 25 11:17:51.368303 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:51.368232    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-9lj95_10a66344-3304-49a1-bbc1-b885800f5acf/console-operator/0.log"
May 25 11:17:51.368303 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:51.368263    2561 generic.go:358] "Generic (PLEG): container finished" podID="10a66344-3304-49a1-bbc1-b885800f5acf" containerID="57a9ef200c8b7e2c42cd01d8d94130a9914de491ddfe9ee4b3fe063e20f57a95" exitCode=255
May 25 11:17:51.368454 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:51.368342    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-76dff48d59-9lj95" event={"ID":"10a66344-3304-49a1-bbc1-b885800f5acf","Type":"ContainerDied","Data":"57a9ef200c8b7e2c42cd01d8d94130a9914de491ddfe9ee4b3fe063e20f57a95"}
May 25 11:17:51.368849 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:51.368591    2561 scope.go:117] "RemoveContainer" containerID="57a9ef200c8b7e2c42cd01d8d94130a9914de491ddfe9ee4b3fe063e20f57a95"
May 25 11:17:51.370690 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:51.370655    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-fwgjs" event={"ID":"aa041c38-250b-4342-9951-7f04c9daf30c","Type":"ContainerStarted","Data":"21cfa1c34e451bb353a0d92285e2b4f12cd14c644261a2b0709edce0824ef3cc"}
May 25 11:17:51.374329 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:51.373505    2561 generic.go:358] "Generic (PLEG): container finished" podID="7c97daaa-b9c1-4ee0-ade1-e8de392e496f" containerID="f823d6f4305508d19df0ce2a4150b7e996f322ac5f0606caffbb135540b7b903" exitCode=0
May 25 11:17:51.374329 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:51.373545    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dmzql" event={"ID":"7c97daaa-b9c1-4ee0-ade1-e8de392e496f","Type":"ContainerDied","Data":"f823d6f4305508d19df0ce2a4150b7e996f322ac5f0606caffbb135540b7b903"}
May 25 11:17:51.419608 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:51.419065    2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-clx9s" podStartSLOduration=34.768733701 podStartE2EDuration="43.41905067s" podCreationTimestamp="2026-05-25 11:17:08 +0000 UTC" firstStartedPulling="2026-05-25 11:17:41.66040928 +0000 UTC m=+35.205586864" lastFinishedPulling="2026-05-25 11:17:50.310726237 +0000 UTC m=+43.855903833" observedRunningTime="2026-05-25 11:17:51.388858176 +0000 UTC m=+44.934035780" watchObservedRunningTime="2026-05-25 11:17:51.41905067 +0000 UTC m=+44.964228274"
May 25 11:17:51.478571 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:51.474361    2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-insights/insights-operator-78bdfb888c-nd5f5" podStartSLOduration=34.763880682 podStartE2EDuration="43.474344963s" podCreationTimestamp="2026-05-25 11:17:08 +0000 UTC" firstStartedPulling="2026-05-25 11:17:41.60145844 +0000 UTC m=+35.146636039" lastFinishedPulling="2026-05-25 11:17:50.311922734 +0000 UTC m=+43.857100320" observedRunningTime="2026-05-25 11:17:51.447295567 +0000 UTC m=+44.992473171" watchObservedRunningTime="2026-05-25 11:17:51.474344963 +0000 UTC m=+45.019522567"
May 25 11:17:51.479972 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:51.479613    2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7f44b6dd87-7b6rg" podStartSLOduration=32.767593751 podStartE2EDuration="41.479600356s" podCreationTimestamp="2026-05-25 11:17:10 +0000 UTC" firstStartedPulling="2026-05-25 11:17:41.611235067 +0000 UTC m=+35.156412648" lastFinishedPulling="2026-05-25 11:17:50.323241656 +0000 UTC m=+43.868419253" observedRunningTime="2026-05-25 11:17:51.473237953 +0000 UTC m=+45.018415557" watchObservedRunningTime="2026-05-25 11:17:51.479600356 +0000 UTC m=+45.024777960"
May 25 11:17:51.496296 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:51.496141    2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-88l9x" podStartSLOduration=34.74107117 podStartE2EDuration="43.496125369s" podCreationTimestamp="2026-05-25 11:17:08 +0000 UTC" firstStartedPulling="2026-05-25 11:17:41.557289746 +0000 UTC m=+35.102467342" lastFinishedPulling="2026-05-25 11:17:50.312343957 +0000 UTC m=+43.857521541" observedRunningTime="2026-05-25 11:17:51.49481813 +0000 UTC m=+45.039995744" watchObservedRunningTime="2026-05-25 11:17:51.496125369 +0000 UTC m=+45.041302973"
May 25 11:17:51.540349 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:51.539074    2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-5cf8c89d64-fwgjs" podStartSLOduration=34.690331166 podStartE2EDuration="43.539059924s" podCreationTimestamp="2026-05-25 11:17:08 +0000 UTC" firstStartedPulling="2026-05-25 11:17:41.462063429 +0000 UTC m=+35.007241009" lastFinishedPulling="2026-05-25 11:17:50.310792182 +0000 UTC m=+43.855969767" observedRunningTime="2026-05-25 11:17:51.53778632 +0000 UTC m=+45.082963924" watchObservedRunningTime="2026-05-25 11:17:51.539059924 +0000 UTC m=+45.084237529"
May 25 11:17:52.378914 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:52.378891    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-9lj95_10a66344-3304-49a1-bbc1-b885800f5acf/console-operator/1.log"
May 25 11:17:52.379321 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:52.379304    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-9lj95_10a66344-3304-49a1-bbc1-b885800f5acf/console-operator/0.log"
May 25 11:17:52.379373 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:52.379348    2561 generic.go:358] "Generic (PLEG): container finished" podID="10a66344-3304-49a1-bbc1-b885800f5acf" containerID="e0e92c59a52a34db5c2f495c0fef0cb27f50f48ab70026e5c7bfc2bc3ee0ff64" exitCode=255
May 25 11:17:52.379451 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:52.379428    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-76dff48d59-9lj95" event={"ID":"10a66344-3304-49a1-bbc1-b885800f5acf","Type":"ContainerDied","Data":"e0e92c59a52a34db5c2f495c0fef0cb27f50f48ab70026e5c7bfc2bc3ee0ff64"}
May 25 11:17:52.379506 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:52.379486    2561 scope.go:117] "RemoveContainer" containerID="57a9ef200c8b7e2c42cd01d8d94130a9914de491ddfe9ee4b3fe063e20f57a95"
May 25 11:17:52.379683 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:52.379661    2561 scope.go:117] "RemoveContainer" containerID="e0e92c59a52a34db5c2f495c0fef0cb27f50f48ab70026e5c7bfc2bc3ee0ff64"
May 25 11:17:52.379866 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:52.379848    2561 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-76dff48d59-9lj95_openshift-console-operator(10a66344-3304-49a1-bbc1-b885800f5acf)\"" pod="openshift-console-operator/console-operator-76dff48d59-9lj95" podUID="10a66344-3304-49a1-bbc1-b885800f5acf"
May 25 11:17:52.383364 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:52.383340    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dmzql" event={"ID":"7c97daaa-b9c1-4ee0-ade1-e8de392e496f","Type":"ContainerStarted","Data":"87ce4711deeb4bcb11b961d982f3024abbb001c86df3d8cbdcc0c2784785473e"}
May 25 11:17:52.428076 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:52.428028    2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-dmzql" podStartSLOduration=11.129939461 podStartE2EDuration="45.428011475s" podCreationTimestamp="2026-05-25 11:17:07 +0000 UTC" firstStartedPulling="2026-05-25 11:17:09.732249865 +0000 UTC m=+3.277427457" lastFinishedPulling="2026-05-25 11:17:44.030321887 +0000 UTC m=+37.575499471" observedRunningTime="2026-05-25 11:17:52.427263508 +0000 UTC m=+45.972441151" watchObservedRunningTime="2026-05-25 11:17:52.428011475 +0000 UTC m=+45.973189079"
May 25 11:17:53.387475 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:53.387439    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-9lj95_10a66344-3304-49a1-bbc1-b885800f5acf/console-operator/1.log"
May 25 11:17:53.387885 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:53.387864    2561 scope.go:117] "RemoveContainer" containerID="e0e92c59a52a34db5c2f495c0fef0cb27f50f48ab70026e5c7bfc2bc3ee0ff64"
May 25 11:17:53.388083 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:53.388064    2561 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-76dff48d59-9lj95_openshift-console-operator(10a66344-3304-49a1-bbc1-b885800f5acf)\"" pod="openshift-console-operator/console-operator-76dff48d59-9lj95" podUID="10a66344-3304-49a1-bbc1-b885800f5acf"
May 25 11:17:54.126661 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:54.126635    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-xvcpl_b55f6df2-1dbc-4fc5-a4e6-e679d289e9f5/dns-node-resolver/0.log"
May 25 11:17:55.127910 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:55.127885    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-q8ng6_9a250955-746b-416a-b798-283a0ea46864/node-ca/0.log"
May 25 11:17:55.393550 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:55.393476    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-dq8s4" event={"ID":"3e92c2a7-1021-4ee8-aded-8ae5ddc6b81c","Type":"ContainerStarted","Data":"9e7f4711c2197b42206cc1de9797bb519623e3e238bbe30aeb50f67a80ee671c"}
May 25 11:17:55.409710 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:55.409665    2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-dq8s4" podStartSLOduration=34.140586732 podStartE2EDuration="43.409653606s" podCreationTimestamp="2026-05-25 11:17:12 +0000 UTC" firstStartedPulling="2026-05-25 11:17:44.993710885 +0000 UTC m=+38.538888471" lastFinishedPulling="2026-05-25 11:17:54.262777749 +0000 UTC m=+47.807955345" observedRunningTime="2026-05-25 11:17:55.408865187 +0000 UTC m=+48.954042791" watchObservedRunningTime="2026-05-25 11:17:55.409653606 +0000 UTC m=+48.954831255"
May 25 11:17:56.802096 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:56.802057    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/21792ece-de04-4eb0-990c-e1adec1c79c5-registry-tls\") pod \"image-registry-7b9fd9b9cb-fpswn\" (UID: \"21792ece-de04-4eb0-990c-e1adec1c79c5\") " pod="openshift-image-registry/image-registry-7b9fd9b9cb-fpswn"
May 25 11:17:56.802512 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:56.802225    2561 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 25 11:17:56.802512 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:56.802247    2561 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7b9fd9b9cb-fpswn: secret "image-registry-tls" not found
May 25 11:17:56.802512 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:56.802347    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/21792ece-de04-4eb0-990c-e1adec1c79c5-registry-tls podName:21792ece-de04-4eb0-990c-e1adec1c79c5 nodeName:}" failed. No retries permitted until 2026-05-25 11:18:12.802326419 +0000 UTC m=+66.347504020 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/21792ece-de04-4eb0-990c-e1adec1c79c5-registry-tls") pod "image-registry-7b9fd9b9cb-fpswn" (UID: "21792ece-de04-4eb0-990c-e1adec1c79c5") : secret "image-registry-tls" not found
May 25 11:17:56.902982 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:56.902950    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/82fb3b0a-798a-406b-99b4-548782582352-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6c75959455-tfzq9\" (UID: \"82fb3b0a-798a-406b-99b4-548782582352\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-tfzq9"
May 25 11:17:56.902982 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:56.902994    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6cac71b-58b3-43ba-8a3a-efb15e35c401-service-ca-bundle\") pod \"router-default-846bb98c44-b4vb2\" (UID: \"c6cac71b-58b3-43ba-8a3a-efb15e35c401\") " pod="openshift-ingress/router-default-846bb98c44-b4vb2"
May 25 11:17:56.903183 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:56.903094    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/8bf76398-20ba-499c-9103-c4017cdad681-samples-operator-tls\") pod \"cluster-samples-operator-697587599-6vz7w\" (UID: \"8bf76398-20ba-499c-9103-c4017cdad681\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-6vz7w"
May 25 11:17:56.903183 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:56.903101    2561 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
May 25 11:17:56.903183 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:56.903134    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c6cac71b-58b3-43ba-8a3a-efb15e35c401-metrics-certs\") pod \"router-default-846bb98c44-b4vb2\" (UID: \"c6cac71b-58b3-43ba-8a3a-efb15e35c401\") " pod="openshift-ingress/router-default-846bb98c44-b4vb2"
May 25 11:17:56.903183 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:56.903175    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82fb3b0a-798a-406b-99b4-548782582352-cluster-monitoring-operator-tls podName:82fb3b0a-798a-406b-99b4-548782582352 nodeName:}" failed. No retries permitted until 2026-05-25 11:18:12.903158604 +0000 UTC m=+66.448336186 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/82fb3b0a-798a-406b-99b4-548782582352-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6c75959455-tfzq9" (UID: "82fb3b0a-798a-406b-99b4-548782582352") : secret "cluster-monitoring-operator-tls" not found
May 25 11:17:56.903340 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:56.903228    2561 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
May 25 11:17:56.903340 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:56.903252    2561 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
May 25 11:17:56.903340 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:56.903297    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c6cac71b-58b3-43ba-8a3a-efb15e35c401-metrics-certs podName:c6cac71b-58b3-43ba-8a3a-efb15e35c401 nodeName:}" failed. No retries permitted until 2026-05-25 11:18:12.903263102 +0000 UTC m=+66.448440687 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c6cac71b-58b3-43ba-8a3a-efb15e35c401-metrics-certs") pod "router-default-846bb98c44-b4vb2" (UID: "c6cac71b-58b3-43ba-8a3a-efb15e35c401") : secret "router-metrics-certs-default" not found
May 25 11:17:56.903340 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:56.903328    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8bf76398-20ba-499c-9103-c4017cdad681-samples-operator-tls podName:8bf76398-20ba-499c-9103-c4017cdad681 nodeName:}" failed. No retries permitted until 2026-05-25 11:18:12.90331625 +0000 UTC m=+66.448493833 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/8bf76398-20ba-499c-9103-c4017cdad681-samples-operator-tls") pod "cluster-samples-operator-697587599-6vz7w" (UID: "8bf76398-20ba-499c-9103-c4017cdad681") : secret "samples-operator-tls" not found
May 25 11:17:56.903466 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:56.903376    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c6cac71b-58b3-43ba-8a3a-efb15e35c401-service-ca-bundle podName:c6cac71b-58b3-43ba-8a3a-efb15e35c401 nodeName:}" failed. No retries permitted until 2026-05-25 11:18:12.903361973 +0000 UTC m=+66.448539560 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/c6cac71b-58b3-43ba-8a3a-efb15e35c401-service-ca-bundle") pod "router-default-846bb98c44-b4vb2" (UID: "c6cac71b-58b3-43ba-8a3a-efb15e35c401") : configmap references non-existent config key: service-ca.crt
May 25 11:17:57.004597 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:57.004561    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/039e2ad5-6518-454a-99b9-33221aaef30a-cert\") pod \"ingress-canary-66tdv\" (UID: \"039e2ad5-6518-454a-99b9-33221aaef30a\") " pod="openshift-ingress-canary/ingress-canary-66tdv"
May 25 11:17:57.004765 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:57.004644    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1aff1fe9-36dd-4aed-8407-73bf7f9c439e-networking-console-plugin-cert\") pod \"networking-console-plugin-d4d8f94f9-8ktvq\" (UID: \"1aff1fe9-36dd-4aed-8407-73bf7f9c439e\") " pod="openshift-network-console/networking-console-plugin-d4d8f94f9-8ktvq"
May 25 11:17:57.004765 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:57.004699    2561 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 25 11:17:57.004845 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:57.004770    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/039e2ad5-6518-454a-99b9-33221aaef30a-cert podName:039e2ad5-6518-454a-99b9-33221aaef30a nodeName:}" failed. No retries permitted until 2026-05-25 11:18:13.004755276 +0000 UTC m=+66.549932856 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/039e2ad5-6518-454a-99b9-33221aaef30a-cert") pod "ingress-canary-66tdv" (UID: "039e2ad5-6518-454a-99b9-33221aaef30a") : secret "canary-serving-cert" not found
May 25 11:17:57.004845 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:57.004784    2561 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
May 25 11:17:57.004845 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:57.004837    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1aff1fe9-36dd-4aed-8407-73bf7f9c439e-networking-console-plugin-cert podName:1aff1fe9-36dd-4aed-8407-73bf7f9c439e nodeName:}" failed. No retries permitted until 2026-05-25 11:18:13.004820754 +0000 UTC m=+66.549998355 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/1aff1fe9-36dd-4aed-8407-73bf7f9c439e-networking-console-plugin-cert") pod "networking-console-plugin-d4d8f94f9-8ktvq" (UID: "1aff1fe9-36dd-4aed-8407-73bf7f9c439e") : secret "networking-console-plugin-cert" not found
May 25 11:17:57.105702 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:17:57.105667    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ffe11530-9960-4b99-862e-093a4377bd45-metrics-tls\") pod \"dns-default-d6xph\" (UID: \"ffe11530-9960-4b99-862e-093a4377bd45\") " pod="openshift-dns/dns-default-d6xph"
May 25 11:17:57.105883 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:57.105805    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 25 11:17:57.105883 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:17:57.105866    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ffe11530-9960-4b99-862e-093a4377bd45-metrics-tls podName:ffe11530-9960-4b99-862e-093a4377bd45 nodeName:}" failed. No retries permitted until 2026-05-25 11:18:13.105849121 +0000 UTC m=+66.651026719 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ffe11530-9960-4b99-862e-093a4377bd45-metrics-tls") pod "dns-default-d6xph" (UID: "ffe11530-9960-4b99-862e-093a4377bd45") : secret "dns-default-metrics-tls" not found
May 25 11:18:01.312874 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:01.312845    2561 kubelet.go:2658] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-76dff48d59-9lj95"
May 25 11:18:01.312874 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:01.312881    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-76dff48d59-9lj95"
May 25 11:18:01.313338 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:01.313184    2561 scope.go:117] "RemoveContainer" containerID="e0e92c59a52a34db5c2f495c0fef0cb27f50f48ab70026e5c7bfc2bc3ee0ff64"
May 25 11:18:01.313374 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:18:01.313355    2561 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-76dff48d59-9lj95_openshift-console-operator(10a66344-3304-49a1-bbc1-b885800f5acf)\"" pod="openshift-console-operator/console-operator-76dff48d59-9lj95" podUID="10a66344-3304-49a1-bbc1-b885800f5acf"
May 25 11:18:05.295997 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:05.295971    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-thvg5"
May 25 11:18:12.836934 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:12.836885    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/21792ece-de04-4eb0-990c-e1adec1c79c5-registry-tls\") pod \"image-registry-7b9fd9b9cb-fpswn\" (UID: \"21792ece-de04-4eb0-990c-e1adec1c79c5\") " pod="openshift-image-registry/image-registry-7b9fd9b9cb-fpswn"
May 25 11:18:12.839471 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:12.839449    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/21792ece-de04-4eb0-990c-e1adec1c79c5-registry-tls\") pod \"image-registry-7b9fd9b9cb-fpswn\" (UID: \"21792ece-de04-4eb0-990c-e1adec1c79c5\") " pod="openshift-image-registry/image-registry-7b9fd9b9cb-fpswn"
May 25 11:18:12.937602 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:12.937578    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c6cac71b-58b3-43ba-8a3a-efb15e35c401-metrics-certs\") pod \"router-default-846bb98c44-b4vb2\" (UID: \"c6cac71b-58b3-43ba-8a3a-efb15e35c401\") " pod="openshift-ingress/router-default-846bb98c44-b4vb2"
May 25 11:18:12.937720 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:12.937614    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ead20bcd-df93-4d45-b5bb-470a8fe373e7-metrics-certs\") pod \"network-metrics-daemon-444mb\" (UID: \"ead20bcd-df93-4d45-b5bb-470a8fe373e7\") " pod="openshift-multus/network-metrics-daemon-444mb"
May 25 11:18:12.937720 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:12.937653    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/82fb3b0a-798a-406b-99b4-548782582352-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6c75959455-tfzq9\" (UID: \"82fb3b0a-798a-406b-99b4-548782582352\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-tfzq9"
May 25 11:18:12.937720 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:12.937671    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-nbhnn\" (UniqueName: \"kubernetes.io/projected/2b05b78f-6ab1-4f94-8273-e5fa70b78d15-kube-api-access-nbhnn\") pod \"network-check-target-rl7hd\" (UID: \"2b05b78f-6ab1-4f94-8273-e5fa70b78d15\") " pod="openshift-network-diagnostics/network-check-target-rl7hd"
May 25 11:18:12.937720 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:12.937688    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6cac71b-58b3-43ba-8a3a-efb15e35c401-service-ca-bundle\") pod \"router-default-846bb98c44-b4vb2\" (UID: \"c6cac71b-58b3-43ba-8a3a-efb15e35c401\") " pod="openshift-ingress/router-default-846bb98c44-b4vb2"
May 25 11:18:12.937942 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:12.937723    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/8bf76398-20ba-499c-9103-c4017cdad681-samples-operator-tls\") pod \"cluster-samples-operator-697587599-6vz7w\" (UID: \"8bf76398-20ba-499c-9103-c4017cdad681\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-6vz7w"
May 25 11:18:12.938422 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:12.938399    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6cac71b-58b3-43ba-8a3a-efb15e35c401-service-ca-bundle\") pod \"router-default-846bb98c44-b4vb2\" (UID: \"c6cac71b-58b3-43ba-8a3a-efb15e35c401\") " pod="openshift-ingress/router-default-846bb98c44-b4vb2"
May 25 11:18:12.940004 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:12.939971    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/82fb3b0a-798a-406b-99b4-548782582352-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6c75959455-tfzq9\" (UID: \"82fb3b0a-798a-406b-99b4-548782582352\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-tfzq9"
May 25 11:18:12.940004 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:12.939976    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c6cac71b-58b3-43ba-8a3a-efb15e35c401-metrics-certs\") pod \"router-default-846bb98c44-b4vb2\" (UID: \"c6cac71b-58b3-43ba-8a3a-efb15e35c401\") " pod="openshift-ingress/router-default-846bb98c44-b4vb2"
May 25 11:18:12.940139 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:12.940112    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
May 25 11:18:12.940344 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:12.940328    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/8bf76398-20ba-499c-9103-c4017cdad681-samples-operator-tls\") pod \"cluster-samples-operator-697587599-6vz7w\" (UID: \"8bf76398-20ba-499c-9103-c4017cdad681\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-6vz7w"
May 25 11:18:12.940571 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:12.940551    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbhnn\" (UniqueName: \"kubernetes.io/projected/2b05b78f-6ab1-4f94-8273-e5fa70b78d15-kube-api-access-nbhnn\") pod \"network-check-target-rl7hd\" (UID: \"2b05b78f-6ab1-4f94-8273-e5fa70b78d15\") " pod="openshift-network-diagnostics/network-check-target-rl7hd"
May 25 11:18:12.950025 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:12.949999    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ead20bcd-df93-4d45-b5bb-470a8fe373e7-metrics-certs\") pod \"network-metrics-daemon-444mb\" (UID: \"ead20bcd-df93-4d45-b5bb-470a8fe373e7\") " pod="openshift-multus/network-metrics-daemon-444mb"
May 25 11:18:12.968637 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:12.968623    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-x64h9\""
May 25 11:18:12.976882 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:12.976865    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-444mb"
May 25 11:18:12.979252 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:12.979236    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-lf8xr\""
May 25 11:18:12.987911 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:12.987894    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rl7hd"
May 25 11:18:13.039560 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:13.039082    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/039e2ad5-6518-454a-99b9-33221aaef30a-cert\") pod \"ingress-canary-66tdv\" (UID: \"039e2ad5-6518-454a-99b9-33221aaef30a\") " pod="openshift-ingress-canary/ingress-canary-66tdv"
May 25 11:18:13.039560 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:13.039162    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1aff1fe9-36dd-4aed-8407-73bf7f9c439e-networking-console-plugin-cert\") pod \"networking-console-plugin-d4d8f94f9-8ktvq\" (UID: \"1aff1fe9-36dd-4aed-8407-73bf7f9c439e\") " pod="openshift-network-console/networking-console-plugin-d4d8f94f9-8ktvq"
May 25 11:18:13.042536 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:13.042482    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1aff1fe9-36dd-4aed-8407-73bf7f9c439e-networking-console-plugin-cert\") pod \"networking-console-plugin-d4d8f94f9-8ktvq\" (UID: \"1aff1fe9-36dd-4aed-8407-73bf7f9c439e\") " pod="openshift-network-console/networking-console-plugin-d4d8f94f9-8ktvq"
May 25 11:18:13.043693 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:13.043668    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/039e2ad5-6518-454a-99b9-33221aaef30a-cert\") pod \"ingress-canary-66tdv\" (UID: \"039e2ad5-6518-454a-99b9-33221aaef30a\") " pod="openshift-ingress-canary/ingress-canary-66tdv"
May 25 11:18:13.072795 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:13.072772    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-gzhs6\""
May 25 11:18:13.081304 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:13.081284    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7b9fd9b9cb-fpswn"
May 25 11:18:13.097530 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:13.097509    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-dmhkh\""
May 25 11:18:13.102399 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:13.102376    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-444mb"]
May 25 11:18:13.104442 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:18:13.104412    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podead20bcd_df93_4d45_b5bb_470a8fe373e7.slice/crio-1f193aab369ef291133ffb5c44bb3ac2338f0a5e6915f9e3c2ccb17c2b9956f8 WatchSource:0}: Error finding container 1f193aab369ef291133ffb5c44bb3ac2338f0a5e6915f9e3c2ccb17c2b9956f8: Status 404 returned error can't find the container with id 1f193aab369ef291133ffb5c44bb3ac2338f0a5e6915f9e3c2ccb17c2b9956f8
May 25 11:18:13.105260 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:13.105243    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-846bb98c44-b4vb2"
May 25 11:18:13.116191 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:13.116171    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-rl7hd"]
May 25 11:18:13.119969 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:18:13.119935    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b05b78f_6ab1_4f94_8273_e5fa70b78d15.slice/crio-27434bd96d7ab8a0403700d65b65bbc81480da7da303a611c51cc212e7431d86 WatchSource:0}: Error finding container 27434bd96d7ab8a0403700d65b65bbc81480da7da303a611c51cc212e7431d86: Status 404 returned error can't find the container with id 27434bd96d7ab8a0403700d65b65bbc81480da7da303a611c51cc212e7431d86
May 25 11:18:13.128069 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:13.128044    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-9rfft\""
May 25 11:18:13.136415 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:13.136393    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-tfzq9"
May 25 11:18:13.139620 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:13.139577    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ffe11530-9960-4b99-862e-093a4377bd45-metrics-tls\") pod \"dns-default-d6xph\" (UID: \"ffe11530-9960-4b99-862e-093a4377bd45\") " pod="openshift-dns/dns-default-d6xph"
May 25 11:18:13.144045 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:13.144002    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ffe11530-9960-4b99-862e-093a4377bd45-metrics-tls\") pod \"dns-default-d6xph\" (UID: \"ffe11530-9960-4b99-862e-093a4377bd45\") " pod="openshift-dns/dns-default-d6xph"
May 25 11:18:13.160414 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:13.160178    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-t8vmz\""
May 25 11:18:13.168724 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:13.167781    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-6vz7w"
May 25 11:18:13.195922 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:13.195835    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-6n45b\""
May 25 11:18:13.205229 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:13.204883    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-d4d8f94f9-8ktvq"
May 25 11:18:13.233257 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:13.233205    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7b9fd9b9cb-fpswn"]
May 25 11:18:13.235869 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:13.235597    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-kttpg\""
May 25 11:18:13.242698 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:13.242666    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-66tdv"
May 25 11:18:13.243453 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:18:13.243421    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21792ece_de04_4eb0_990c_e1adec1c79c5.slice/crio-11e54098a27429630057f50ffaa3cd85effca9a1fc6c63b84c5383a833507f41 WatchSource:0}: Error finding container 11e54098a27429630057f50ffaa3cd85effca9a1fc6c63b84c5383a833507f41: Status 404 returned error can't find the container with id 11e54098a27429630057f50ffaa3cd85effca9a1fc6c63b84c5383a833507f41
May 25 11:18:13.297403 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:13.297335    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-846bb98c44-b4vb2"]
May 25 11:18:13.297938 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:13.297713    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-nlbp4\""
May 25 11:18:13.303251 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:13.301393    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-d6xph"
May 25 11:18:13.328217 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:13.325017    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6c75959455-tfzq9"]
May 25 11:18:13.332687 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:18:13.332659    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82fb3b0a_798a_406b_99b4_548782582352.slice/crio-705a3828f280523f4634ae5c8cd0249f018a0481639e4e0b37d8a4cee25d4f82 WatchSource:0}: Error finding container 705a3828f280523f4634ae5c8cd0249f018a0481639e4e0b37d8a4cee25d4f82: Status 404 returned error can't find the container with id 705a3828f280523f4634ae5c8cd0249f018a0481639e4e0b37d8a4cee25d4f82
May 25 11:18:13.350179 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:13.350122    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-697587599-6vz7w"]
May 25 11:18:13.400305 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:13.400243    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-d4d8f94f9-8ktvq"]
May 25 11:18:13.406828 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:18:13.406799    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1aff1fe9_36dd_4aed_8407_73bf7f9c439e.slice/crio-2f6f8a880e358ab4544e117a827badf630fd4b485aa94f1b6e8cc7dbf0c01793 WatchSource:0}: Error finding container 2f6f8a880e358ab4544e117a827badf630fd4b485aa94f1b6e8cc7dbf0c01793: Status 404 returned error can't find the container with id 2f6f8a880e358ab4544e117a827badf630fd4b485aa94f1b6e8cc7dbf0c01793
May 25 11:18:13.444387 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:13.444290    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-6vz7w" event={"ID":"8bf76398-20ba-499c-9103-c4017cdad681","Type":"ContainerStarted","Data":"3931e9912c05824bae960f4ed42d2e786480029c7f7a134d20ef2d19311e327b"}
May 25 11:18:13.445452 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:13.445419    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-d4d8f94f9-8ktvq" event={"ID":"1aff1fe9-36dd-4aed-8407-73bf7f9c439e","Type":"ContainerStarted","Data":"2f6f8a880e358ab4544e117a827badf630fd4b485aa94f1b6e8cc7dbf0c01793"}
May 25 11:18:13.446732 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:13.446706    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7b9fd9b9cb-fpswn" event={"ID":"21792ece-de04-4eb0-990c-e1adec1c79c5","Type":"ContainerStarted","Data":"e2d738c1e0b77b6c83930e0926a55dd4acbdb14207af212e2f336a48be1620e7"}
May 25 11:18:13.446858 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:13.446741    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7b9fd9b9cb-fpswn" event={"ID":"21792ece-de04-4eb0-990c-e1adec1c79c5","Type":"ContainerStarted","Data":"11e54098a27429630057f50ffaa3cd85effca9a1fc6c63b84c5383a833507f41"}
May 25 11:18:13.447074 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:13.447045    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7b9fd9b9cb-fpswn"
May 25 11:18:13.448571 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:13.448524    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-rl7hd" event={"ID":"2b05b78f-6ab1-4f94-8273-e5fa70b78d15","Type":"ContainerStarted","Data":"fb7ffdf71a09fe83259e55f26c6da747758cff91449e8e870642aafae1634e81"}
May 25 11:18:13.448571 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:13.448554    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-rl7hd" event={"ID":"2b05b78f-6ab1-4f94-8273-e5fa70b78d15","Type":"ContainerStarted","Data":"27434bd96d7ab8a0403700d65b65bbc81480da7da303a611c51cc212e7431d86"}
May 25 11:18:13.448727 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:13.448646    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-rl7hd"
May 25 11:18:13.449893 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:13.449851    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-tfzq9" event={"ID":"82fb3b0a-798a-406b-99b4-548782582352","Type":"ContainerStarted","Data":"705a3828f280523f4634ae5c8cd0249f018a0481639e4e0b37d8a4cee25d4f82"}
May 25 11:18:13.452553 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:13.452499    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-444mb" event={"ID":"ead20bcd-df93-4d45-b5bb-470a8fe373e7","Type":"ContainerStarted","Data":"1f193aab369ef291133ffb5c44bb3ac2338f0a5e6915f9e3c2ccb17c2b9956f8"}
May 25 11:18:13.454338 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:13.454177    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-846bb98c44-b4vb2" event={"ID":"c6cac71b-58b3-43ba-8a3a-efb15e35c401","Type":"ContainerStarted","Data":"12e8776d44b5f0b3ebb55fab14e2c9e5175e520d7415f0bf7a46bcf34dbc9f14"}
May 25 11:18:13.454338 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:13.454203    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-846bb98c44-b4vb2" event={"ID":"c6cac71b-58b3-43ba-8a3a-efb15e35c401","Type":"ContainerStarted","Data":"921fb596e482140d9513902e9ffd22ea34e2f992887cdb9673ea1c723047f61a"}
May 25 11:18:13.456562 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:13.456508    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-66tdv"]
May 25 11:18:13.470139 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:13.470079    2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7b9fd9b9cb-fpswn" podStartSLOduration=66.470062028 podStartE2EDuration="1m6.470062028s" podCreationTimestamp="2026-05-25 11:17:07 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 11:18:13.469039593 +0000 UTC m=+67.014217197" watchObservedRunningTime="2026-05-25 11:18:13.470062028 +0000 UTC m=+67.015239639"
May 25 11:18:13.484834 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:13.484791    2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-rl7hd" podStartSLOduration=66.484778848 podStartE2EDuration="1m6.484778848s" podCreationTimestamp="2026-05-25 11:17:07 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 11:18:13.484111682 +0000 UTC m=+67.029289277" watchObservedRunningTime="2026-05-25 11:18:13.484778848 +0000 UTC m=+67.029956504"
May 25 11:18:13.502023 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:13.501995    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-d6xph"]
May 25 11:18:13.504962 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:13.504894    2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ingress/router-default-846bb98c44-b4vb2" podStartSLOduration=65.504881766 podStartE2EDuration="1m5.504881766s" podCreationTimestamp="2026-05-25 11:17:08 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 11:18:13.504004555 +0000 UTC m=+67.049182149" watchObservedRunningTime="2026-05-25 11:18:13.504881766 +0000 UTC m=+67.050059370"
May 25 11:18:13.507563 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:18:13.507537    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffe11530_9960_4b99_862e_093a4377bd45.slice/crio-3b08f6d55b49d1bbf5d6178d3893b850cdf382290f5ae62a86eaa30379bde53c WatchSource:0}: Error finding container 3b08f6d55b49d1bbf5d6178d3893b850cdf382290f5ae62a86eaa30379bde53c: Status 404 returned error can't find the container with id 3b08f6d55b49d1bbf5d6178d3893b850cdf382290f5ae62a86eaa30379bde53c
May 25 11:18:14.105933 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:14.105900    2561 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-846bb98c44-b4vb2"
May 25 11:18:14.109845 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:14.109818    2561 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-846bb98c44-b4vb2"
May 25 11:18:14.471447 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:14.471335    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-66tdv" event={"ID":"039e2ad5-6518-454a-99b9-33221aaef30a","Type":"ContainerStarted","Data":"f8f5b713a3fad2e066c8954298f5b22adf323727616a5843b04396b23aedec05"}
May 25 11:18:14.473725 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:14.473480    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-d6xph" event={"ID":"ffe11530-9960-4b99-862e-093a4377bd45","Type":"ContainerStarted","Data":"3b08f6d55b49d1bbf5d6178d3893b850cdf382290f5ae62a86eaa30379bde53c"}
May 25 11:18:14.473725 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:14.473521    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-846bb98c44-b4vb2"
May 25 11:18:14.475600 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:14.475536    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-846bb98c44-b4vb2"
May 25 11:18:16.113518 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:16.113484    2561 scope.go:117] "RemoveContainer" containerID="e0e92c59a52a34db5c2f495c0fef0cb27f50f48ab70026e5c7bfc2bc3ee0ff64"
May 25 11:18:18.486441 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:18.486403    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-tfzq9" event={"ID":"82fb3b0a-798a-406b-99b4-548782582352","Type":"ContainerStarted","Data":"7839f4a52dad54d86f8fc6442cd95c9511a941bcb7102f150fe818c274d0157e"}
May 25 11:18:18.488621 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:18.488542    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-444mb" event={"ID":"ead20bcd-df93-4d45-b5bb-470a8fe373e7","Type":"ContainerStarted","Data":"1ce0673d2e9a09c5f5a192479cdb3ad0acc9e7425ba73553a00432d97a946360"}
May 25 11:18:18.488621 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:18.488581    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-444mb" event={"ID":"ead20bcd-df93-4d45-b5bb-470a8fe373e7","Type":"ContainerStarted","Data":"69a3bfddc692dedcd26bad80d49db4ae06ff3f222552d9b85052b9c8751bf3d5"}
May 25 11:18:18.490239 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:18.490209    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-66tdv" event={"ID":"039e2ad5-6518-454a-99b9-33221aaef30a","Type":"ContainerStarted","Data":"543d7aa2fcd7aedd1c02abb84ed4dd765107171abfbd7ee4e127263bf15fae5a"}
May 25 11:18:18.492142 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:18.492126    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-9lj95_10a66344-3304-49a1-bbc1-b885800f5acf/console-operator/1.log"
May 25 11:18:18.492223 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:18.492195    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-76dff48d59-9lj95" event={"ID":"10a66344-3304-49a1-bbc1-b885800f5acf","Type":"ContainerStarted","Data":"072c423669b91cfae693774a8f57ff7c01a1f9114c2c99eda6e3a64512d6c1e0"}
May 25 11:18:18.492484 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:18.492463    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-76dff48d59-9lj95"
May 25 11:18:18.493977 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:18.493951    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-d6xph" event={"ID":"ffe11530-9960-4b99-862e-093a4377bd45","Type":"ContainerStarted","Data":"cb066208b82187a093a8dfaff91acd546c2bc2eb669651584c260a97cad078e7"}
May 25 11:18:18.494075 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:18.493980    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-d6xph" event={"ID":"ffe11530-9960-4b99-862e-093a4377bd45","Type":"ContainerStarted","Data":"0d922f3c2c648adf0ffa949f669d858a0c6b18f76adbb6356ffacd14a5294dc3"}
May 25 11:18:18.494075 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:18.494063    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-d6xph"
May 25 11:18:18.495549 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:18.495526    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-6vz7w" event={"ID":"8bf76398-20ba-499c-9103-c4017cdad681","Type":"ContainerStarted","Data":"f62a5330ffe313bf1d68a841259201d5a6775bf9d8b154e56c48d85d2cadbfbf"}
May 25 11:18:18.495659 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:18.495553    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-6vz7w" event={"ID":"8bf76398-20ba-499c-9103-c4017cdad681","Type":"ContainerStarted","Data":"6fd94f792b4b3166e8b8abd9318d8801c45fa25e1d56f5e16bf0bd9b1163f163"}
May 25 11:18:18.496973 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:18.496954    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-d4d8f94f9-8ktvq" event={"ID":"1aff1fe9-36dd-4aed-8407-73bf7f9c439e","Type":"ContainerStarted","Data":"4f24e4f1977ffabbdd782c00b6b95b0c3c5653d23c809d4705b3bddedaa18f37"}
May 25 11:18:18.582358 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:18.582320    2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-tfzq9" podStartSLOduration=66.347654838 podStartE2EDuration="1m10.58230859s" podCreationTimestamp="2026-05-25 11:17:08 +0000 UTC" firstStartedPulling="2026-05-25 11:18:13.334974923 +0000 UTC m=+66.880152508" lastFinishedPulling="2026-05-25 11:18:17.569628664 +0000 UTC m=+71.114806260" observedRunningTime="2026-05-25 11:18:18.581616273 +0000 UTC m=+72.126793878" watchObservedRunningTime="2026-05-25 11:18:18.58230859 +0000 UTC m=+72.127486171"
May 25 11:18:18.595217 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:18.595197    2561 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-7nvl9"]
May 25 11:18:18.611036 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:18.611006    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-7nvl9"
May 25 11:18:18.615608 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:18.615586    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
May 25 11:18:18.615710 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:18.615653    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
May 25 11:18:18.615954 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:18.615936    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-xd7cl\""
May 25 11:18:18.680779 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:18.680750    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-76dff48d59-9lj95"
May 25 11:18:18.682138 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:18.682116    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-7nvl9"]
May 25 11:18:18.682735 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:18.682697    2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-444mb" podStartSLOduration=67.22059932 podStartE2EDuration="1m11.682684383s" podCreationTimestamp="2026-05-25 11:17:07 +0000 UTC" firstStartedPulling="2026-05-25 11:18:13.106420998 +0000 UTC m=+66.651598579" lastFinishedPulling="2026-05-25 11:18:17.568506045 +0000 UTC m=+71.113683642" observedRunningTime="2026-05-25 11:18:18.679066151 +0000 UTC m=+72.224243754" watchObservedRunningTime="2026-05-25 11:18:18.682684383 +0000 UTC m=+72.227861986"
May 25 11:18:18.687087 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:18.687062    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c18099d5-e3a6-4e23-b792-8fc91dcb9343-crio-socket\") pod \"insights-runtime-extractor-7nvl9\" (UID: \"c18099d5-e3a6-4e23-b792-8fc91dcb9343\") " pod="openshift-insights/insights-runtime-extractor-7nvl9"
May 25 11:18:18.687387 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:18.687113    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c18099d5-e3a6-4e23-b792-8fc91dcb9343-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7nvl9\" (UID: \"c18099d5-e3a6-4e23-b792-8fc91dcb9343\") " pod="openshift-insights/insights-runtime-extractor-7nvl9"
May 25 11:18:18.687387 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:18.687165    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c18099d5-e3a6-4e23-b792-8fc91dcb9343-data-volume\") pod \"insights-runtime-extractor-7nvl9\" (UID: \"c18099d5-e3a6-4e23-b792-8fc91dcb9343\") " pod="openshift-insights/insights-runtime-extractor-7nvl9"
May 25 11:18:18.687387 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:18.687201    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gprw\" (UniqueName: \"kubernetes.io/projected/c18099d5-e3a6-4e23-b792-8fc91dcb9343-kube-api-access-8gprw\") pod \"insights-runtime-extractor-7nvl9\" (UID: \"c18099d5-e3a6-4e23-b792-8fc91dcb9343\") " pod="openshift-insights/insights-runtime-extractor-7nvl9"
May 25 11:18:18.687387 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:18.687262    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c18099d5-e3a6-4e23-b792-8fc91dcb9343-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7nvl9\" (UID: \"c18099d5-e3a6-4e23-b792-8fc91dcb9343\") " pod="openshift-insights/insights-runtime-extractor-7nvl9"
May 25 11:18:18.776561 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:18.776475    2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console-operator/console-operator-76dff48d59-9lj95" podStartSLOduration=61.99092181 podStartE2EDuration="1m10.776461158s" podCreationTimestamp="2026-05-25 11:17:08 +0000 UTC" firstStartedPulling="2026-05-25 11:17:41.525643475 +0000 UTC m=+35.070821058" lastFinishedPulling="2026-05-25 11:17:50.311182822 +0000 UTC m=+43.856360406" observedRunningTime="2026-05-25 11:18:18.775485091 +0000 UTC m=+72.320662694" watchObservedRunningTime="2026-05-25 11:18:18.776461158 +0000 UTC m=+72.321638762"
May 25 11:18:18.788496 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:18.788468    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c18099d5-e3a6-4e23-b792-8fc91dcb9343-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7nvl9\" (UID: \"c18099d5-e3a6-4e23-b792-8fc91dcb9343\") " pod="openshift-insights/insights-runtime-extractor-7nvl9"
May 25 11:18:18.788620 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:18.788504    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c18099d5-e3a6-4e23-b792-8fc91dcb9343-data-volume\") pod \"insights-runtime-extractor-7nvl9\" (UID: \"c18099d5-e3a6-4e23-b792-8fc91dcb9343\") " pod="openshift-insights/insights-runtime-extractor-7nvl9"
May 25 11:18:18.788620 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:18.788527    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-8gprw\" (UniqueName: \"kubernetes.io/projected/c18099d5-e3a6-4e23-b792-8fc91dcb9343-kube-api-access-8gprw\") pod \"insights-runtime-extractor-7nvl9\" (UID: \"c18099d5-e3a6-4e23-b792-8fc91dcb9343\") " pod="openshift-insights/insights-runtime-extractor-7nvl9"
May 25 11:18:18.788620 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:18.788552    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c18099d5-e3a6-4e23-b792-8fc91dcb9343-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7nvl9\" (UID: \"c18099d5-e3a6-4e23-b792-8fc91dcb9343\") " pod="openshift-insights/insights-runtime-extractor-7nvl9"
May 25 11:18:18.788620 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:18.788582    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c18099d5-e3a6-4e23-b792-8fc91dcb9343-crio-socket\") pod \"insights-runtime-extractor-7nvl9\" (UID: \"c18099d5-e3a6-4e23-b792-8fc91dcb9343\") " pod="openshift-insights/insights-runtime-extractor-7nvl9"
May 25 11:18:18.788816 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:18.788764    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c18099d5-e3a6-4e23-b792-8fc91dcb9343-crio-socket\") pod \"insights-runtime-extractor-7nvl9\" (UID: \"c18099d5-e3a6-4e23-b792-8fc91dcb9343\") " pod="openshift-insights/insights-runtime-extractor-7nvl9"
May 25 11:18:18.788871 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:18.788856    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c18099d5-e3a6-4e23-b792-8fc91dcb9343-data-volume\") pod \"insights-runtime-extractor-7nvl9\" (UID: \"c18099d5-e3a6-4e23-b792-8fc91dcb9343\") " pod="openshift-insights/insights-runtime-extractor-7nvl9"
May 25 11:18:18.789138 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:18.789116    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c18099d5-e3a6-4e23-b792-8fc91dcb9343-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7nvl9\" (UID: \"c18099d5-e3a6-4e23-b792-8fc91dcb9343\") " pod="openshift-insights/insights-runtime-extractor-7nvl9"
May 25 11:18:18.790791 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:18.790766    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c18099d5-e3a6-4e23-b792-8fc91dcb9343-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7nvl9\" (UID: \"c18099d5-e3a6-4e23-b792-8fc91dcb9343\") " pod="openshift-insights/insights-runtime-extractor-7nvl9"
May 25 11:18:18.858760 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:18.857003    2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-6vz7w" podStartSLOduration=66.323522814 podStartE2EDuration="1m10.856986497s" podCreationTimestamp="2026-05-25 11:17:08 +0000 UTC" firstStartedPulling="2026-05-25 11:18:13.415031098 +0000 UTC m=+66.960208685" lastFinishedPulling="2026-05-25 11:18:17.948494787 +0000 UTC m=+71.493672368" observedRunningTime="2026-05-25 11:18:18.856286369 +0000 UTC m=+72.401463972" watchObservedRunningTime="2026-05-25 11:18:18.856986497 +0000 UTC m=+72.402164103"
May 25 11:18:18.867663 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:18.867633    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gprw\" (UniqueName: \"kubernetes.io/projected/c18099d5-e3a6-4e23-b792-8fc91dcb9343-kube-api-access-8gprw\") pod \"insights-runtime-extractor-7nvl9\" (UID: \"c18099d5-e3a6-4e23-b792-8fc91dcb9343\") " pod="openshift-insights/insights-runtime-extractor-7nvl9"
May 25 11:18:18.920415 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:18.920389    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-7nvl9"
May 25 11:18:18.932108 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:18.932057    2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-d4d8f94f9-8ktvq" podStartSLOduration=59.772461267 podStartE2EDuration="1m3.932040753s" podCreationTimestamp="2026-05-25 11:17:15 +0000 UTC" firstStartedPulling="2026-05-25 11:18:13.409683799 +0000 UTC m=+66.954861379" lastFinishedPulling="2026-05-25 11:18:17.56926327 +0000 UTC m=+71.114440865" observedRunningTime="2026-05-25 11:18:18.927423212 +0000 UTC m=+72.472600816" watchObservedRunningTime="2026-05-25 11:18:18.932040753 +0000 UTC m=+72.477218357"
May 25 11:18:18.986305 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:18.986137    2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-66tdv" podStartSLOduration=34.880039256 podStartE2EDuration="38.986116085s" podCreationTimestamp="2026-05-25 11:17:40 +0000 UTC" firstStartedPulling="2026-05-25 11:18:13.463163177 +0000 UTC m=+67.008340758" lastFinishedPulling="2026-05-25 11:18:17.569239992 +0000 UTC m=+71.114417587" observedRunningTime="2026-05-25 11:18:18.984086945 +0000 UTC m=+72.529264548" watchObservedRunningTime="2026-05-25 11:18:18.986116085 +0000 UTC m=+72.531293690"
May 25 11:18:19.033791 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:19.033703    2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-dns/dns-default-d6xph" podStartSLOduration=34.973193161 podStartE2EDuration="39.033687398s" podCreationTimestamp="2026-05-25 11:17:40 +0000 UTC" firstStartedPulling="2026-05-25 11:18:13.509726146 +0000 UTC m=+67.054903726" lastFinishedPulling="2026-05-25 11:18:17.570220366 +0000 UTC m=+71.115397963" observedRunningTime="2026-05-25 11:18:19.033023922 +0000 UTC m=+72.578201535" watchObservedRunningTime="2026-05-25 11:18:19.033687398 +0000 UTC m=+72.578865013"
May 25 11:18:19.099564 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:19.099537    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-7nvl9"]
May 25 11:18:19.100426 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:18:19.100403    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc18099d5_e3a6_4e23_b792_8fc91dcb9343.slice/crio-3aeed0590806365ee18841e43202b9b12cbc3d8aab9d9fd85b0db57b685548e0 WatchSource:0}: Error finding container 3aeed0590806365ee18841e43202b9b12cbc3d8aab9d9fd85b0db57b685548e0: Status 404 returned error can't find the container with id 3aeed0590806365ee18841e43202b9b12cbc3d8aab9d9fd85b0db57b685548e0
May 25 11:18:19.501505 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:19.501471    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7nvl9" event={"ID":"c18099d5-e3a6-4e23-b792-8fc91dcb9343","Type":"ContainerStarted","Data":"81eabe169cb7e9c0b76217bcc7d976efda36257f4466dbc04365b646e69fa170"}
May 25 11:18:19.501908 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:19.501518    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7nvl9" event={"ID":"c18099d5-e3a6-4e23-b792-8fc91dcb9343","Type":"ContainerStarted","Data":"3aeed0590806365ee18841e43202b9b12cbc3d8aab9d9fd85b0db57b685548e0"}
May 25 11:18:20.505844 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:20.505812    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7nvl9" event={"ID":"c18099d5-e3a6-4e23-b792-8fc91dcb9343","Type":"ContainerStarted","Data":"eb42d503fcd3961773ed4c5b00a43aca5a98157950003a4c17c2acf29f481c3d"}
May 25 11:18:22.513242 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:22.513208    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7nvl9" event={"ID":"c18099d5-e3a6-4e23-b792-8fc91dcb9343","Type":"ContainerStarted","Data":"35786efbe995c53f759beecb21a3ef2975b88d7a8b365c3fade0db51980ed7dc"}
May 25 11:18:22.544157 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:22.544101    2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-7nvl9" podStartSLOduration=1.723766527 podStartE2EDuration="4.544084647s" podCreationTimestamp="2026-05-25 11:18:18 +0000 UTC" firstStartedPulling="2026-05-25 11:18:19.230195989 +0000 UTC m=+72.775373579" lastFinishedPulling="2026-05-25 11:18:22.050514114 +0000 UTC m=+75.595691699" observedRunningTime="2026-05-25 11:18:22.543886661 +0000 UTC m=+76.089064266" watchObservedRunningTime="2026-05-25 11:18:22.544084647 +0000 UTC m=+76.089262252"
May 25 11:18:25.085596 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:25.085563    2561 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console/console-56544555f7-lczsn"]
May 25 11:18:25.102323 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:25.102287    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-56544555f7-lczsn"]
May 25 11:18:25.102465 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:25.102353    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-56544555f7-lczsn"
May 25 11:18:25.104817 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:25.104791    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
May 25 11:18:25.104952 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:25.104825    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
May 25 11:18:25.104952 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:25.104791    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
May 25 11:18:25.105861 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:25.105837    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
May 25 11:18:25.105995 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:25.105894    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
May 25 11:18:25.105995 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:25.105913    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
May 25 11:18:25.105995 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:25.105941    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-wkb8g\""
May 25 11:18:25.106172 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:25.106100    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
May 25 11:18:25.230867 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:25.230844    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/12498c7a-38d3-4424-919e-0e430256aa0c-service-ca\") pod \"console-56544555f7-lczsn\" (UID: \"12498c7a-38d3-4424-919e-0e430256aa0c\") " pod="openshift-console/console-56544555f7-lczsn"
May 25 11:18:25.230991 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:25.230875    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/12498c7a-38d3-4424-919e-0e430256aa0c-console-config\") pod \"console-56544555f7-lczsn\" (UID: \"12498c7a-38d3-4424-919e-0e430256aa0c\") " pod="openshift-console/console-56544555f7-lczsn"
May 25 11:18:25.230991 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:25.230900    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/12498c7a-38d3-4424-919e-0e430256aa0c-oauth-serving-cert\") pod \"console-56544555f7-lczsn\" (UID: \"12498c7a-38d3-4424-919e-0e430256aa0c\") " pod="openshift-console/console-56544555f7-lczsn"
May 25 11:18:25.230991 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:25.230960    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxwql\" (UniqueName: \"kubernetes.io/projected/12498c7a-38d3-4424-919e-0e430256aa0c-kube-api-access-qxwql\") pod \"console-56544555f7-lczsn\" (UID: \"12498c7a-38d3-4424-919e-0e430256aa0c\") " pod="openshift-console/console-56544555f7-lczsn"
May 25 11:18:25.231153 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:25.231074    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/12498c7a-38d3-4424-919e-0e430256aa0c-console-serving-cert\") pod \"console-56544555f7-lczsn\" (UID: \"12498c7a-38d3-4424-919e-0e430256aa0c\") " pod="openshift-console/console-56544555f7-lczsn"
May 25 11:18:25.231153 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:25.231140    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/12498c7a-38d3-4424-919e-0e430256aa0c-console-oauth-config\") pod \"console-56544555f7-lczsn\" (UID: \"12498c7a-38d3-4424-919e-0e430256aa0c\") " pod="openshift-console/console-56544555f7-lczsn"
May 25 11:18:25.332096 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:25.332066    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/12498c7a-38d3-4424-919e-0e430256aa0c-oauth-serving-cert\") pod \"console-56544555f7-lczsn\" (UID: \"12498c7a-38d3-4424-919e-0e430256aa0c\") " pod="openshift-console/console-56544555f7-lczsn"
May 25 11:18:25.332096 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:25.332098    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-qxwql\" (UniqueName: \"kubernetes.io/projected/12498c7a-38d3-4424-919e-0e430256aa0c-kube-api-access-qxwql\") pod \"console-56544555f7-lczsn\" (UID: \"12498c7a-38d3-4424-919e-0e430256aa0c\") " pod="openshift-console/console-56544555f7-lczsn"
May 25 11:18:25.332289 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:25.332119    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/12498c7a-38d3-4424-919e-0e430256aa0c-console-serving-cert\") pod \"console-56544555f7-lczsn\" (UID: \"12498c7a-38d3-4424-919e-0e430256aa0c\") " pod="openshift-console/console-56544555f7-lczsn"
May 25 11:18:25.332289 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:25.332153    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/12498c7a-38d3-4424-919e-0e430256aa0c-console-oauth-config\") pod \"console-56544555f7-lczsn\" (UID: \"12498c7a-38d3-4424-919e-0e430256aa0c\") " pod="openshift-console/console-56544555f7-lczsn"
May 25 11:18:25.332289 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:25.332175    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/12498c7a-38d3-4424-919e-0e430256aa0c-service-ca\") pod \"console-56544555f7-lczsn\" (UID: \"12498c7a-38d3-4424-919e-0e430256aa0c\") " pod="openshift-console/console-56544555f7-lczsn"
May 25 11:18:25.332289 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:25.332205    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/12498c7a-38d3-4424-919e-0e430256aa0c-console-config\") pod \"console-56544555f7-lczsn\" (UID: \"12498c7a-38d3-4424-919e-0e430256aa0c\") " pod="openshift-console/console-56544555f7-lczsn"
May 25 11:18:25.332834 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:25.332802    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/12498c7a-38d3-4424-919e-0e430256aa0c-oauth-serving-cert\") pod \"console-56544555f7-lczsn\" (UID: \"12498c7a-38d3-4424-919e-0e430256aa0c\") " pod="openshift-console/console-56544555f7-lczsn"
May 25 11:18:25.332936 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:25.332865    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/12498c7a-38d3-4424-919e-0e430256aa0c-console-config\") pod \"console-56544555f7-lczsn\" (UID: \"12498c7a-38d3-4424-919e-0e430256aa0c\") " pod="openshift-console/console-56544555f7-lczsn"
May 25 11:18:25.332936 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:25.332869    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/12498c7a-38d3-4424-919e-0e430256aa0c-service-ca\") pod \"console-56544555f7-lczsn\" (UID: \"12498c7a-38d3-4424-919e-0e430256aa0c\") " pod="openshift-console/console-56544555f7-lczsn"
May 25 11:18:25.334589 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:25.334572    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/12498c7a-38d3-4424-919e-0e430256aa0c-console-serving-cert\") pod \"console-56544555f7-lczsn\" (UID: \"12498c7a-38d3-4424-919e-0e430256aa0c\") " pod="openshift-console/console-56544555f7-lczsn"
May 25 11:18:25.334656 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:25.334628    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/12498c7a-38d3-4424-919e-0e430256aa0c-console-oauth-config\") pod \"console-56544555f7-lczsn\" (UID: \"12498c7a-38d3-4424-919e-0e430256aa0c\") " pod="openshift-console/console-56544555f7-lczsn"
May 25 11:18:25.341522 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:25.341470    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxwql\" (UniqueName: \"kubernetes.io/projected/12498c7a-38d3-4424-919e-0e430256aa0c-kube-api-access-qxwql\") pod \"console-56544555f7-lczsn\" (UID: \"12498c7a-38d3-4424-919e-0e430256aa0c\") " pod="openshift-console/console-56544555f7-lczsn"
May 25 11:18:25.412453 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:25.412424    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-56544555f7-lczsn"
May 25 11:18:25.545319 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:25.545290    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-56544555f7-lczsn"]
May 25 11:18:25.547917 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:18:25.547882    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12498c7a_38d3_4424_919e_0e430256aa0c.slice/crio-b5abac5deef213ba536a04b069287e5db4fb5441206154f24060990c8a83a69f WatchSource:0}: Error finding container b5abac5deef213ba536a04b069287e5db4fb5441206154f24060990c8a83a69f: Status 404 returned error can't find the container with id b5abac5deef213ba536a04b069287e5db4fb5441206154f24060990c8a83a69f
May 25 11:18:26.525719 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:26.525588    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-56544555f7-lczsn" event={"ID":"12498c7a-38d3-4424-919e-0e430256aa0c","Type":"ContainerStarted","Data":"b5abac5deef213ba536a04b069287e5db4fb5441206154f24060990c8a83a69f"}
May 25 11:18:28.503749 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:28.503670    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-d6xph"
May 25 11:18:28.532360 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:28.532326    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-56544555f7-lczsn" event={"ID":"12498c7a-38d3-4424-919e-0e430256aa0c","Type":"ContainerStarted","Data":"10d97418541642d142f78c765d0b08dc1de23422396366fe952edd5699e99701"}
May 25 11:18:28.563560 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:28.563515    2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console/console-56544555f7-lczsn" podStartSLOduration=1.003023732 podStartE2EDuration="3.563498391s" podCreationTimestamp="2026-05-25 11:18:25 +0000 UTC" firstStartedPulling="2026-05-25 11:18:25.54960767 +0000 UTC m=+79.094785255" lastFinishedPulling="2026-05-25 11:18:28.110082334 +0000 UTC m=+81.655259914" observedRunningTime="2026-05-25 11:18:28.561371538 +0000 UTC m=+82.106549165" watchObservedRunningTime="2026-05-25 11:18:28.563498391 +0000 UTC m=+82.108675994"
May 25 11:18:32.160229 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:32.160197    2561 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-54fd6bc5c4-n7x6m"]
May 25 11:18:32.162987 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:32.162970    2561 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-r8hl4"]
May 25 11:18:32.163136 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:32.163119    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-n7x6m"
May 25 11:18:32.166003 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:32.165983    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-r8hl4"
May 25 11:18:32.166550 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:32.166524    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
May 25 11:18:32.166841 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:32.166824    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
May 25 11:18:32.167117 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:32.167101    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-kvpl6\""
May 25 11:18:32.167487 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:32.167474    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
May 25 11:18:32.167545 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:32.167519    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
May 25 11:18:32.168507 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:32.168478    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
May 25 11:18:32.168689 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:32.168672    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
May 25 11:18:32.168773 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:32.168752    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
May 25 11:18:32.170735 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:32.170706    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-wmnhm\""
May 25 11:18:32.206214 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:32.206193    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-54fd6bc5c4-n7x6m"]
May 25 11:18:32.286898 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:32.286876    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/744866a4-40aa-4412-9df2-8d1a2f4d1ea2-node-exporter-accelerators-collector-config\") pod \"node-exporter-r8hl4\" (UID: \"744866a4-40aa-4412-9df2-8d1a2f4d1ea2\") " pod="openshift-monitoring/node-exporter-r8hl4"
May 25 11:18:32.286991 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:32.286909    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fn2zd\" (UniqueName: \"kubernetes.io/projected/4efe0bc1-af7a-4e08-bb46-9cac545bc309-kube-api-access-fn2zd\") pod \"kube-state-metrics-54fd6bc5c4-n7x6m\" (UID: \"4efe0bc1-af7a-4e08-bb46-9cac545bc309\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-n7x6m"
May 25 11:18:32.286991 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:32.286928    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/744866a4-40aa-4412-9df2-8d1a2f4d1ea2-sys\") pod \"node-exporter-r8hl4\" (UID: \"744866a4-40aa-4412-9df2-8d1a2f4d1ea2\") " pod="openshift-monitoring/node-exporter-r8hl4"
May 25 11:18:32.286991 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:32.286950    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/4efe0bc1-af7a-4e08-bb46-9cac545bc309-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-54fd6bc5c4-n7x6m\" (UID: \"4efe0bc1-af7a-4e08-bb46-9cac545bc309\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-n7x6m"
May 25 11:18:32.286991 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:32.286971    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/744866a4-40aa-4412-9df2-8d1a2f4d1ea2-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-r8hl4\" (UID: \"744866a4-40aa-4412-9df2-8d1a2f4d1ea2\") " pod="openshift-monitoring/node-exporter-r8hl4"
May 25 11:18:32.287127 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:32.287015    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/744866a4-40aa-4412-9df2-8d1a2f4d1ea2-node-exporter-wtmp\") pod \"node-exporter-r8hl4\" (UID: \"744866a4-40aa-4412-9df2-8d1a2f4d1ea2\") " pod="openshift-monitoring/node-exporter-r8hl4"
May 25 11:18:32.287127 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:32.287069    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-csdjz\" (UniqueName: \"kubernetes.io/projected/744866a4-40aa-4412-9df2-8d1a2f4d1ea2-kube-api-access-csdjz\") pod \"node-exporter-r8hl4\" (UID: \"744866a4-40aa-4412-9df2-8d1a2f4d1ea2\") " pod="openshift-monitoring/node-exporter-r8hl4"
May 25 11:18:32.287127 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:32.287088    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4efe0bc1-af7a-4e08-bb46-9cac545bc309-metrics-client-ca\") pod \"kube-state-metrics-54fd6bc5c4-n7x6m\" (UID: \"4efe0bc1-af7a-4e08-bb46-9cac545bc309\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-n7x6m"
May 25 11:18:32.287127 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:32.287115    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/744866a4-40aa-4412-9df2-8d1a2f4d1ea2-metrics-client-ca\") pod \"node-exporter-r8hl4\" (UID: \"744866a4-40aa-4412-9df2-8d1a2f4d1ea2\") " pod="openshift-monitoring/node-exporter-r8hl4"
May 25 11:18:32.287243 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:32.287134    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/4efe0bc1-af7a-4e08-bb46-9cac545bc309-volume-directive-shadow\") pod \"kube-state-metrics-54fd6bc5c4-n7x6m\" (UID: \"4efe0bc1-af7a-4e08-bb46-9cac545bc309\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-n7x6m"
May 25 11:18:32.287243 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:32.287157    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/4efe0bc1-af7a-4e08-bb46-9cac545bc309-kube-state-metrics-tls\") pod \"kube-state-metrics-54fd6bc5c4-n7x6m\" (UID: \"4efe0bc1-af7a-4e08-bb46-9cac545bc309\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-n7x6m"
May 25 11:18:32.287243 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:32.287185    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/744866a4-40aa-4412-9df2-8d1a2f4d1ea2-node-exporter-textfile\") pod \"node-exporter-r8hl4\" (UID: \"744866a4-40aa-4412-9df2-8d1a2f4d1ea2\") " pod="openshift-monitoring/node-exporter-r8hl4"
May 25 11:18:32.287243 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:32.287206    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4efe0bc1-af7a-4e08-bb46-9cac545bc309-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-54fd6bc5c4-n7x6m\" (UID: \"4efe0bc1-af7a-4e08-bb46-9cac545bc309\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-n7x6m"
May 25 11:18:32.287243 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:32.287237    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/744866a4-40aa-4412-9df2-8d1a2f4d1ea2-root\") pod \"node-exporter-r8hl4\" (UID: \"744866a4-40aa-4412-9df2-8d1a2f4d1ea2\") " pod="openshift-monitoring/node-exporter-r8hl4"
May 25 11:18:32.287409 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:32.287252    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/744866a4-40aa-4412-9df2-8d1a2f4d1ea2-node-exporter-tls\") pod \"node-exporter-r8hl4\" (UID: \"744866a4-40aa-4412-9df2-8d1a2f4d1ea2\") " pod="openshift-monitoring/node-exporter-r8hl4"
May 25 11:18:32.387955 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:32.387928    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-csdjz\" (UniqueName: \"kubernetes.io/projected/744866a4-40aa-4412-9df2-8d1a2f4d1ea2-kube-api-access-csdjz\") pod \"node-exporter-r8hl4\" (UID: \"744866a4-40aa-4412-9df2-8d1a2f4d1ea2\") " pod="openshift-monitoring/node-exporter-r8hl4"
May 25 11:18:32.388076 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:32.387960    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4efe0bc1-af7a-4e08-bb46-9cac545bc309-metrics-client-ca\") pod \"kube-state-metrics-54fd6bc5c4-n7x6m\" (UID: \"4efe0bc1-af7a-4e08-bb46-9cac545bc309\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-n7x6m"
May 25 11:18:32.388076 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:32.387990    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/744866a4-40aa-4412-9df2-8d1a2f4d1ea2-metrics-client-ca\") pod \"node-exporter-r8hl4\" (UID: \"744866a4-40aa-4412-9df2-8d1a2f4d1ea2\") " pod="openshift-monitoring/node-exporter-r8hl4"
May 25 11:18:32.388076 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:32.388012    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/4efe0bc1-af7a-4e08-bb46-9cac545bc309-volume-directive-shadow\") pod \"kube-state-metrics-54fd6bc5c4-n7x6m\" (UID: \"4efe0bc1-af7a-4e08-bb46-9cac545bc309\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-n7x6m"
May 25 11:18:32.388076 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:32.388035    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/4efe0bc1-af7a-4e08-bb46-9cac545bc309-kube-state-metrics-tls\") pod \"kube-state-metrics-54fd6bc5c4-n7x6m\" (UID: \"4efe0bc1-af7a-4e08-bb46-9cac545bc309\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-n7x6m"
May 25 11:18:32.388251 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:32.388121    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/744866a4-40aa-4412-9df2-8d1a2f4d1ea2-node-exporter-textfile\") pod \"node-exporter-r8hl4\" (UID: \"744866a4-40aa-4412-9df2-8d1a2f4d1ea2\") " pod="openshift-monitoring/node-exporter-r8hl4"
May 25 11:18:32.388251 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:32.388230    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4efe0bc1-af7a-4e08-bb46-9cac545bc309-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-54fd6bc5c4-n7x6m\" (UID: \"4efe0bc1-af7a-4e08-bb46-9cac545bc309\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-n7x6m"
May 25 11:18:32.388395 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:32.388306    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/744866a4-40aa-4412-9df2-8d1a2f4d1ea2-root\") pod \"node-exporter-r8hl4\" (UID: \"744866a4-40aa-4412-9df2-8d1a2f4d1ea2\") " pod="openshift-monitoring/node-exporter-r8hl4"
May 25 11:18:32.388395 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:32.388337    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/744866a4-40aa-4412-9df2-8d1a2f4d1ea2-node-exporter-tls\") pod \"node-exporter-r8hl4\" (UID: \"744866a4-40aa-4412-9df2-8d1a2f4d1ea2\") " pod="openshift-monitoring/node-exporter-r8hl4"
May 25 11:18:32.388492 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:32.388395    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/744866a4-40aa-4412-9df2-8d1a2f4d1ea2-node-exporter-accelerators-collector-config\") pod \"node-exporter-r8hl4\" (UID: \"744866a4-40aa-4412-9df2-8d1a2f4d1ea2\") " pod="openshift-monitoring/node-exporter-r8hl4"
May 25 11:18:32.388492 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:32.388438    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-fn2zd\" (UniqueName: \"kubernetes.io/projected/4efe0bc1-af7a-4e08-bb46-9cac545bc309-kube-api-access-fn2zd\") pod \"kube-state-metrics-54fd6bc5c4-n7x6m\" (UID: \"4efe0bc1-af7a-4e08-bb46-9cac545bc309\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-n7x6m"
May 25 11:18:32.388492 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:32.388470    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/744866a4-40aa-4412-9df2-8d1a2f4d1ea2-sys\") pod \"node-exporter-r8hl4\" (UID: \"744866a4-40aa-4412-9df2-8d1a2f4d1ea2\") " pod="openshift-monitoring/node-exporter-r8hl4"
May 25 11:18:32.388644 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:32.388508    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/4efe0bc1-af7a-4e08-bb46-9cac545bc309-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-54fd6bc5c4-n7x6m\" (UID: \"4efe0bc1-af7a-4e08-bb46-9cac545bc309\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-n7x6m"
May 25 11:18:32.388644 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:32.388510    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/4efe0bc1-af7a-4e08-bb46-9cac545bc309-volume-directive-shadow\") pod \"kube-state-metrics-54fd6bc5c4-n7x6m\" (UID: \"4efe0bc1-af7a-4e08-bb46-9cac545bc309\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-n7x6m"
May 25 11:18:32.388644 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:32.388519    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/744866a4-40aa-4412-9df2-8d1a2f4d1ea2-node-exporter-textfile\") pod \"node-exporter-r8hl4\" (UID: \"744866a4-40aa-4412-9df2-8d1a2f4d1ea2\") " pod="openshift-monitoring/node-exporter-r8hl4"
May 25 11:18:32.388644 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:32.388540    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/744866a4-40aa-4412-9df2-8d1a2f4d1ea2-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-r8hl4\" (UID: \"744866a4-40aa-4412-9df2-8d1a2f4d1ea2\") " pod="openshift-monitoring/node-exporter-r8hl4"
May 25 11:18:32.388644 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:32.388572    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/744866a4-40aa-4412-9df2-8d1a2f4d1ea2-node-exporter-wtmp\") pod \"node-exporter-r8hl4\" (UID: \"744866a4-40aa-4412-9df2-8d1a2f4d1ea2\") " pod="openshift-monitoring/node-exporter-r8hl4"
May 25 11:18:32.388644 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:32.388590    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/744866a4-40aa-4412-9df2-8d1a2f4d1ea2-sys\") pod \"node-exporter-r8hl4\" (UID: \"744866a4-40aa-4412-9df2-8d1a2f4d1ea2\") " pod="openshift-monitoring/node-exporter-r8hl4"
May 25 11:18:32.388921 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:32.388664    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/744866a4-40aa-4412-9df2-8d1a2f4d1ea2-root\") pod \"node-exporter-r8hl4\" (UID: \"744866a4-40aa-4412-9df2-8d1a2f4d1ea2\") " pod="openshift-monitoring/node-exporter-r8hl4"
May 25 11:18:32.388921 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:32.388699    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/744866a4-40aa-4412-9df2-8d1a2f4d1ea2-metrics-client-ca\") pod \"node-exporter-r8hl4\" (UID: \"744866a4-40aa-4412-9df2-8d1a2f4d1ea2\") " pod="openshift-monitoring/node-exporter-r8hl4"
May 25 11:18:32.388921 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:32.388767    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4efe0bc1-af7a-4e08-bb46-9cac545bc309-metrics-client-ca\") pod \"kube-state-metrics-54fd6bc5c4-n7x6m\" (UID: \"4efe0bc1-af7a-4e08-bb46-9cac545bc309\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-n7x6m"
May 25 11:18:32.388921 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:32.388841    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/744866a4-40aa-4412-9df2-8d1a2f4d1ea2-node-exporter-wtmp\") pod \"node-exporter-r8hl4\" (UID: \"744866a4-40aa-4412-9df2-8d1a2f4d1ea2\") " pod="openshift-monitoring/node-exporter-r8hl4"
May 25 11:18:32.389201 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:32.389177    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/744866a4-40aa-4412-9df2-8d1a2f4d1ea2-node-exporter-accelerators-collector-config\") pod \"node-exporter-r8hl4\" (UID: \"744866a4-40aa-4412-9df2-8d1a2f4d1ea2\") " pod="openshift-monitoring/node-exporter-r8hl4"
May 25 11:18:32.389311 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:32.389291    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/4efe0bc1-af7a-4e08-bb46-9cac545bc309-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-54fd6bc5c4-n7x6m\" (UID: \"4efe0bc1-af7a-4e08-bb46-9cac545bc309\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-n7x6m"
May 25 11:18:32.390794 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:32.390771    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/744866a4-40aa-4412-9df2-8d1a2f4d1ea2-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-r8hl4\" (UID: \"744866a4-40aa-4412-9df2-8d1a2f4d1ea2\") " pod="openshift-monitoring/node-exporter-r8hl4"
May 25 11:18:32.390929 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:32.390912    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/744866a4-40aa-4412-9df2-8d1a2f4d1ea2-node-exporter-tls\") pod \"node-exporter-r8hl4\" (UID: \"744866a4-40aa-4412-9df2-8d1a2f4d1ea2\") " pod="openshift-monitoring/node-exporter-r8hl4"
May 25 11:18:32.391147 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:32.391129    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4efe0bc1-af7a-4e08-bb46-9cac545bc309-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-54fd6bc5c4-n7x6m\" (UID: \"4efe0bc1-af7a-4e08-bb46-9cac545bc309\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-n7x6m"
May 25 11:18:32.391320 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:32.391297    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/4efe0bc1-af7a-4e08-bb46-9cac545bc309-kube-state-metrics-tls\") pod \"kube-state-metrics-54fd6bc5c4-n7x6m\" (UID: \"4efe0bc1-af7a-4e08-bb46-9cac545bc309\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-n7x6m"
May 25 11:18:32.410664 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:32.410608    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-csdjz\" (UniqueName: \"kubernetes.io/projected/744866a4-40aa-4412-9df2-8d1a2f4d1ea2-kube-api-access-csdjz\") pod \"node-exporter-r8hl4\" (UID: \"744866a4-40aa-4412-9df2-8d1a2f4d1ea2\") " pod="openshift-monitoring/node-exporter-r8hl4"
May 25 11:18:32.414093 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:32.414070    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fn2zd\" (UniqueName: \"kubernetes.io/projected/4efe0bc1-af7a-4e08-bb46-9cac545bc309-kube-api-access-fn2zd\") pod \"kube-state-metrics-54fd6bc5c4-n7x6m\" (UID: \"4efe0bc1-af7a-4e08-bb46-9cac545bc309\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-n7x6m"
May 25 11:18:32.473409 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:32.473390    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-n7x6m"
May 25 11:18:32.479366 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:32.479345    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-r8hl4"
May 25 11:18:32.488652 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:18:32.488622    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod744866a4_40aa_4412_9df2_8d1a2f4d1ea2.slice/crio-d7a8d1a941197db23577ddd6c1370859162c73afe20a1dc562c4d2b222c9d250 WatchSource:0}: Error finding container d7a8d1a941197db23577ddd6c1370859162c73afe20a1dc562c4d2b222c9d250: Status 404 returned error can't find the container with id d7a8d1a941197db23577ddd6c1370859162c73afe20a1dc562c4d2b222c9d250
May 25 11:18:32.544000 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:32.543966    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-r8hl4" event={"ID":"744866a4-40aa-4412-9df2-8d1a2f4d1ea2","Type":"ContainerStarted","Data":"d7a8d1a941197db23577ddd6c1370859162c73afe20a1dc562c4d2b222c9d250"}
May 25 11:18:32.676530 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:18:32.676488    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4efe0bc1_af7a_4e08_bb46_9cac545bc309.slice/crio-64ad17ee365eeaa27d52c1c3ac7ae00082f38ef05e12410ddd6f2f05892ec97c WatchSource:0}: Error finding container 64ad17ee365eeaa27d52c1c3ac7ae00082f38ef05e12410ddd6f2f05892ec97c: Status 404 returned error can't find the container with id 64ad17ee365eeaa27d52c1c3ac7ae00082f38ef05e12410ddd6f2f05892ec97c
May 25 11:18:32.677388 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:32.677351    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-54fd6bc5c4-n7x6m"]
May 25 11:18:33.164372 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:33.164322    2561 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
May 25 11:18:33.167559 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:33.167536    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
May 25 11:18:33.171033 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:33.170731    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
May 25 11:18:33.171033 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:33.170782    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
May 25 11:18:33.171033 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:33.170788    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
May 25 11:18:33.171297 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:33.171072    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
May 25 11:18:33.171950 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:33.171928    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
May 25 11:18:33.172107 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:33.171969    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-k6rq2\""
May 25 11:18:33.172340 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:33.172186    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
May 25 11:18:33.172340 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:33.172255    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
May 25 11:18:33.172949 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:33.172928    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
May 25 11:18:33.172949 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:33.172946    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
May 25 11:18:33.209729 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:33.209682    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
May 25 11:18:33.295725 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:33.295577    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5805521a-51c5-4810-9cf6-ba306bfb50d9-config-out\") pod \"alertmanager-main-0\" (UID: \"5805521a-51c5-4810-9cf6-ba306bfb50d9\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 11:18:33.295725 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:33.295618    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5805521a-51c5-4810-9cf6-ba306bfb50d9-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"5805521a-51c5-4810-9cf6-ba306bfb50d9\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 11:18:33.295725 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:33.295666    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5805521a-51c5-4810-9cf6-ba306bfb50d9-tls-assets\") pod \"alertmanager-main-0\" (UID: \"5805521a-51c5-4810-9cf6-ba306bfb50d9\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 11:18:33.295946 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:33.295723    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5805521a-51c5-4810-9cf6-ba306bfb50d9-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"5805521a-51c5-4810-9cf6-ba306bfb50d9\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 11:18:33.295946 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:33.295758    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5805521a-51c5-4810-9cf6-ba306bfb50d9-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"5805521a-51c5-4810-9cf6-ba306bfb50d9\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 11:18:33.295946 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:33.295806    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5805521a-51c5-4810-9cf6-ba306bfb50d9-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"5805521a-51c5-4810-9cf6-ba306bfb50d9\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 11:18:33.295946 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:33.295829    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rp6fg\" (UniqueName: \"kubernetes.io/projected/5805521a-51c5-4810-9cf6-ba306bfb50d9-kube-api-access-rp6fg\") pod \"alertmanager-main-0\" (UID: \"5805521a-51c5-4810-9cf6-ba306bfb50d9\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 11:18:33.295946 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:33.295854    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5805521a-51c5-4810-9cf6-ba306bfb50d9-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"5805521a-51c5-4810-9cf6-ba306bfb50d9\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 11:18:33.295946 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:33.295919    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5805521a-51c5-4810-9cf6-ba306bfb50d9-config-volume\") pod \"alertmanager-main-0\" (UID: \"5805521a-51c5-4810-9cf6-ba306bfb50d9\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 11:18:33.296244 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:33.295970    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5805521a-51c5-4810-9cf6-ba306bfb50d9-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"5805521a-51c5-4810-9cf6-ba306bfb50d9\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 11:18:33.296244 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:33.296016    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5805521a-51c5-4810-9cf6-ba306bfb50d9-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"5805521a-51c5-4810-9cf6-ba306bfb50d9\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 11:18:33.296244 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:33.296041    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5805521a-51c5-4810-9cf6-ba306bfb50d9-web-config\") pod \"alertmanager-main-0\" (UID: \"5805521a-51c5-4810-9cf6-ba306bfb50d9\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 11:18:33.296244 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:33.296068    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5805521a-51c5-4810-9cf6-ba306bfb50d9-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"5805521a-51c5-4810-9cf6-ba306bfb50d9\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 11:18:33.396958 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:33.396927    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5805521a-51c5-4810-9cf6-ba306bfb50d9-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"5805521a-51c5-4810-9cf6-ba306bfb50d9\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 11:18:33.397109 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:33.396971    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5805521a-51c5-4810-9cf6-ba306bfb50d9-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"5805521a-51c5-4810-9cf6-ba306bfb50d9\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 11:18:33.397109 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:33.397016    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5805521a-51c5-4810-9cf6-ba306bfb50d9-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"5805521a-51c5-4810-9cf6-ba306bfb50d9\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 11:18:33.397109 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:33.397042    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-rp6fg\" (UniqueName: \"kubernetes.io/projected/5805521a-51c5-4810-9cf6-ba306bfb50d9-kube-api-access-rp6fg\") pod \"alertmanager-main-0\" (UID: \"5805521a-51c5-4810-9cf6-ba306bfb50d9\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 11:18:33.397359 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:33.397193    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5805521a-51c5-4810-9cf6-ba306bfb50d9-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"5805521a-51c5-4810-9cf6-ba306bfb50d9\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 11:18:33.397359 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:33.397240    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5805521a-51c5-4810-9cf6-ba306bfb50d9-config-volume\") pod \"alertmanager-main-0\" (UID: \"5805521a-51c5-4810-9cf6-ba306bfb50d9\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 11:18:33.397359 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:33.397297    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5805521a-51c5-4810-9cf6-ba306bfb50d9-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"5805521a-51c5-4810-9cf6-ba306bfb50d9\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 11:18:33.397359 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:33.397338    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5805521a-51c5-4810-9cf6-ba306bfb50d9-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"5805521a-51c5-4810-9cf6-ba306bfb50d9\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 11:18:33.397575 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:33.397363    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5805521a-51c5-4810-9cf6-ba306bfb50d9-web-config\") pod \"alertmanager-main-0\" (UID: \"5805521a-51c5-4810-9cf6-ba306bfb50d9\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 11:18:33.397575 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:33.397390    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5805521a-51c5-4810-9cf6-ba306bfb50d9-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"5805521a-51c5-4810-9cf6-ba306bfb50d9\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 11:18:33.397575 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:33.397428    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5805521a-51c5-4810-9cf6-ba306bfb50d9-config-out\") pod \"alertmanager-main-0\" (UID: \"5805521a-51c5-4810-9cf6-ba306bfb50d9\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 11:18:33.397575 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:33.397454    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5805521a-51c5-4810-9cf6-ba306bfb50d9-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"5805521a-51c5-4810-9cf6-ba306bfb50d9\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 11:18:33.397575 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:33.397508    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5805521a-51c5-4810-9cf6-ba306bfb50d9-tls-assets\") pod \"alertmanager-main-0\" (UID: \"5805521a-51c5-4810-9cf6-ba306bfb50d9\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 11:18:33.399037 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:33.397999    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5805521a-51c5-4810-9cf6-ba306bfb50d9-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"5805521a-51c5-4810-9cf6-ba306bfb50d9\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 11:18:33.399037 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:33.398411    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5805521a-51c5-4810-9cf6-ba306bfb50d9-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"5805521a-51c5-4810-9cf6-ba306bfb50d9\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 11:18:33.399037 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:33.398547    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5805521a-51c5-4810-9cf6-ba306bfb50d9-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"5805521a-51c5-4810-9cf6-ba306bfb50d9\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 11:18:33.399037 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:18:33.398657    2561 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found
May 25 11:18:33.399037 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:18:33.398715    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5805521a-51c5-4810-9cf6-ba306bfb50d9-secret-alertmanager-main-tls podName:5805521a-51c5-4810-9cf6-ba306bfb50d9 nodeName:}" failed. No retries permitted until 2026-05-25 11:18:33.898695406 +0000 UTC m=+87.443872990 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/5805521a-51c5-4810-9cf6-ba306bfb50d9-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "5805521a-51c5-4810-9cf6-ba306bfb50d9") : secret "alertmanager-main-tls" not found
May 25 11:18:33.400337 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:33.400317    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5805521a-51c5-4810-9cf6-ba306bfb50d9-tls-assets\") pod \"alertmanager-main-0\" (UID: \"5805521a-51c5-4810-9cf6-ba306bfb50d9\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 11:18:33.401207 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:33.401164    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5805521a-51c5-4810-9cf6-ba306bfb50d9-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"5805521a-51c5-4810-9cf6-ba306bfb50d9\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 11:18:33.401843 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:33.401776    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5805521a-51c5-4810-9cf6-ba306bfb50d9-config-volume\") pod \"alertmanager-main-0\" (UID: \"5805521a-51c5-4810-9cf6-ba306bfb50d9\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 11:18:33.401843 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:33.401815    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5805521a-51c5-4810-9cf6-ba306bfb50d9-config-out\") pod \"alertmanager-main-0\" (UID: \"5805521a-51c5-4810-9cf6-ba306bfb50d9\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 11:18:33.402134 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:33.402108    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5805521a-51c5-4810-9cf6-ba306bfb50d9-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"5805521a-51c5-4810-9cf6-ba306bfb50d9\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 11:18:33.402228 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:33.402209    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5805521a-51c5-4810-9cf6-ba306bfb50d9-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"5805521a-51c5-4810-9cf6-ba306bfb50d9\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 11:18:33.402898 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:33.402857    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5805521a-51c5-4810-9cf6-ba306bfb50d9-web-config\") pod \"alertmanager-main-0\" (UID: \"5805521a-51c5-4810-9cf6-ba306bfb50d9\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 11:18:33.403369 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:33.403346    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5805521a-51c5-4810-9cf6-ba306bfb50d9-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"5805521a-51c5-4810-9cf6-ba306bfb50d9\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 11:18:33.423483 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:33.423423    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rp6fg\" (UniqueName: \"kubernetes.io/projected/5805521a-51c5-4810-9cf6-ba306bfb50d9-kube-api-access-rp6fg\") pod \"alertmanager-main-0\" (UID: \"5805521a-51c5-4810-9cf6-ba306bfb50d9\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 11:18:33.552809 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:33.552773    2561 generic.go:358] "Generic (PLEG): container finished" podID="744866a4-40aa-4412-9df2-8d1a2f4d1ea2" containerID="4916e14eba795f9a0521a9b3b3a465af100e70a1b084f0cd31af1e70c2c28043" exitCode=0
May 25 11:18:33.552973 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:33.552865    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-r8hl4" event={"ID":"744866a4-40aa-4412-9df2-8d1a2f4d1ea2","Type":"ContainerDied","Data":"4916e14eba795f9a0521a9b3b3a465af100e70a1b084f0cd31af1e70c2c28043"}
May 25 11:18:33.554139 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:33.554113    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-n7x6m" event={"ID":"4efe0bc1-af7a-4e08-bb46-9cac545bc309","Type":"ContainerStarted","Data":"64ad17ee365eeaa27d52c1c3ac7ae00082f38ef05e12410ddd6f2f05892ec97c"}
May 25 11:18:33.903490 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:33.902843    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5805521a-51c5-4810-9cf6-ba306bfb50d9-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"5805521a-51c5-4810-9cf6-ba306bfb50d9\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 11:18:33.908463 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:33.908437    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5805521a-51c5-4810-9cf6-ba306bfb50d9-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"5805521a-51c5-4810-9cf6-ba306bfb50d9\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 11:18:34.080720 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:34.080685    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
May 25 11:18:34.222997 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:34.222972    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
May 25 11:18:34.225084 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:18:34.225055    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5805521a_51c5_4810_9cf6_ba306bfb50d9.slice/crio-0a7ed63ea4a5c20053794ffb3ef3dc12fbb037022771a7ab7b035d142ac730c3 WatchSource:0}: Error finding container 0a7ed63ea4a5c20053794ffb3ef3dc12fbb037022771a7ab7b035d142ac730c3: Status 404 returned error can't find the container with id 0a7ed63ea4a5c20053794ffb3ef3dc12fbb037022771a7ab7b035d142ac730c3
May 25 11:18:34.477731 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:34.477668    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7b9fd9b9cb-fpswn"
May 25 11:18:34.559586 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:34.559549    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-r8hl4" event={"ID":"744866a4-40aa-4412-9df2-8d1a2f4d1ea2","Type":"ContainerStarted","Data":"633496f60633597f6f36e18b7e56cb73ae4431eb6e725f5867cc278fc4be5a52"}
May 25 11:18:34.559735 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:34.559593    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-r8hl4" event={"ID":"744866a4-40aa-4412-9df2-8d1a2f4d1ea2","Type":"ContainerStarted","Data":"33af9b06ce27a7bfce3d1981708327d3b82a96fc0c692f811ca069a5512e2050"}
May 25 11:18:34.561345 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:34.561314    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-n7x6m" event={"ID":"4efe0bc1-af7a-4e08-bb46-9cac545bc309","Type":"ContainerStarted","Data":"7c0123297437204eaad1f1a3af6226168f8412ca1cda07f414f441bfabc7445f"}
May 25 11:18:34.561345 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:34.561343    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-n7x6m" event={"ID":"4efe0bc1-af7a-4e08-bb46-9cac545bc309","Type":"ContainerStarted","Data":"ac0a8df758b6ea7db90f87f3173393bc9d685b31cd6b9686a02e20203715fc6f"}
May 25 11:18:34.561529 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:34.561357    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-n7x6m" event={"ID":"4efe0bc1-af7a-4e08-bb46-9cac545bc309","Type":"ContainerStarted","Data":"5f007a9608116d3344b34e27d462c0d154ac4dc8c183622773d0c847fad845cb"}
May 25 11:18:34.562324 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:34.562292    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5805521a-51c5-4810-9cf6-ba306bfb50d9","Type":"ContainerStarted","Data":"0a7ed63ea4a5c20053794ffb3ef3dc12fbb037022771a7ab7b035d142ac730c3"}
May 25 11:18:34.580121 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:34.580078    2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-r8hl4" podStartSLOduration=1.795700831 podStartE2EDuration="2.580063181s" podCreationTimestamp="2026-05-25 11:18:32 +0000 UTC" firstStartedPulling="2026-05-25 11:18:32.490138877 +0000 UTC m=+86.035316457" lastFinishedPulling="2026-05-25 11:18:33.274501222 +0000 UTC m=+86.819678807" observedRunningTime="2026-05-25 11:18:34.578853071 +0000 UTC m=+88.124030674" watchObservedRunningTime="2026-05-25 11:18:34.580063181 +0000 UTC m=+88.125240785"
May 25 11:18:34.598036 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:34.597994    2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-n7x6m" podStartSLOduration=1.441938177 podStartE2EDuration="2.597983307s" podCreationTimestamp="2026-05-25 11:18:32 +0000 UTC" firstStartedPulling="2026-05-25 11:18:32.678584952 +0000 UTC m=+86.223762546" lastFinishedPulling="2026-05-25 11:18:33.834630093 +0000 UTC m=+87.379807676" observedRunningTime="2026-05-25 11:18:34.5970113 +0000 UTC m=+88.142188906" watchObservedRunningTime="2026-05-25 11:18:34.597983307 +0000 UTC m=+88.143160909"
May 25 11:18:35.412839 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:35.412808    2561 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-56544555f7-lczsn"
May 25 11:18:35.413224 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:35.412850    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-56544555f7-lczsn"
May 25 11:18:35.417481 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:35.417461    2561 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-56544555f7-lczsn"
May 25 11:18:35.566363 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:35.566255    2561 generic.go:358] "Generic (PLEG): container finished" podID="5805521a-51c5-4810-9cf6-ba306bfb50d9" containerID="77fb2740ebe94c49986d18fe57931a73e6da46f8bbf9f6166aa62b6e1b70ab28" exitCode=0
May 25 11:18:35.566363 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:35.566350    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5805521a-51c5-4810-9cf6-ba306bfb50d9","Type":"ContainerDied","Data":"77fb2740ebe94c49986d18fe57931a73e6da46f8bbf9f6166aa62b6e1b70ab28"}
May 25 11:18:35.570466 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:35.570448    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-56544555f7-lczsn"
May 25 11:18:37.577532 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:37.577498    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5805521a-51c5-4810-9cf6-ba306bfb50d9","Type":"ContainerStarted","Data":"3d1dbdc6eed315467e25094ce0fc197b9c952d54bef5721ee7849bd090e96a80"}
May 25 11:18:37.577532 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:37.577537    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5805521a-51c5-4810-9cf6-ba306bfb50d9","Type":"ContainerStarted","Data":"cd02c74d52650e3e7e20d32f9f80033352f5ccbaaa7b7474a50b996eeaa1d4fe"}
May 25 11:18:37.577908 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:37.577550    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5805521a-51c5-4810-9cf6-ba306bfb50d9","Type":"ContainerStarted","Data":"73c9ecb470e2b22eac14037ba0972ae6efa213c1e92c0afa10ae424d5715160d"}
May 25 11:18:37.577908 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:37.577562    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5805521a-51c5-4810-9cf6-ba306bfb50d9","Type":"ContainerStarted","Data":"22b12500d9d4a100d1ec3b2ad330660b597d416a172c59244eb5a9a819068c8c"}
May 25 11:18:37.577908 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:37.577575    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5805521a-51c5-4810-9cf6-ba306bfb50d9","Type":"ContainerStarted","Data":"1f4635273454350be029575dd68c8b83dc653de3ddac1cc66544bfc46af2023b"}
May 25 11:18:38.583247 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:38.583212    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5805521a-51c5-4810-9cf6-ba306bfb50d9","Type":"ContainerStarted","Data":"c595ca6d95d1e83b483036990a03d60a9e0e3951f796199cb98404528b1c8e03"}
May 25 11:18:38.613918 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:38.613867    2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.149435228 podStartE2EDuration="5.613850105s" podCreationTimestamp="2026-05-25 11:18:33 +0000 UTC" firstStartedPulling="2026-05-25 11:18:34.227203035 +0000 UTC m=+87.772380618" lastFinishedPulling="2026-05-25 11:18:37.691617915 +0000 UTC m=+91.236795495" observedRunningTime="2026-05-25 11:18:38.613704863 +0000 UTC m=+92.158882466" watchObservedRunningTime="2026-05-25 11:18:38.613850105 +0000 UTC m=+92.159027708"
May 25 11:18:40.672836 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:40.672799    2561 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7b9fd9b9cb-fpswn"]
May 25 11:18:41.774190 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:41.774156    2561 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console/console-77d77bf5f6-ggwqf"]
May 25 11:18:41.777649 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:41.777630    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-77d77bf5f6-ggwqf"
May 25 11:18:41.785435 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:41.785415    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
May 25 11:18:41.793002 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:41.792981    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-77d77bf5f6-ggwqf"]
May 25 11:18:41.872865 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:41.872836    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b9e4ef06-b5da-4a36-846a-f21769ac72d1-service-ca\") pod \"console-77d77bf5f6-ggwqf\" (UID: \"b9e4ef06-b5da-4a36-846a-f21769ac72d1\") " pod="openshift-console/console-77d77bf5f6-ggwqf"
May 25 11:18:41.873015 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:41.872888    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b9e4ef06-b5da-4a36-846a-f21769ac72d1-trusted-ca-bundle\") pod \"console-77d77bf5f6-ggwqf\" (UID: \"b9e4ef06-b5da-4a36-846a-f21769ac72d1\") " pod="openshift-console/console-77d77bf5f6-ggwqf"
May 25 11:18:41.873015 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:41.872944    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b9e4ef06-b5da-4a36-846a-f21769ac72d1-console-serving-cert\") pod \"console-77d77bf5f6-ggwqf\" (UID: \"b9e4ef06-b5da-4a36-846a-f21769ac72d1\") " pod="openshift-console/console-77d77bf5f6-ggwqf"
May 25 11:18:41.873015 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:41.872973    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b9e4ef06-b5da-4a36-846a-f21769ac72d1-console-oauth-config\") pod \"console-77d77bf5f6-ggwqf\" (UID: \"b9e4ef06-b5da-4a36-846a-f21769ac72d1\") " pod="openshift-console/console-77d77bf5f6-ggwqf"
May 25 11:18:41.873015 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:41.873011    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b9e4ef06-b5da-4a36-846a-f21769ac72d1-console-config\") pod \"console-77d77bf5f6-ggwqf\" (UID: \"b9e4ef06-b5da-4a36-846a-f21769ac72d1\") " pod="openshift-console/console-77d77bf5f6-ggwqf"
May 25 11:18:41.873145 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:41.873039    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b9e4ef06-b5da-4a36-846a-f21769ac72d1-oauth-serving-cert\") pod \"console-77d77bf5f6-ggwqf\" (UID: \"b9e4ef06-b5da-4a36-846a-f21769ac72d1\") " pod="openshift-console/console-77d77bf5f6-ggwqf"
May 25 11:18:41.873145 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:41.873069    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltbbv\" (UniqueName: \"kubernetes.io/projected/b9e4ef06-b5da-4a36-846a-f21769ac72d1-kube-api-access-ltbbv\") pod \"console-77d77bf5f6-ggwqf\" (UID: \"b9e4ef06-b5da-4a36-846a-f21769ac72d1\") " pod="openshift-console/console-77d77bf5f6-ggwqf"
May 25 11:18:41.973912 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:41.973883    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b9e4ef06-b5da-4a36-846a-f21769ac72d1-console-oauth-config\") pod \"console-77d77bf5f6-ggwqf\" (UID: \"b9e4ef06-b5da-4a36-846a-f21769ac72d1\") " pod="openshift-console/console-77d77bf5f6-ggwqf"
May 25 11:18:41.973912 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:41.973913    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b9e4ef06-b5da-4a36-846a-f21769ac72d1-console-config\") pod \"console-77d77bf5f6-ggwqf\" (UID: \"b9e4ef06-b5da-4a36-846a-f21769ac72d1\") " pod="openshift-console/console-77d77bf5f6-ggwqf"
May 25 11:18:41.974103 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:41.973935    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b9e4ef06-b5da-4a36-846a-f21769ac72d1-oauth-serving-cert\") pod \"console-77d77bf5f6-ggwqf\" (UID: \"b9e4ef06-b5da-4a36-846a-f21769ac72d1\") " pod="openshift-console/console-77d77bf5f6-ggwqf"
May 25 11:18:41.974103 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:41.973952    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-ltbbv\" (UniqueName: \"kubernetes.io/projected/b9e4ef06-b5da-4a36-846a-f21769ac72d1-kube-api-access-ltbbv\") pod \"console-77d77bf5f6-ggwqf\" (UID: \"b9e4ef06-b5da-4a36-846a-f21769ac72d1\") " pod="openshift-console/console-77d77bf5f6-ggwqf"
May 25 11:18:41.974103 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:41.973979    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b9e4ef06-b5da-4a36-846a-f21769ac72d1-service-ca\") pod \"console-77d77bf5f6-ggwqf\" (UID: \"b9e4ef06-b5da-4a36-846a-f21769ac72d1\") " pod="openshift-console/console-77d77bf5f6-ggwqf"
May 25 11:18:41.974103 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:41.974016    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b9e4ef06-b5da-4a36-846a-f21769ac72d1-trusted-ca-bundle\") pod \"console-77d77bf5f6-ggwqf\" (UID: \"b9e4ef06-b5da-4a36-846a-f21769ac72d1\") " pod="openshift-console/console-77d77bf5f6-ggwqf"
May 25 11:18:41.974103 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:41.974043    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b9e4ef06-b5da-4a36-846a-f21769ac72d1-console-serving-cert\") pod \"console-77d77bf5f6-ggwqf\" (UID: \"b9e4ef06-b5da-4a36-846a-f21769ac72d1\") " pod="openshift-console/console-77d77bf5f6-ggwqf"
May 25 11:18:41.974737 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:41.974705    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b9e4ef06-b5da-4a36-846a-f21769ac72d1-console-config\") pod \"console-77d77bf5f6-ggwqf\" (UID: \"b9e4ef06-b5da-4a36-846a-f21769ac72d1\") " pod="openshift-console/console-77d77bf5f6-ggwqf"
May 25 11:18:41.974850 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:41.974746    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b9e4ef06-b5da-4a36-846a-f21769ac72d1-oauth-serving-cert\") pod \"console-77d77bf5f6-ggwqf\" (UID: \"b9e4ef06-b5da-4a36-846a-f21769ac72d1\") " pod="openshift-console/console-77d77bf5f6-ggwqf"
May 25 11:18:41.974850 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:41.974802    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b9e4ef06-b5da-4a36-846a-f21769ac72d1-service-ca\") pod \"console-77d77bf5f6-ggwqf\" (UID: \"b9e4ef06-b5da-4a36-846a-f21769ac72d1\") " pod="openshift-console/console-77d77bf5f6-ggwqf"
May 25 11:18:41.974850 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:41.974808    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b9e4ef06-b5da-4a36-846a-f21769ac72d1-trusted-ca-bundle\") pod \"console-77d77bf5f6-ggwqf\" (UID: \"b9e4ef06-b5da-4a36-846a-f21769ac72d1\") " pod="openshift-console/console-77d77bf5f6-ggwqf"
May 25 11:18:41.976517 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:41.976494    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b9e4ef06-b5da-4a36-846a-f21769ac72d1-console-oauth-config\") pod \"console-77d77bf5f6-ggwqf\" (UID: \"b9e4ef06-b5da-4a36-846a-f21769ac72d1\") " pod="openshift-console/console-77d77bf5f6-ggwqf"
May 25 11:18:41.976604 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:41.976592    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b9e4ef06-b5da-4a36-846a-f21769ac72d1-console-serving-cert\") pod \"console-77d77bf5f6-ggwqf\" (UID: \"b9e4ef06-b5da-4a36-846a-f21769ac72d1\") " pod="openshift-console/console-77d77bf5f6-ggwqf"
May 25 11:18:41.983740 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:41.983722    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltbbv\" (UniqueName: \"kubernetes.io/projected/b9e4ef06-b5da-4a36-846a-f21769ac72d1-kube-api-access-ltbbv\") pod \"console-77d77bf5f6-ggwqf\" (UID: \"b9e4ef06-b5da-4a36-846a-f21769ac72d1\") " pod="openshift-console/console-77d77bf5f6-ggwqf"
May 25 11:18:42.088109 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:42.088079    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-77d77bf5f6-ggwqf"
May 25 11:18:42.229378 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:42.229348    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-77d77bf5f6-ggwqf"]
May 25 11:18:42.231415 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:18:42.231393    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9e4ef06_b5da_4a36_846a_f21769ac72d1.slice/crio-98ce06e0b39540021c1e2104a03ef1cd3367be27dd1a115728b13b676ac0abf4 WatchSource:0}: Error finding container 98ce06e0b39540021c1e2104a03ef1cd3367be27dd1a115728b13b676ac0abf4: Status 404 returned error can't find the container with id 98ce06e0b39540021c1e2104a03ef1cd3367be27dd1a115728b13b676ac0abf4
May 25 11:18:42.595725 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:42.595688    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-77d77bf5f6-ggwqf" event={"ID":"b9e4ef06-b5da-4a36-846a-f21769ac72d1","Type":"ContainerStarted","Data":"8dfa963c42ac57cb9d0deca58cc4f6eb5fd09459fe7016d28eca7f16378dda8e"}
May 25 11:18:42.595725 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:42.595731    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-77d77bf5f6-ggwqf" event={"ID":"b9e4ef06-b5da-4a36-846a-f21769ac72d1","Type":"ContainerStarted","Data":"98ce06e0b39540021c1e2104a03ef1cd3367be27dd1a115728b13b676ac0abf4"}
May 25 11:18:42.617056 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:42.617012    2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console/console-77d77bf5f6-ggwqf" podStartSLOduration=1.61699913 podStartE2EDuration="1.61699913s" podCreationTimestamp="2026-05-25 11:18:41 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 11:18:42.615771446 +0000 UTC m=+96.160949052" watchObservedRunningTime="2026-05-25 11:18:42.61699913 +0000 UTC m=+96.162176736"
May 25 11:18:44.476366 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:44.476332    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-rl7hd"
May 25 11:18:52.089263 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:52.089229    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-77d77bf5f6-ggwqf"
May 25 11:18:52.089962 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:52.089388    2561 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-77d77bf5f6-ggwqf"
May 25 11:18:52.094069 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:52.094046    2561 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-77d77bf5f6-ggwqf"
May 25 11:18:52.630518 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:52.630489    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-77d77bf5f6-ggwqf"
May 25 11:18:52.683526 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:18:52.683496    2561 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-56544555f7-lczsn"]
May 25 11:19:01.654039 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:01.654008    2561 generic.go:358] "Generic (PLEG): container finished" podID="a9377ad6-eac5-4df9-b7a2-b479c9c86277" containerID="6b10e7c715738a0e361b7f1cf4eff3fa9fd6ff4ae718cdbc8026558e52be04d3" exitCode=0
May 25 11:19:01.654501 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:01.654087    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-clx9s" event={"ID":"a9377ad6-eac5-4df9-b7a2-b479c9c86277","Type":"ContainerDied","Data":"6b10e7c715738a0e361b7f1cf4eff3fa9fd6ff4ae718cdbc8026558e52be04d3"}
May 25 11:19:01.654570 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:01.654525    2561 scope.go:117] "RemoveContainer" containerID="6b10e7c715738a0e361b7f1cf4eff3fa9fd6ff4ae718cdbc8026558e52be04d3"
May 25 11:19:02.658357 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:02.658307    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-clx9s" event={"ID":"a9377ad6-eac5-4df9-b7a2-b479c9c86277","Type":"ContainerStarted","Data":"3d0bf0e81aebb6d538ca9e86c6bc282bce0fbf1e7244bd8036f9d4c81ecf6625"}
May 25 11:19:05.691922 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:05.691881    2561 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-image-registry/image-registry-7b9fd9b9cb-fpswn" podUID="21792ece-de04-4eb0-990c-e1adec1c79c5" containerName="registry" containerID="cri-o://e2d738c1e0b77b6c83930e0926a55dd4acbdb14207af212e2f336a48be1620e7" gracePeriod=30
May 25 11:19:05.932049 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:05.932027    2561 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7b9fd9b9cb-fpswn"
May 25 11:19:06.069254 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:06.069229    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/21792ece-de04-4eb0-990c-e1adec1c79c5-trusted-ca\") pod \"21792ece-de04-4eb0-990c-e1adec1c79c5\" (UID: \"21792ece-de04-4eb0-990c-e1adec1c79c5\") "
May 25 11:19:06.069417 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:06.069283    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/21792ece-de04-4eb0-990c-e1adec1c79c5-registry-tls\") pod \"21792ece-de04-4eb0-990c-e1adec1c79c5\" (UID: \"21792ece-de04-4eb0-990c-e1adec1c79c5\") "
May 25 11:19:06.069417 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:06.069323    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/21792ece-de04-4eb0-990c-e1adec1c79c5-ca-trust-extracted\") pod \"21792ece-de04-4eb0-990c-e1adec1c79c5\" (UID: \"21792ece-de04-4eb0-990c-e1adec1c79c5\") "
May 25 11:19:06.069417 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:06.069357    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/21792ece-de04-4eb0-990c-e1adec1c79c5-bound-sa-token\") pod \"21792ece-de04-4eb0-990c-e1adec1c79c5\" (UID: \"21792ece-de04-4eb0-990c-e1adec1c79c5\") "
May 25 11:19:06.069417 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:06.069384    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/21792ece-de04-4eb0-990c-e1adec1c79c5-image-registry-private-configuration\") pod \"21792ece-de04-4eb0-990c-e1adec1c79c5\" (UID: \"21792ece-de04-4eb0-990c-e1adec1c79c5\") "
May 25 11:19:06.069583 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:06.069423    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/21792ece-de04-4eb0-990c-e1adec1c79c5-registry-certificates\") pod \"21792ece-de04-4eb0-990c-e1adec1c79c5\" (UID: \"21792ece-de04-4eb0-990c-e1adec1c79c5\") "
May 25 11:19:06.069583 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:06.069466    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x5zjj\" (UniqueName: \"kubernetes.io/projected/21792ece-de04-4eb0-990c-e1adec1c79c5-kube-api-access-x5zjj\") pod \"21792ece-de04-4eb0-990c-e1adec1c79c5\" (UID: \"21792ece-de04-4eb0-990c-e1adec1c79c5\") "
May 25 11:19:06.069686 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:06.069624    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/21792ece-de04-4eb0-990c-e1adec1c79c5-installation-pull-secrets\") pod \"21792ece-de04-4eb0-990c-e1adec1c79c5\" (UID: \"21792ece-de04-4eb0-990c-e1adec1c79c5\") "
May 25 11:19:06.069736 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:06.069678    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21792ece-de04-4eb0-990c-e1adec1c79c5-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "21792ece-de04-4eb0-990c-e1adec1c79c5" (UID: "21792ece-de04-4eb0-990c-e1adec1c79c5"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 11:19:06.069987 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:06.069924    2561 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/21792ece-de04-4eb0-990c-e1adec1c79c5-trusted-ca\") on node \"ip-10-0-136-227.ec2.internal\" DevicePath \"\""
May 25 11:19:06.070160 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:06.070135    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21792ece-de04-4eb0-990c-e1adec1c79c5-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "21792ece-de04-4eb0-990c-e1adec1c79c5" (UID: "21792ece-de04-4eb0-990c-e1adec1c79c5"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 11:19:06.072376 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:06.072339    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21792ece-de04-4eb0-990c-e1adec1c79c5-kube-api-access-x5zjj" (OuterVolumeSpecName: "kube-api-access-x5zjj") pod "21792ece-de04-4eb0-990c-e1adec1c79c5" (UID: "21792ece-de04-4eb0-990c-e1adec1c79c5"). InnerVolumeSpecName "kube-api-access-x5zjj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 11:19:06.072376 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:06.072351    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21792ece-de04-4eb0-990c-e1adec1c79c5-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "21792ece-de04-4eb0-990c-e1adec1c79c5" (UID: "21792ece-de04-4eb0-990c-e1adec1c79c5"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 11:19:06.072376 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:06.072340    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21792ece-de04-4eb0-990c-e1adec1c79c5-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "21792ece-de04-4eb0-990c-e1adec1c79c5" (UID: "21792ece-de04-4eb0-990c-e1adec1c79c5"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 11:19:06.072573 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:06.072439    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21792ece-de04-4eb0-990c-e1adec1c79c5-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "21792ece-de04-4eb0-990c-e1adec1c79c5" (UID: "21792ece-de04-4eb0-990c-e1adec1c79c5"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 11:19:06.072573 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:06.072507    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21792ece-de04-4eb0-990c-e1adec1c79c5-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "21792ece-de04-4eb0-990c-e1adec1c79c5" (UID: "21792ece-de04-4eb0-990c-e1adec1c79c5"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 11:19:06.078013 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:06.077990    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21792ece-de04-4eb0-990c-e1adec1c79c5-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "21792ece-de04-4eb0-990c-e1adec1c79c5" (UID: "21792ece-de04-4eb0-990c-e1adec1c79c5"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:19:06.171287 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:06.171251    2561 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/21792ece-de04-4eb0-990c-e1adec1c79c5-bound-sa-token\") on node \"ip-10-0-136-227.ec2.internal\" DevicePath \"\""
May 25 11:19:06.171287 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:06.171290    2561 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/21792ece-de04-4eb0-990c-e1adec1c79c5-image-registry-private-configuration\") on node \"ip-10-0-136-227.ec2.internal\" DevicePath \"\""
May 25 11:19:06.171423 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:06.171300    2561 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/21792ece-de04-4eb0-990c-e1adec1c79c5-registry-certificates\") on node \"ip-10-0-136-227.ec2.internal\" DevicePath \"\""
May 25 11:19:06.171423 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:06.171311    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x5zjj\" (UniqueName: \"kubernetes.io/projected/21792ece-de04-4eb0-990c-e1adec1c79c5-kube-api-access-x5zjj\") on node \"ip-10-0-136-227.ec2.internal\" DevicePath \"\""
May 25 11:19:06.171423 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:06.171320    2561 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/21792ece-de04-4eb0-990c-e1adec1c79c5-installation-pull-secrets\") on node \"ip-10-0-136-227.ec2.internal\" DevicePath \"\""
May 25 11:19:06.171423 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:06.171329    2561 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/21792ece-de04-4eb0-990c-e1adec1c79c5-registry-tls\") on node \"ip-10-0-136-227.ec2.internal\" DevicePath \"\""
May 25 11:19:06.171423 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:06.171338    2561 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/21792ece-de04-4eb0-990c-e1adec1c79c5-ca-trust-extracted\") on node \"ip-10-0-136-227.ec2.internal\" DevicePath \"\""
May 25 11:19:06.671593 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:06.671558    2561 generic.go:358] "Generic (PLEG): container finished" podID="21792ece-de04-4eb0-990c-e1adec1c79c5" containerID="e2d738c1e0b77b6c83930e0926a55dd4acbdb14207af212e2f336a48be1620e7" exitCode=0
May 25 11:19:06.671806 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:06.671615    2561 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7b9fd9b9cb-fpswn"
May 25 11:19:06.671806 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:06.671646    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7b9fd9b9cb-fpswn" event={"ID":"21792ece-de04-4eb0-990c-e1adec1c79c5","Type":"ContainerDied","Data":"e2d738c1e0b77b6c83930e0926a55dd4acbdb14207af212e2f336a48be1620e7"}
May 25 11:19:06.671806 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:06.671683    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7b9fd9b9cb-fpswn" event={"ID":"21792ece-de04-4eb0-990c-e1adec1c79c5","Type":"ContainerDied","Data":"11e54098a27429630057f50ffaa3cd85effca9a1fc6c63b84c5383a833507f41"}
May 25 11:19:06.671806 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:06.671701    2561 scope.go:117] "RemoveContainer" containerID="e2d738c1e0b77b6c83930e0926a55dd4acbdb14207af212e2f336a48be1620e7"
May 25 11:19:06.679878 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:06.679861    2561 scope.go:117] "RemoveContainer" containerID="e2d738c1e0b77b6c83930e0926a55dd4acbdb14207af212e2f336a48be1620e7"
May 25 11:19:06.680122 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:19:06.680103    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2d738c1e0b77b6c83930e0926a55dd4acbdb14207af212e2f336a48be1620e7\": container with ID starting with e2d738c1e0b77b6c83930e0926a55dd4acbdb14207af212e2f336a48be1620e7 not found: ID does not exist" containerID="e2d738c1e0b77b6c83930e0926a55dd4acbdb14207af212e2f336a48be1620e7"
May 25 11:19:06.680175 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:06.680130    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2d738c1e0b77b6c83930e0926a55dd4acbdb14207af212e2f336a48be1620e7"} err="failed to get container status \"e2d738c1e0b77b6c83930e0926a55dd4acbdb14207af212e2f336a48be1620e7\": rpc error: code = NotFound desc = could not find container \"e2d738c1e0b77b6c83930e0926a55dd4acbdb14207af212e2f336a48be1620e7\": container with ID starting with e2d738c1e0b77b6c83930e0926a55dd4acbdb14207af212e2f336a48be1620e7 not found: ID does not exist"
May 25 11:19:06.693245 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:06.693225    2561 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7b9fd9b9cb-fpswn"]
May 25 11:19:06.701940 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:06.701921    2561 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-7b9fd9b9cb-fpswn"]
May 25 11:19:07.117355 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:07.117326    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="21792ece-de04-4eb0-990c-e1adec1c79c5" path="/var/lib/kubelet/pods/21792ece-de04-4eb0-990c-e1adec1c79c5/volumes"
May 25 11:19:16.701471 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:16.701442    2561 generic.go:358] "Generic (PLEG): container finished" podID="13894946-8385-4b6f-a82a-4188f46d576c" containerID="7c70edf00194ca405e70f794cf546ab437cf0174098cce7cdd9ee77cc031db66" exitCode=0
May 25 11:19:16.701837 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:16.701515    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-78bdfb888c-nd5f5" event={"ID":"13894946-8385-4b6f-a82a-4188f46d576c","Type":"ContainerDied","Data":"7c70edf00194ca405e70f794cf546ab437cf0174098cce7cdd9ee77cc031db66"}
May 25 11:19:16.701935 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:16.701918    2561 scope.go:117] "RemoveContainer" containerID="7c70edf00194ca405e70f794cf546ab437cf0174098cce7cdd9ee77cc031db66"
May 25 11:19:16.702829 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:16.702810    2561 generic.go:358] "Generic (PLEG): container finished" podID="9b9cd68b-6d33-40ae-b547-43f4ef7a2f5c" containerID="0b177d42e419ed65bec60d24bb23ebe6798a9b55ef59d1f9735337811bcbc530" exitCode=0
May 25 11:19:16.702911 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:16.702846    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-88l9x" event={"ID":"9b9cd68b-6d33-40ae-b547-43f4ef7a2f5c","Type":"ContainerDied","Data":"0b177d42e419ed65bec60d24bb23ebe6798a9b55ef59d1f9735337811bcbc530"}
May 25 11:19:16.703128 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:16.703115    2561 scope.go:117] "RemoveContainer" containerID="0b177d42e419ed65bec60d24bb23ebe6798a9b55ef59d1f9735337811bcbc530"
May 25 11:19:17.704454 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:17.704393    2561 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-console/console-56544555f7-lczsn" podUID="12498c7a-38d3-4424-919e-0e430256aa0c" containerName="console" containerID="cri-o://10d97418541642d142f78c765d0b08dc1de23422396366fe952edd5699e99701" gracePeriod=15
May 25 11:19:17.707008 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:17.706973    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-78bdfb888c-nd5f5" event={"ID":"13894946-8385-4b6f-a82a-4188f46d576c","Type":"ContainerStarted","Data":"41e339fce7a3dd4558cc397c44ad5bf9e5b6ebd4a2e6f7720d0d11cba7ebdf67"}
May 25 11:19:17.708684 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:17.708659    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-88l9x" event={"ID":"9b9cd68b-6d33-40ae-b547-43f4ef7a2f5c","Type":"ContainerStarted","Data":"7c694894b03c3c8df2e82a34950a4406c969410fa616beee0a1afd31a1e3c478"}
May 25 11:19:17.945089 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:17.945069    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-56544555f7-lczsn_12498c7a-38d3-4424-919e-0e430256aa0c/console/0.log"
May 25 11:19:17.945199 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:17.945126    2561 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-56544555f7-lczsn"
May 25 11:19:18.068470 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:18.068441    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/12498c7a-38d3-4424-919e-0e430256aa0c-console-config\") pod \"12498c7a-38d3-4424-919e-0e430256aa0c\" (UID: \"12498c7a-38d3-4424-919e-0e430256aa0c\") "
May 25 11:19:18.068634 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:18.068483    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/12498c7a-38d3-4424-919e-0e430256aa0c-oauth-serving-cert\") pod \"12498c7a-38d3-4424-919e-0e430256aa0c\" (UID: \"12498c7a-38d3-4424-919e-0e430256aa0c\") "
May 25 11:19:18.068698 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:18.068669    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/12498c7a-38d3-4424-919e-0e430256aa0c-console-oauth-config\") pod \"12498c7a-38d3-4424-919e-0e430256aa0c\" (UID: \"12498c7a-38d3-4424-919e-0e430256aa0c\") "
May 25 11:19:18.068756 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:18.068707    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/12498c7a-38d3-4424-919e-0e430256aa0c-service-ca\") pod \"12498c7a-38d3-4424-919e-0e430256aa0c\" (UID: \"12498c7a-38d3-4424-919e-0e430256aa0c\") "
May 25 11:19:18.068756 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:18.068727    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/12498c7a-38d3-4424-919e-0e430256aa0c-console-serving-cert\") pod \"12498c7a-38d3-4424-919e-0e430256aa0c\" (UID: \"12498c7a-38d3-4424-919e-0e430256aa0c\") "
May 25 11:19:18.068856 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:18.068762    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qxwql\" (UniqueName: \"kubernetes.io/projected/12498c7a-38d3-4424-919e-0e430256aa0c-kube-api-access-qxwql\") pod \"12498c7a-38d3-4424-919e-0e430256aa0c\" (UID: \"12498c7a-38d3-4424-919e-0e430256aa0c\") "
May 25 11:19:18.068856 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:18.068830    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12498c7a-38d3-4424-919e-0e430256aa0c-console-config" (OuterVolumeSpecName: "console-config") pod "12498c7a-38d3-4424-919e-0e430256aa0c" (UID: "12498c7a-38d3-4424-919e-0e430256aa0c"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 11:19:18.068957 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:18.068867    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12498c7a-38d3-4424-919e-0e430256aa0c-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "12498c7a-38d3-4424-919e-0e430256aa0c" (UID: "12498c7a-38d3-4424-919e-0e430256aa0c"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 11:19:18.069122 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:18.069096    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12498c7a-38d3-4424-919e-0e430256aa0c-service-ca" (OuterVolumeSpecName: "service-ca") pod "12498c7a-38d3-4424-919e-0e430256aa0c" (UID: "12498c7a-38d3-4424-919e-0e430256aa0c"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 11:19:18.069194 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:18.069126    2561 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/12498c7a-38d3-4424-919e-0e430256aa0c-console-config\") on node \"ip-10-0-136-227.ec2.internal\" DevicePath \"\""
May 25 11:19:18.069194 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:18.069149    2561 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/12498c7a-38d3-4424-919e-0e430256aa0c-oauth-serving-cert\") on node \"ip-10-0-136-227.ec2.internal\" DevicePath \"\""
May 25 11:19:18.070915 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:18.070889    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12498c7a-38d3-4424-919e-0e430256aa0c-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "12498c7a-38d3-4424-919e-0e430256aa0c" (UID: "12498c7a-38d3-4424-919e-0e430256aa0c"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 11:19:18.070999 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:18.070928    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12498c7a-38d3-4424-919e-0e430256aa0c-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "12498c7a-38d3-4424-919e-0e430256aa0c" (UID: "12498c7a-38d3-4424-919e-0e430256aa0c"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 11:19:18.070999 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:18.070938    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12498c7a-38d3-4424-919e-0e430256aa0c-kube-api-access-qxwql" (OuterVolumeSpecName: "kube-api-access-qxwql") pod "12498c7a-38d3-4424-919e-0e430256aa0c" (UID: "12498c7a-38d3-4424-919e-0e430256aa0c"). InnerVolumeSpecName "kube-api-access-qxwql". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 11:19:18.170541 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:18.170510    2561 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/12498c7a-38d3-4424-919e-0e430256aa0c-console-oauth-config\") on node \"ip-10-0-136-227.ec2.internal\" DevicePath \"\""
May 25 11:19:18.170541 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:18.170540    2561 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/12498c7a-38d3-4424-919e-0e430256aa0c-service-ca\") on node \"ip-10-0-136-227.ec2.internal\" DevicePath \"\""
May 25 11:19:18.170688 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:18.170556    2561 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/12498c7a-38d3-4424-919e-0e430256aa0c-console-serving-cert\") on node \"ip-10-0-136-227.ec2.internal\" DevicePath \"\""
May 25 11:19:18.170688 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:18.170570    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qxwql\" (UniqueName: \"kubernetes.io/projected/12498c7a-38d3-4424-919e-0e430256aa0c-kube-api-access-qxwql\") on node \"ip-10-0-136-227.ec2.internal\" DevicePath \"\""
May 25 11:19:18.712726 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:18.712693    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-56544555f7-lczsn_12498c7a-38d3-4424-919e-0e430256aa0c/console/0.log"
May 25 11:19:18.713163 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:18.712738    2561 generic.go:358] "Generic (PLEG): container finished" podID="12498c7a-38d3-4424-919e-0e430256aa0c" containerID="10d97418541642d142f78c765d0b08dc1de23422396366fe952edd5699e99701" exitCode=2
May 25 11:19:18.713163 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:18.712780    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-56544555f7-lczsn" event={"ID":"12498c7a-38d3-4424-919e-0e430256aa0c","Type":"ContainerDied","Data":"10d97418541642d142f78c765d0b08dc1de23422396366fe952edd5699e99701"}
May 25 11:19:18.713163 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:18.712808    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-56544555f7-lczsn" event={"ID":"12498c7a-38d3-4424-919e-0e430256aa0c","Type":"ContainerDied","Data":"b5abac5deef213ba536a04b069287e5db4fb5441206154f24060990c8a83a69f"}
May 25 11:19:18.713163 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:18.712816    2561 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-56544555f7-lczsn"
May 25 11:19:18.713163 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:18.712829    2561 scope.go:117] "RemoveContainer" containerID="10d97418541642d142f78c765d0b08dc1de23422396366fe952edd5699e99701"
May 25 11:19:18.721399 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:18.721379    2561 scope.go:117] "RemoveContainer" containerID="10d97418541642d142f78c765d0b08dc1de23422396366fe952edd5699e99701"
May 25 11:19:18.721648 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:19:18.721626    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10d97418541642d142f78c765d0b08dc1de23422396366fe952edd5699e99701\": container with ID starting with 10d97418541642d142f78c765d0b08dc1de23422396366fe952edd5699e99701 not found: ID does not exist" containerID="10d97418541642d142f78c765d0b08dc1de23422396366fe952edd5699e99701"
May 25 11:19:18.721701 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:18.721658    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10d97418541642d142f78c765d0b08dc1de23422396366fe952edd5699e99701"} err="failed to get container status \"10d97418541642d142f78c765d0b08dc1de23422396366fe952edd5699e99701\": rpc error: code = NotFound desc = could not find container \"10d97418541642d142f78c765d0b08dc1de23422396366fe952edd5699e99701\": container with ID starting with 10d97418541642d142f78c765d0b08dc1de23422396366fe952edd5699e99701 not found: ID does not exist"
May 25 11:19:18.734412 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:18.734384    2561 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-56544555f7-lczsn"]
May 25 11:19:18.740614 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:18.740595    2561 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-56544555f7-lczsn"]
May 25 11:19:19.116247 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:19.116208    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="12498c7a-38d3-4424-919e-0e430256aa0c" path="/var/lib/kubelet/pods/12498c7a-38d3-4424-919e-0e430256aa0c/volumes"
May 25 11:19:52.407634 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:52.407601    2561 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
May 25 11:19:52.408075 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:52.408012    2561 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="5805521a-51c5-4810-9cf6-ba306bfb50d9" containerName="alertmanager" containerID="cri-o://1f4635273454350be029575dd68c8b83dc653de3ddac1cc66544bfc46af2023b" gracePeriod=120
May 25 11:19:52.408127 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:52.408079    2561 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="5805521a-51c5-4810-9cf6-ba306bfb50d9" containerName="kube-rbac-proxy-metric" containerID="cri-o://3d1dbdc6eed315467e25094ce0fc197b9c952d54bef5721ee7849bd090e96a80" gracePeriod=120
May 25 11:19:52.408180 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:52.408139    2561 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="5805521a-51c5-4810-9cf6-ba306bfb50d9" containerName="config-reloader" containerID="cri-o://22b12500d9d4a100d1ec3b2ad330660b597d416a172c59244eb5a9a819068c8c" gracePeriod=120
May 25 11:19:52.408180 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:52.408115    2561 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="5805521a-51c5-4810-9cf6-ba306bfb50d9" containerName="kube-rbac-proxy-web" containerID="cri-o://73c9ecb470e2b22eac14037ba0972ae6efa213c1e92c0afa10ae424d5715160d" gracePeriod=120
May 25 11:19:52.408307 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:52.408134    2561 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="5805521a-51c5-4810-9cf6-ba306bfb50d9" containerName="kube-rbac-proxy" containerID="cri-o://cd02c74d52650e3e7e20d32f9f80033352f5ccbaaa7b7474a50b996eeaa1d4fe" gracePeriod=120
May 25 11:19:52.408307 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:52.408179    2561 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="5805521a-51c5-4810-9cf6-ba306bfb50d9" containerName="prom-label-proxy" containerID="cri-o://c595ca6d95d1e83b483036990a03d60a9e0e3951f796199cb98404528b1c8e03" gracePeriod=120
May 25 11:19:52.822475 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:52.822442    2561 generic.go:358] "Generic (PLEG): container finished" podID="5805521a-51c5-4810-9cf6-ba306bfb50d9" containerID="c595ca6d95d1e83b483036990a03d60a9e0e3951f796199cb98404528b1c8e03" exitCode=0
May 25 11:19:52.822475 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:52.822468    2561 generic.go:358] "Generic (PLEG): container finished" podID="5805521a-51c5-4810-9cf6-ba306bfb50d9" containerID="cd02c74d52650e3e7e20d32f9f80033352f5ccbaaa7b7474a50b996eeaa1d4fe" exitCode=0
May 25 11:19:52.822475 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:52.822475    2561 generic.go:358] "Generic (PLEG): container finished" podID="5805521a-51c5-4810-9cf6-ba306bfb50d9" containerID="22b12500d9d4a100d1ec3b2ad330660b597d416a172c59244eb5a9a819068c8c" exitCode=0
May 25 11:19:52.822475 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:52.822480    2561 generic.go:358] "Generic (PLEG): container finished" podID="5805521a-51c5-4810-9cf6-ba306bfb50d9" containerID="1f4635273454350be029575dd68c8b83dc653de3ddac1cc66544bfc46af2023b" exitCode=0
May 25 11:19:52.822722 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:52.822510    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5805521a-51c5-4810-9cf6-ba306bfb50d9","Type":"ContainerDied","Data":"c595ca6d95d1e83b483036990a03d60a9e0e3951f796199cb98404528b1c8e03"}
May 25 11:19:52.822722 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:52.822545    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5805521a-51c5-4810-9cf6-ba306bfb50d9","Type":"ContainerDied","Data":"cd02c74d52650e3e7e20d32f9f80033352f5ccbaaa7b7474a50b996eeaa1d4fe"}
May 25 11:19:52.822722 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:52.822556    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5805521a-51c5-4810-9cf6-ba306bfb50d9","Type":"ContainerDied","Data":"22b12500d9d4a100d1ec3b2ad330660b597d416a172c59244eb5a9a819068c8c"}
May 25 11:19:52.822722 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:52.822566    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5805521a-51c5-4810-9cf6-ba306bfb50d9","Type":"ContainerDied","Data":"1f4635273454350be029575dd68c8b83dc653de3ddac1cc66544bfc46af2023b"}
May 25 11:19:53.647553 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:53.647533    2561 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
May 25 11:19:53.739763 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:53.739702    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5805521a-51c5-4810-9cf6-ba306bfb50d9-alertmanager-main-db\") pod \"5805521a-51c5-4810-9cf6-ba306bfb50d9\" (UID: \"5805521a-51c5-4810-9cf6-ba306bfb50d9\") "
May 25 11:19:53.739763 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:53.739739    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5805521a-51c5-4810-9cf6-ba306bfb50d9-secret-alertmanager-kube-rbac-proxy-metric\") pod \"5805521a-51c5-4810-9cf6-ba306bfb50d9\" (UID: \"5805521a-51c5-4810-9cf6-ba306bfb50d9\") "
May 25 11:19:53.739763 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:53.739760    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5805521a-51c5-4810-9cf6-ba306bfb50d9-secret-alertmanager-main-tls\") pod \"5805521a-51c5-4810-9cf6-ba306bfb50d9\" (UID: \"5805521a-51c5-4810-9cf6-ba306bfb50d9\") "
May 25 11:19:53.740007 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:53.739785    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5805521a-51c5-4810-9cf6-ba306bfb50d9-secret-alertmanager-kube-rbac-proxy-web\") pod \"5805521a-51c5-4810-9cf6-ba306bfb50d9\" (UID: \"5805521a-51c5-4810-9cf6-ba306bfb50d9\") "
May 25 11:19:53.740007 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:53.739807    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5805521a-51c5-4810-9cf6-ba306bfb50d9-tls-assets\") pod \"5805521a-51c5-4810-9cf6-ba306bfb50d9\" (UID: \"5805521a-51c5-4810-9cf6-ba306bfb50d9\") "
May 25 11:19:53.740007 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:53.739917    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5805521a-51c5-4810-9cf6-ba306bfb50d9-cluster-tls-config\") pod \"5805521a-51c5-4810-9cf6-ba306bfb50d9\" (UID: \"5805521a-51c5-4810-9cf6-ba306bfb50d9\") "
May 25 11:19:53.740336 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:53.740288    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5805521a-51c5-4810-9cf6-ba306bfb50d9-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "5805521a-51c5-4810-9cf6-ba306bfb50d9" (UID: "5805521a-51c5-4810-9cf6-ba306bfb50d9"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:19:53.740461 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:53.739967    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5805521a-51c5-4810-9cf6-ba306bfb50d9-secret-alertmanager-kube-rbac-proxy\") pod \"5805521a-51c5-4810-9cf6-ba306bfb50d9\" (UID: \"5805521a-51c5-4810-9cf6-ba306bfb50d9\") "
May 25 11:19:53.740534 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:53.740460    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5805521a-51c5-4810-9cf6-ba306bfb50d9-config-out\") pod \"5805521a-51c5-4810-9cf6-ba306bfb50d9\" (UID: \"5805521a-51c5-4810-9cf6-ba306bfb50d9\") "
May 25 11:19:53.740534 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:53.740512    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5805521a-51c5-4810-9cf6-ba306bfb50d9-metrics-client-ca\") pod \"5805521a-51c5-4810-9cf6-ba306bfb50d9\" (UID: \"5805521a-51c5-4810-9cf6-ba306bfb50d9\") "
May 25 11:19:53.740637 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:53.740553    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rp6fg\" (UniqueName: \"kubernetes.io/projected/5805521a-51c5-4810-9cf6-ba306bfb50d9-kube-api-access-rp6fg\") pod \"5805521a-51c5-4810-9cf6-ba306bfb50d9\" (UID: \"5805521a-51c5-4810-9cf6-ba306bfb50d9\") "
May 25 11:19:53.740637 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:53.740592    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5805521a-51c5-4810-9cf6-ba306bfb50d9-web-config\") pod \"5805521a-51c5-4810-9cf6-ba306bfb50d9\" (UID: \"5805521a-51c5-4810-9cf6-ba306bfb50d9\") "
May 25 11:19:53.740733 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:53.740634    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5805521a-51c5-4810-9cf6-ba306bfb50d9-config-volume\") pod \"5805521a-51c5-4810-9cf6-ba306bfb50d9\" (UID: \"5805521a-51c5-4810-9cf6-ba306bfb50d9\") "
May 25 11:19:53.740733 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:53.740671    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5805521a-51c5-4810-9cf6-ba306bfb50d9-alertmanager-trusted-ca-bundle\") pod \"5805521a-51c5-4810-9cf6-ba306bfb50d9\" (UID: \"5805521a-51c5-4810-9cf6-ba306bfb50d9\") "
May 25 11:19:53.741767 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:53.741298    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5805521a-51c5-4810-9cf6-ba306bfb50d9-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "5805521a-51c5-4810-9cf6-ba306bfb50d9" (UID: "5805521a-51c5-4810-9cf6-ba306bfb50d9"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 11:19:53.742850 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:53.742366    2561 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5805521a-51c5-4810-9cf6-ba306bfb50d9-metrics-client-ca\") on node \"ip-10-0-136-227.ec2.internal\" DevicePath \"\""
May 25 11:19:53.742850 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:53.742397    2561 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5805521a-51c5-4810-9cf6-ba306bfb50d9-alertmanager-main-db\") on node \"ip-10-0-136-227.ec2.internal\" DevicePath \"\""
May 25 11:19:53.743162 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:53.743135    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5805521a-51c5-4810-9cf6-ba306bfb50d9-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "5805521a-51c5-4810-9cf6-ba306bfb50d9" (UID: "5805521a-51c5-4810-9cf6-ba306bfb50d9"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 11:19:53.743516 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:53.743407    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5805521a-51c5-4810-9cf6-ba306bfb50d9-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "5805521a-51c5-4810-9cf6-ba306bfb50d9" (UID: "5805521a-51c5-4810-9cf6-ba306bfb50d9"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 11:19:53.743516 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:53.743458    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5805521a-51c5-4810-9cf6-ba306bfb50d9-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "5805521a-51c5-4810-9cf6-ba306bfb50d9" (UID: "5805521a-51c5-4810-9cf6-ba306bfb50d9"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 11:19:53.744978 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:53.744950    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5805521a-51c5-4810-9cf6-ba306bfb50d9-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "5805521a-51c5-4810-9cf6-ba306bfb50d9" (UID: "5805521a-51c5-4810-9cf6-ba306bfb50d9"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 11:19:53.749081 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:53.748989    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5805521a-51c5-4810-9cf6-ba306bfb50d9-config-out" (OuterVolumeSpecName: "config-out") pod "5805521a-51c5-4810-9cf6-ba306bfb50d9" (UID: "5805521a-51c5-4810-9cf6-ba306bfb50d9"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:19:53.749540 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:53.749514    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5805521a-51c5-4810-9cf6-ba306bfb50d9-kube-api-access-rp6fg" (OuterVolumeSpecName: "kube-api-access-rp6fg") pod "5805521a-51c5-4810-9cf6-ba306bfb50d9" (UID: "5805521a-51c5-4810-9cf6-ba306bfb50d9"). InnerVolumeSpecName "kube-api-access-rp6fg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 11:19:53.749690 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:53.749664    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5805521a-51c5-4810-9cf6-ba306bfb50d9-config-volume" (OuterVolumeSpecName: "config-volume") pod "5805521a-51c5-4810-9cf6-ba306bfb50d9" (UID: "5805521a-51c5-4810-9cf6-ba306bfb50d9"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 11:19:53.750450 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:53.750375    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5805521a-51c5-4810-9cf6-ba306bfb50d9-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "5805521a-51c5-4810-9cf6-ba306bfb50d9" (UID: "5805521a-51c5-4810-9cf6-ba306bfb50d9"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 11:19:53.751678 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:53.751644    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5805521a-51c5-4810-9cf6-ba306bfb50d9-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "5805521a-51c5-4810-9cf6-ba306bfb50d9" (UID: "5805521a-51c5-4810-9cf6-ba306bfb50d9"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 11:19:53.753154 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:53.753133    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5805521a-51c5-4810-9cf6-ba306bfb50d9-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "5805521a-51c5-4810-9cf6-ba306bfb50d9" (UID: "5805521a-51c5-4810-9cf6-ba306bfb50d9"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 11:19:53.758524 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:53.758502    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5805521a-51c5-4810-9cf6-ba306bfb50d9-web-config" (OuterVolumeSpecName: "web-config") pod "5805521a-51c5-4810-9cf6-ba306bfb50d9" (UID: "5805521a-51c5-4810-9cf6-ba306bfb50d9"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 11:19:53.828194 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:53.828166    2561 generic.go:358] "Generic (PLEG): container finished" podID="5805521a-51c5-4810-9cf6-ba306bfb50d9" containerID="3d1dbdc6eed315467e25094ce0fc197b9c952d54bef5721ee7849bd090e96a80" exitCode=0
May 25 11:19:53.828194 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:53.828191    2561 generic.go:358] "Generic (PLEG): container finished" podID="5805521a-51c5-4810-9cf6-ba306bfb50d9" containerID="73c9ecb470e2b22eac14037ba0972ae6efa213c1e92c0afa10ae424d5715160d" exitCode=0
May 25 11:19:53.828331 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:53.828228    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5805521a-51c5-4810-9cf6-ba306bfb50d9","Type":"ContainerDied","Data":"3d1dbdc6eed315467e25094ce0fc197b9c952d54bef5721ee7849bd090e96a80"}
May 25 11:19:53.828331 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:53.828261    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5805521a-51c5-4810-9cf6-ba306bfb50d9","Type":"ContainerDied","Data":"73c9ecb470e2b22eac14037ba0972ae6efa213c1e92c0afa10ae424d5715160d"}
May 25 11:19:53.828331 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:53.828288    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5805521a-51c5-4810-9cf6-ba306bfb50d9","Type":"ContainerDied","Data":"0a7ed63ea4a5c20053794ffb3ef3dc12fbb037022771a7ab7b035d142ac730c3"}
May 25 11:19:53.828331 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:53.828291    2561 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
May 25 11:19:53.828331 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:53.828302    2561 scope.go:117] "RemoveContainer" containerID="c595ca6d95d1e83b483036990a03d60a9e0e3951f796199cb98404528b1c8e03"
May 25 11:19:53.835736 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:53.835652    2561 scope.go:117] "RemoveContainer" containerID="3d1dbdc6eed315467e25094ce0fc197b9c952d54bef5721ee7849bd090e96a80"
May 25 11:19:53.842703 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:53.842687    2561 scope.go:117] "RemoveContainer" containerID="cd02c74d52650e3e7e20d32f9f80033352f5ccbaaa7b7474a50b996eeaa1d4fe"
May 25 11:19:53.842942 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:53.842926    2561 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5805521a-51c5-4810-9cf6-ba306bfb50d9-config-volume\") on node \"ip-10-0-136-227.ec2.internal\" DevicePath \"\""
May 25 11:19:53.843000 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:53.842946    2561 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5805521a-51c5-4810-9cf6-ba306bfb50d9-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-136-227.ec2.internal\" DevicePath \"\""
May 25 11:19:53.843000 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:53.842962    2561 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5805521a-51c5-4810-9cf6-ba306bfb50d9-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-136-227.ec2.internal\" DevicePath \"\""
May 25 11:19:53.843000 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:53.842974    2561 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5805521a-51c5-4810-9cf6-ba306bfb50d9-secret-alertmanager-main-tls\") on node \"ip-10-0-136-227.ec2.internal\" DevicePath \"\""
May 25 11:19:53.843000 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:53.842983    2561 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5805521a-51c5-4810-9cf6-ba306bfb50d9-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-136-227.ec2.internal\" DevicePath \"\""
May 25 11:19:53.843000 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:53.842992    2561 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5805521a-51c5-4810-9cf6-ba306bfb50d9-tls-assets\") on node \"ip-10-0-136-227.ec2.internal\" DevicePath \"\""
May 25 11:19:53.843000 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:53.843000    2561 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5805521a-51c5-4810-9cf6-ba306bfb50d9-cluster-tls-config\") on node \"ip-10-0-136-227.ec2.internal\" DevicePath \"\""
May 25 11:19:53.843241 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:53.843009    2561 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5805521a-51c5-4810-9cf6-ba306bfb50d9-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-136-227.ec2.internal\" DevicePath \"\""
May 25 11:19:53.843241 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:53.843019    2561 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5805521a-51c5-4810-9cf6-ba306bfb50d9-config-out\") on node \"ip-10-0-136-227.ec2.internal\" DevicePath \"\""
May 25 11:19:53.843241 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:53.843027    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rp6fg\" (UniqueName: \"kubernetes.io/projected/5805521a-51c5-4810-9cf6-ba306bfb50d9-kube-api-access-rp6fg\") on node \"ip-10-0-136-227.ec2.internal\" DevicePath \"\""
May 25 11:19:53.843241 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:53.843035    2561 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5805521a-51c5-4810-9cf6-ba306bfb50d9-web-config\") on node \"ip-10-0-136-227.ec2.internal\" DevicePath \"\""
May 25 11:19:53.849165 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:53.849151    2561 scope.go:117] "RemoveContainer" containerID="73c9ecb470e2b22eac14037ba0972ae6efa213c1e92c0afa10ae424d5715160d"
May 25 11:19:53.851751 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:53.851719    2561 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
May 25 11:19:53.856472 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:53.856411    2561 scope.go:117] "RemoveContainer" containerID="22b12500d9d4a100d1ec3b2ad330660b597d416a172c59244eb5a9a819068c8c"
May 25 11:19:53.858460 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:53.858438    2561 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
May 25 11:19:53.862547 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:53.862531    2561 scope.go:117] "RemoveContainer" containerID="1f4635273454350be029575dd68c8b83dc653de3ddac1cc66544bfc46af2023b"
May 25 11:19:53.868574 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:53.868558    2561 scope.go:117] "RemoveContainer" containerID="77fb2740ebe94c49986d18fe57931a73e6da46f8bbf9f6166aa62b6e1b70ab28"
May 25 11:19:53.874320 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:53.874304    2561 scope.go:117] "RemoveContainer" containerID="c595ca6d95d1e83b483036990a03d60a9e0e3951f796199cb98404528b1c8e03"
May 25 11:19:53.874544 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:19:53.874526    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c595ca6d95d1e83b483036990a03d60a9e0e3951f796199cb98404528b1c8e03\": container with ID starting with c595ca6d95d1e83b483036990a03d60a9e0e3951f796199cb98404528b1c8e03 not found: ID does not exist" containerID="c595ca6d95d1e83b483036990a03d60a9e0e3951f796199cb98404528b1c8e03"
May 25 11:19:53.874591 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:53.874550    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c595ca6d95d1e83b483036990a03d60a9e0e3951f796199cb98404528b1c8e03"} err="failed to get container status \"c595ca6d95d1e83b483036990a03d60a9e0e3951f796199cb98404528b1c8e03\": rpc error: code = NotFound desc = could not find container \"c595ca6d95d1e83b483036990a03d60a9e0e3951f796199cb98404528b1c8e03\": container with ID starting with c595ca6d95d1e83b483036990a03d60a9e0e3951f796199cb98404528b1c8e03 not found: ID does not exist"
May 25 11:19:53.874591 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:53.874566    2561 scope.go:117] "RemoveContainer" containerID="3d1dbdc6eed315467e25094ce0fc197b9c952d54bef5721ee7849bd090e96a80"
May 25 11:19:53.874802 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:19:53.874771    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d1dbdc6eed315467e25094ce0fc197b9c952d54bef5721ee7849bd090e96a80\": container with ID starting with 3d1dbdc6eed315467e25094ce0fc197b9c952d54bef5721ee7849bd090e96a80 not found: ID does not exist" containerID="3d1dbdc6eed315467e25094ce0fc197b9c952d54bef5721ee7849bd090e96a80"
May 25 11:19:53.874884 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:53.874809    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d1dbdc6eed315467e25094ce0fc197b9c952d54bef5721ee7849bd090e96a80"} err="failed to get container status \"3d1dbdc6eed315467e25094ce0fc197b9c952d54bef5721ee7849bd090e96a80\": rpc error: code = NotFound desc = could not find container \"3d1dbdc6eed315467e25094ce0fc197b9c952d54bef5721ee7849bd090e96a80\": container with ID starting with 3d1dbdc6eed315467e25094ce0fc197b9c952d54bef5721ee7849bd090e96a80 not found: ID does not exist"
May 25 11:19:53.874884 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:53.874831    2561 scope.go:117] "RemoveContainer" containerID="cd02c74d52650e3e7e20d32f9f80033352f5ccbaaa7b7474a50b996eeaa1d4fe"
May 25 11:19:53.875243 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:19:53.875109    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd02c74d52650e3e7e20d32f9f80033352f5ccbaaa7b7474a50b996eeaa1d4fe\": container with ID starting with cd02c74d52650e3e7e20d32f9f80033352f5ccbaaa7b7474a50b996eeaa1d4fe not found: ID does not exist" containerID="cd02c74d52650e3e7e20d32f9f80033352f5ccbaaa7b7474a50b996eeaa1d4fe"
May 25 11:19:53.875243 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:53.875139    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd02c74d52650e3e7e20d32f9f80033352f5ccbaaa7b7474a50b996eeaa1d4fe"} err="failed to get container status \"cd02c74d52650e3e7e20d32f9f80033352f5ccbaaa7b7474a50b996eeaa1d4fe\": rpc error: code = NotFound desc = could not find container \"cd02c74d52650e3e7e20d32f9f80033352f5ccbaaa7b7474a50b996eeaa1d4fe\": container with ID starting with cd02c74d52650e3e7e20d32f9f80033352f5ccbaaa7b7474a50b996eeaa1d4fe not found: ID does not exist"
May 25 11:19:53.875243 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:53.875157    2561 scope.go:117] "RemoveContainer" containerID="73c9ecb470e2b22eac14037ba0972ae6efa213c1e92c0afa10ae424d5715160d"
May 25 11:19:53.875463 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:19:53.875411    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73c9ecb470e2b22eac14037ba0972ae6efa213c1e92c0afa10ae424d5715160d\": container with ID starting with 73c9ecb470e2b22eac14037ba0972ae6efa213c1e92c0afa10ae424d5715160d not found: ID does not exist" containerID="73c9ecb470e2b22eac14037ba0972ae6efa213c1e92c0afa10ae424d5715160d"
May 25 11:19:53.875463 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:53.875432    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73c9ecb470e2b22eac14037ba0972ae6efa213c1e92c0afa10ae424d5715160d"} err="failed to get container status \"73c9ecb470e2b22eac14037ba0972ae6efa213c1e92c0afa10ae424d5715160d\": rpc error: code = NotFound desc = could not find container \"73c9ecb470e2b22eac14037ba0972ae6efa213c1e92c0afa10ae424d5715160d\": container with ID starting with 73c9ecb470e2b22eac14037ba0972ae6efa213c1e92c0afa10ae424d5715160d not found: ID does not exist"
May 25 11:19:53.875463 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:53.875454    2561 scope.go:117] "RemoveContainer" containerID="22b12500d9d4a100d1ec3b2ad330660b597d416a172c59244eb5a9a819068c8c"
May 25 11:19:53.875676 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:19:53.875660    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22b12500d9d4a100d1ec3b2ad330660b597d416a172c59244eb5a9a819068c8c\": container with ID starting with 22b12500d9d4a100d1ec3b2ad330660b597d416a172c59244eb5a9a819068c8c not found: ID does not exist" containerID="22b12500d9d4a100d1ec3b2ad330660b597d416a172c59244eb5a9a819068c8c"
May 25 11:19:53.875714 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:53.875689    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22b12500d9d4a100d1ec3b2ad330660b597d416a172c59244eb5a9a819068c8c"} err="failed to get container status \"22b12500d9d4a100d1ec3b2ad330660b597d416a172c59244eb5a9a819068c8c\": rpc error: code = NotFound desc = could not find container \"22b12500d9d4a100d1ec3b2ad330660b597d416a172c59244eb5a9a819068c8c\": container with ID starting with 22b12500d9d4a100d1ec3b2ad330660b597d416a172c59244eb5a9a819068c8c not found: ID does not exist"
May 25 11:19:53.875714 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:53.875702    2561 scope.go:117] "RemoveContainer" containerID="1f4635273454350be029575dd68c8b83dc653de3ddac1cc66544bfc46af2023b"
May 25 11:19:53.875916 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:19:53.875898    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f4635273454350be029575dd68c8b83dc653de3ddac1cc66544bfc46af2023b\": container with ID starting with 1f4635273454350be029575dd68c8b83dc653de3ddac1cc66544bfc46af2023b not found: ID does not exist" containerID="1f4635273454350be029575dd68c8b83dc653de3ddac1cc66544bfc46af2023b"
May 25 11:19:53.875950 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:53.875922    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f4635273454350be029575dd68c8b83dc653de3ddac1cc66544bfc46af2023b"} err="failed to get container status \"1f4635273454350be029575dd68c8b83dc653de3ddac1cc66544bfc46af2023b\": rpc error: code = NotFound desc = could not find container \"1f4635273454350be029575dd68c8b83dc653de3ddac1cc66544bfc46af2023b\": container with ID starting with 1f4635273454350be029575dd68c8b83dc653de3ddac1cc66544bfc46af2023b not found: ID does not exist"
May 25 11:19:53.875950 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:53.875935    2561 scope.go:117] "RemoveContainer" containerID="77fb2740ebe94c49986d18fe57931a73e6da46f8bbf9f6166aa62b6e1b70ab28"
May 25 11:19:53.876134 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:19:53.876117    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77fb2740ebe94c49986d18fe57931a73e6da46f8bbf9f6166aa62b6e1b70ab28\": container with ID starting with 77fb2740ebe94c49986d18fe57931a73e6da46f8bbf9f6166aa62b6e1b70ab28 not found: ID does not exist" containerID="77fb2740ebe94c49986d18fe57931a73e6da46f8bbf9f6166aa62b6e1b70ab28"
May 25 11:19:53.876170 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:53.876141    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77fb2740ebe94c49986d18fe57931a73e6da46f8bbf9f6166aa62b6e1b70ab28"} err="failed to get container status \"77fb2740ebe94c49986d18fe57931a73e6da46f8bbf9f6166aa62b6e1b70ab28\": rpc error: code = NotFound desc = could not find container \"77fb2740ebe94c49986d18fe57931a73e6da46f8bbf9f6166aa62b6e1b70ab28\": container with ID starting with 77fb2740ebe94c49986d18fe57931a73e6da46f8bbf9f6166aa62b6e1b70ab28 not found: ID does not exist"
May 25 11:19:53.876170 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:53.876156    2561 scope.go:117] "RemoveContainer" containerID="c595ca6d95d1e83b483036990a03d60a9e0e3951f796199cb98404528b1c8e03"
May 25 11:19:53.876512 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:53.876493    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c595ca6d95d1e83b483036990a03d60a9e0e3951f796199cb98404528b1c8e03"} err="failed to get container status \"c595ca6d95d1e83b483036990a03d60a9e0e3951f796199cb98404528b1c8e03\": rpc error: code = NotFound desc = could not find container \"c595ca6d95d1e83b483036990a03d60a9e0e3951f796199cb98404528b1c8e03\": container with ID starting with c595ca6d95d1e83b483036990a03d60a9e0e3951f796199cb98404528b1c8e03 not found: ID does not exist"
May 25 11:19:53.876576 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:53.876512    2561 scope.go:117] "RemoveContainer" containerID="3d1dbdc6eed315467e25094ce0fc197b9c952d54bef5721ee7849bd090e96a80"
May 25 11:19:53.876724 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:53.876705    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d1dbdc6eed315467e25094ce0fc197b9c952d54bef5721ee7849bd090e96a80"} err="failed to get container status \"3d1dbdc6eed315467e25094ce0fc197b9c952d54bef5721ee7849bd090e96a80\": rpc error: code = NotFound desc = could not find container \"3d1dbdc6eed315467e25094ce0fc197b9c952d54bef5721ee7849bd090e96a80\": container with ID starting with 3d1dbdc6eed315467e25094ce0fc197b9c952d54bef5721ee7849bd090e96a80 not found: ID does not exist"
May 25 11:19:53.876793 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:53.876726    2561 scope.go:117] "RemoveContainer" containerID="cd02c74d52650e3e7e20d32f9f80033352f5ccbaaa7b7474a50b996eeaa1d4fe"
May 25 11:19:53.876963 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:53.876945    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd02c74d52650e3e7e20d32f9f80033352f5ccbaaa7b7474a50b996eeaa1d4fe"} err="failed to get container status \"cd02c74d52650e3e7e20d32f9f80033352f5ccbaaa7b7474a50b996eeaa1d4fe\": rpc error: code = NotFound desc = could not find container \"cd02c74d52650e3e7e20d32f9f80033352f5ccbaaa7b7474a50b996eeaa1d4fe\": container with ID starting with cd02c74d52650e3e7e20d32f9f80033352f5ccbaaa7b7474a50b996eeaa1d4fe not found: ID does not exist"
May 25 11:19:53.877006 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:53.876964    2561 scope.go:117] "RemoveContainer" containerID="73c9ecb470e2b22eac14037ba0972ae6efa213c1e92c0afa10ae424d5715160d"
May 25 11:19:53.877164 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:53.877147    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73c9ecb470e2b22eac14037ba0972ae6efa213c1e92c0afa10ae424d5715160d"} err="failed to get container status \"73c9ecb470e2b22eac14037ba0972ae6efa213c1e92c0afa10ae424d5715160d\": rpc error: code = NotFound desc = could not find container \"73c9ecb470e2b22eac14037ba0972ae6efa213c1e92c0afa10ae424d5715160d\": container with ID starting with 73c9ecb470e2b22eac14037ba0972ae6efa213c1e92c0afa10ae424d5715160d not found: ID does not exist"
May 25 11:19:53.877207 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:53.877165    2561 scope.go:117] "RemoveContainer" containerID="22b12500d9d4a100d1ec3b2ad330660b597d416a172c59244eb5a9a819068c8c"
May 25 11:19:53.877379 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:53.877362    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22b12500d9d4a100d1ec3b2ad330660b597d416a172c59244eb5a9a819068c8c"} err="failed to get container status \"22b12500d9d4a100d1ec3b2ad330660b597d416a172c59244eb5a9a819068c8c\": rpc error: code = NotFound desc = could not find container \"22b12500d9d4a100d1ec3b2ad330660b597d416a172c59244eb5a9a819068c8c\": container with ID starting with 22b12500d9d4a100d1ec3b2ad330660b597d416a172c59244eb5a9a819068c8c not found: ID does not exist"
May 25 11:19:53.877451 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:53.877380    2561 scope.go:117] "RemoveContainer" containerID="1f4635273454350be029575dd68c8b83dc653de3ddac1cc66544bfc46af2023b"
May 25 11:19:53.877581 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:53.877564    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f4635273454350be029575dd68c8b83dc653de3ddac1cc66544bfc46af2023b"} err="failed to get container status \"1f4635273454350be029575dd68c8b83dc653de3ddac1cc66544bfc46af2023b\": rpc error: code = NotFound desc = could not find container \"1f4635273454350be029575dd68c8b83dc653de3ddac1cc66544bfc46af2023b\": container with ID starting with 1f4635273454350be029575dd68c8b83dc653de3ddac1cc66544bfc46af2023b not found: ID does not exist"
May 25 11:19:53.877645 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:53.877582    2561 scope.go:117] "RemoveContainer" containerID="77fb2740ebe94c49986d18fe57931a73e6da46f8bbf9f6166aa62b6e1b70ab28"
May 25 11:19:53.877804 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:53.877786    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77fb2740ebe94c49986d18fe57931a73e6da46f8bbf9f6166aa62b6e1b70ab28"} err="failed to get container status \"77fb2740ebe94c49986d18fe57931a73e6da46f8bbf9f6166aa62b6e1b70ab28\": rpc error: code = NotFound desc = could not find container \"77fb2740ebe94c49986d18fe57931a73e6da46f8bbf9f6166aa62b6e1b70ab28\": container with ID starting with 77fb2740ebe94c49986d18fe57931a73e6da46f8bbf9f6166aa62b6e1b70ab28 not found: ID does not exist"
May 25 11:19:53.884218 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:53.884199    2561 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
May 25 11:19:53.884548 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:53.884533    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5805521a-51c5-4810-9cf6-ba306bfb50d9" containerName="kube-rbac-proxy"
May 25 11:19:53.884602 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:53.884550    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="5805521a-51c5-4810-9cf6-ba306bfb50d9" containerName="kube-rbac-proxy"
May 25 11:19:53.884602 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:53.884559    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5805521a-51c5-4810-9cf6-ba306bfb50d9" containerName="kube-rbac-proxy-metric"
May 25 11:19:53.884602 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:53.884564    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="5805521a-51c5-4810-9cf6-ba306bfb50d9" containerName="kube-rbac-proxy-metric"
May 25 11:19:53.884602 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:53.884573    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5805521a-51c5-4810-9cf6-ba306bfb50d9" containerName="alertmanager"
May 25 11:19:53.884602 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:53.884578    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="5805521a-51c5-4810-9cf6-ba306bfb50d9" containerName="alertmanager"
May 25 11:19:53.884602 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:53.884583    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5805521a-51c5-4810-9cf6-ba306bfb50d9" containerName="init-config-reloader"
May 25 11:19:53.884602 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:53.884588    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="5805521a-51c5-4810-9cf6-ba306bfb50d9" containerName="init-config-reloader"
May 25 11:19:53.884602 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:53.884596    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21792ece-de04-4eb0-990c-e1adec1c79c5" containerName="registry"
May 25 11:19:53.884602 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:53.884600    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="21792ece-de04-4eb0-990c-e1adec1c79c5" containerName="registry"
May 25 11:19:53.884871 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:53.884607    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12498c7a-38d3-4424-919e-0e430256aa0c" containerName="console"
May 25 11:19:53.884871 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:53.884612    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="12498c7a-38d3-4424-919e-0e430256aa0c" containerName="console"
May 25 11:19:53.884871 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:53.884619    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5805521a-51c5-4810-9cf6-ba306bfb50d9" containerName="config-reloader"
May 25 11:19:53.884871 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:53.884624    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="5805521a-51c5-4810-9cf6-ba306bfb50d9" containerName="config-reloader"
May 25 11:19:53.884871 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:53.884629    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5805521a-51c5-4810-9cf6-ba306bfb50d9" containerName="prom-label-proxy"
May 25 11:19:53.884871 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:53.884634    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="5805521a-51c5-4810-9cf6-ba306bfb50d9" containerName="prom-label-proxy"
May 25 11:19:53.884871 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:53.884648    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5805521a-51c5-4810-9cf6-ba306bfb50d9" containerName="kube-rbac-proxy-web"
May 25 11:19:53.884871 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:53.884653    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="5805521a-51c5-4810-9cf6-ba306bfb50d9" containerName="kube-rbac-proxy-web"
May 25 11:19:53.884871 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:53.884698    2561 memory_manager.go:356] "RemoveStaleState removing state" podUID="5805521a-51c5-4810-9cf6-ba306bfb50d9" containerName="kube-rbac-proxy-metric"
May 25 11:19:53.884871 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:53.884706    2561 memory_manager.go:356] "RemoveStaleState removing state" podUID="5805521a-51c5-4810-9cf6-ba306bfb50d9" containerName="kube-rbac-proxy"
May 25 11:19:53.884871 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:53.884712    2561 memory_manager.go:356] "RemoveStaleState removing state" podUID="5805521a-51c5-4810-9cf6-ba306bfb50d9" containerName="alertmanager"
May 25 11:19:53.884871 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:53.884718    2561 memory_manager.go:356] "RemoveStaleState removing state" podUID="5805521a-51c5-4810-9cf6-ba306bfb50d9" containerName="prom-label-proxy"
May 25 11:19:53.884871 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:53.884723    2561 memory_manager.go:356] "RemoveStaleState removing state" podUID="21792ece-de04-4eb0-990c-e1adec1c79c5" containerName="registry"
May 25 11:19:53.884871 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:53.884729    2561 memory_manager.go:356] "RemoveStaleState removing state" podUID="5805521a-51c5-4810-9cf6-ba306bfb50d9" containerName="kube-rbac-proxy-web"
May 25 11:19:53.884871 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:53.884737    2561 memory_manager.go:356] "RemoveStaleState removing state" podUID="12498c7a-38d3-4424-919e-0e430256aa0c" containerName="console"
May 25 11:19:53.884871 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:53.884743    2561 memory_manager.go:356] "RemoveStaleState removing state" podUID="5805521a-51c5-4810-9cf6-ba306bfb50d9" containerName="config-reloader"
May 25 11:19:53.887667 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:53.887653    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
May 25 11:19:53.890411 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:53.890391    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
May 25 11:19:53.890411 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:53.890403    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-k6rq2\""
May 25 11:19:53.890612 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:53.890454    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
May 25 11:19:53.890612 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:53.890504    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
May 25 11:19:53.890612 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:53.890536    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
May 25 11:19:53.890612 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:53.890454    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
May 25 11:19:53.890871 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:53.890778    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
May 25 11:19:53.890871 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:53.890801    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
May 25 11:19:53.890871 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:53.890821    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
May 25 11:19:53.896053 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:53.896035    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
May 25 11:19:53.899928 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:53.899910    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
May 25 11:19:53.944032 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:53.944007    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5c807139-8565-45a2-98d3-aed4e9ba4d37-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"5c807139-8565-45a2-98d3-aed4e9ba4d37\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 11:19:53.944112 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:53.944045    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5c807139-8565-45a2-98d3-aed4e9ba4d37-web-config\") pod \"alertmanager-main-0\" (UID: \"5c807139-8565-45a2-98d3-aed4e9ba4d37\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 11:19:53.944112 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:53.944062    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5c807139-8565-45a2-98d3-aed4e9ba4d37-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"5c807139-8565-45a2-98d3-aed4e9ba4d37\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 11:19:53.944112 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:53.944086    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqjgc\" (UniqueName: \"kubernetes.io/projected/5c807139-8565-45a2-98d3-aed4e9ba4d37-kube-api-access-hqjgc\") pod \"alertmanager-main-0\" (UID: \"5c807139-8565-45a2-98d3-aed4e9ba4d37\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 11:19:53.944210 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:53.944143    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5c807139-8565-45a2-98d3-aed4e9ba4d37-config-volume\") pod \"alertmanager-main-0\" (UID: \"5c807139-8565-45a2-98d3-aed4e9ba4d37\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 11:19:53.944210 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:53.944183    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5c807139-8565-45a2-98d3-aed4e9ba4d37-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"5c807139-8565-45a2-98d3-aed4e9ba4d37\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 11:19:53.944210 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:53.944204    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5c807139-8565-45a2-98d3-aed4e9ba4d37-tls-assets\") pod \"alertmanager-main-0\" (UID: \"5c807139-8565-45a2-98d3-aed4e9ba4d37\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 11:19:53.944315 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:53.944221    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5c807139-8565-45a2-98d3-aed4e9ba4d37-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"5c807139-8565-45a2-98d3-aed4e9ba4d37\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 11:19:53.944315 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:53.944242    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5c807139-8565-45a2-98d3-aed4e9ba4d37-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"5c807139-8565-45a2-98d3-aed4e9ba4d37\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 11:19:53.944377 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:53.944327    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5c807139-8565-45a2-98d3-aed4e9ba4d37-config-out\") pod \"alertmanager-main-0\" (UID: \"5c807139-8565-45a2-98d3-aed4e9ba4d37\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 11:19:53.944377 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:53.944359    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5c807139-8565-45a2-98d3-aed4e9ba4d37-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"5c807139-8565-45a2-98d3-aed4e9ba4d37\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 11:19:53.944433 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:53.944381    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5c807139-8565-45a2-98d3-aed4e9ba4d37-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"5c807139-8565-45a2-98d3-aed4e9ba4d37\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 11:19:53.944433 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:53.944406    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5c807139-8565-45a2-98d3-aed4e9ba4d37-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"5c807139-8565-45a2-98d3-aed4e9ba4d37\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 11:19:54.045046 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:54.044999    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5c807139-8565-45a2-98d3-aed4e9ba4d37-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"5c807139-8565-45a2-98d3-aed4e9ba4d37\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 11:19:54.045046 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:54.045028    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5c807139-8565-45a2-98d3-aed4e9ba4d37-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"5c807139-8565-45a2-98d3-aed4e9ba4d37\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 11:19:54.045152 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:54.045049    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5c807139-8565-45a2-98d3-aed4e9ba4d37-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"5c807139-8565-45a2-98d3-aed4e9ba4d37\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 11:19:54.045152 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:54.045094    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5c807139-8565-45a2-98d3-aed4e9ba4d37-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"5c807139-8565-45a2-98d3-aed4e9ba4d37\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 11:19:54.045152 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:54.045141    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5c807139-8565-45a2-98d3-aed4e9ba4d37-web-config\") pod \"alertmanager-main-0\" (UID: \"5c807139-8565-45a2-98d3-aed4e9ba4d37\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 11:19:54.045315 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:54.045164    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5c807139-8565-45a2-98d3-aed4e9ba4d37-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"5c807139-8565-45a2-98d3-aed4e9ba4d37\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 11:19:54.045315 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:54.045192    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-hqjgc\" (UniqueName: \"kubernetes.io/projected/5c807139-8565-45a2-98d3-aed4e9ba4d37-kube-api-access-hqjgc\") pod \"alertmanager-main-0\" (UID: \"5c807139-8565-45a2-98d3-aed4e9ba4d37\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 11:19:54.045414 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:54.045315    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5c807139-8565-45a2-98d3-aed4e9ba4d37-config-volume\") pod \"alertmanager-main-0\" (UID: \"5c807139-8565-45a2-98d3-aed4e9ba4d37\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 11:19:54.045414 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:54.045358    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5c807139-8565-45a2-98d3-aed4e9ba4d37-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"5c807139-8565-45a2-98d3-aed4e9ba4d37\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 11:19:54.045414 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:54.045387    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5c807139-8565-45a2-98d3-aed4e9ba4d37-tls-assets\") pod \"alertmanager-main-0\" (UID: \"5c807139-8565-45a2-98d3-aed4e9ba4d37\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 11:19:54.045597 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:54.045417    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5c807139-8565-45a2-98d3-aed4e9ba4d37-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"5c807139-8565-45a2-98d3-aed4e9ba4d37\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 11:19:54.045597 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:54.045448    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5c807139-8565-45a2-98d3-aed4e9ba4d37-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"5c807139-8565-45a2-98d3-aed4e9ba4d37\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 11:19:54.045597 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:54.045482    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5c807139-8565-45a2-98d3-aed4e9ba4d37-config-out\") pod \"alertmanager-main-0\" (UID: \"5c807139-8565-45a2-98d3-aed4e9ba4d37\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 11:19:54.045873 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:54.045852    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5c807139-8565-45a2-98d3-aed4e9ba4d37-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"5c807139-8565-45a2-98d3-aed4e9ba4d37\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 11:19:54.046399 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:54.046375    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5c807139-8565-45a2-98d3-aed4e9ba4d37-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"5c807139-8565-45a2-98d3-aed4e9ba4d37\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 11:19:54.046767 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:54.046748    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5c807139-8565-45a2-98d3-aed4e9ba4d37-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"5c807139-8565-45a2-98d3-aed4e9ba4d37\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 11:19:54.048022 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:54.047992    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5c807139-8565-45a2-98d3-aed4e9ba4d37-config-out\") pod \"alertmanager-main-0\" (UID: \"5c807139-8565-45a2-98d3-aed4e9ba4d37\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 11:19:54.048164 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:54.048050    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5c807139-8565-45a2-98d3-aed4e9ba4d37-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"5c807139-8565-45a2-98d3-aed4e9ba4d37\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 11:19:54.048164 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:54.048074    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5c807139-8565-45a2-98d3-aed4e9ba4d37-web-config\") pod \"alertmanager-main-0\" (UID: \"5c807139-8565-45a2-98d3-aed4e9ba4d37\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 11:19:54.048164 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:54.048137    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5c807139-8565-45a2-98d3-aed4e9ba4d37-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"5c807139-8565-45a2-98d3-aed4e9ba4d37\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 11:19:54.048164 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:54.048143    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5c807139-8565-45a2-98d3-aed4e9ba4d37-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"5c807139-8565-45a2-98d3-aed4e9ba4d37\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 11:19:54.048428 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:54.048417    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5c807139-8565-45a2-98d3-aed4e9ba4d37-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"5c807139-8565-45a2-98d3-aed4e9ba4d37\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 11:19:54.048484 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:54.048459    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5c807139-8565-45a2-98d3-aed4e9ba4d37-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"5c807139-8565-45a2-98d3-aed4e9ba4d37\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 11:19:54.049052 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:54.049033    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5c807139-8565-45a2-98d3-aed4e9ba4d37-config-volume\") pod \"alertmanager-main-0\" (UID: \"5c807139-8565-45a2-98d3-aed4e9ba4d37\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 11:19:54.049808 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:54.049793    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5c807139-8565-45a2-98d3-aed4e9ba4d37-tls-assets\") pod \"alertmanager-main-0\" (UID: \"5c807139-8565-45a2-98d3-aed4e9ba4d37\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 11:19:54.052740 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:54.052721    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqjgc\" (UniqueName: \"kubernetes.io/projected/5c807139-8565-45a2-98d3-aed4e9ba4d37-kube-api-access-hqjgc\") pod \"alertmanager-main-0\" (UID: \"5c807139-8565-45a2-98d3-aed4e9ba4d37\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 11:19:54.196990 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:54.196954    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
May 25 11:19:54.332154 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:54.332125    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
May 25 11:19:54.334819 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:19:54.334794    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c807139_8565_45a2_98d3_aed4e9ba4d37.slice/crio-51d30d9a18da59b7cc55ed51660064b975882da5b6e06784779b6901ff29d00c WatchSource:0}: Error finding container 51d30d9a18da59b7cc55ed51660064b975882da5b6e06784779b6901ff29d00c: Status 404 returned error can't find the container with id 51d30d9a18da59b7cc55ed51660064b975882da5b6e06784779b6901ff29d00c
May 25 11:19:54.832602 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:54.832572    2561 generic.go:358] "Generic (PLEG): container finished" podID="5c807139-8565-45a2-98d3-aed4e9ba4d37" containerID="c2410c9ccbe6ca7cae4c9740e4738de540b2a9fa91b0da127da44d0cb38c2252" exitCode=0
May 25 11:19:54.832987 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:54.832636    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5c807139-8565-45a2-98d3-aed4e9ba4d37","Type":"ContainerDied","Data":"c2410c9ccbe6ca7cae4c9740e4738de540b2a9fa91b0da127da44d0cb38c2252"}
May 25 11:19:54.832987 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:54.832660    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5c807139-8565-45a2-98d3-aed4e9ba4d37","Type":"ContainerStarted","Data":"51d30d9a18da59b7cc55ed51660064b975882da5b6e06784779b6901ff29d00c"}
May 25 11:19:55.118802 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:55.118773    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5805521a-51c5-4810-9cf6-ba306bfb50d9" path="/var/lib/kubelet/pods/5805521a-51c5-4810-9cf6-ba306bfb50d9/volumes"
May 25 11:19:55.839485 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:55.839449    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5c807139-8565-45a2-98d3-aed4e9ba4d37","Type":"ContainerStarted","Data":"e6330d9dc38a69a5efe06cbc906a3eb885b42b8f41ad6c3640c42c6e4d7e2170"}
May 25 11:19:55.839485 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:55.839489    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5c807139-8565-45a2-98d3-aed4e9ba4d37","Type":"ContainerStarted","Data":"20fb000f54f2eada81c1d45b35927ad3938ebbc6a6049880abb3f0e828de1815"}
May 25 11:19:55.839886 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:55.839502    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5c807139-8565-45a2-98d3-aed4e9ba4d37","Type":"ContainerStarted","Data":"5eefffc6e787672b8eebea4d256355fb6c3a5ab7815294d8b719a93fb26d0a94"}
May 25 11:19:55.839886 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:55.839515    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5c807139-8565-45a2-98d3-aed4e9ba4d37","Type":"ContainerStarted","Data":"1715f62903375a6a00d8fba9c2f4e3ca64a4568a68c6c8ca27275b6d0b760963"}
May 25 11:19:55.839886 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:55.839525    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5c807139-8565-45a2-98d3-aed4e9ba4d37","Type":"ContainerStarted","Data":"49fa76f7f13549fc17dfbbbbaf4e123ed3febfaf10b18b670d9a9423de3ad2d6"}
May 25 11:19:55.839886 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:55.839537    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5c807139-8565-45a2-98d3-aed4e9ba4d37","Type":"ContainerStarted","Data":"c9b97aa1ede248f916cd5a489fd6c945132a903c09e950cfb75e5519a34d42c3"}
May 25 11:19:55.881157 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:55.881028    2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.881009872 podStartE2EDuration="2.881009872s" podCreationTimestamp="2026-05-25 11:19:53 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 11:19:55.878723326 +0000 UTC m=+169.423900929" watchObservedRunningTime="2026-05-25 11:19:55.881009872 +0000 UTC m=+169.426187476"
May 25 11:19:56.477958 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:56.477875    2561 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-5fc4d84cc6-jh48n"]
May 25 11:19:56.480062 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:56.480047    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5fc4d84cc6-jh48n"
May 25 11:19:56.482455 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:56.482427    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
May 25 11:19:56.482586 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:56.482453    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
May 25 11:19:56.482867 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:56.482849    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
May 25 11:19:56.483038 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:56.483017    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
May 25 11:19:56.483131 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:56.483114    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
May 25 11:19:56.483197 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:56.483143    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-mmttp\""
May 25 11:19:56.491167 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:56.491149    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
May 25 11:19:56.501893 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:56.501872    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5fc4d84cc6-jh48n"]
May 25 11:19:56.566833 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:56.566804    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0df23aa9-66a4-415d-9c86-402e09fc814a-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5fc4d84cc6-jh48n\" (UID: \"0df23aa9-66a4-415d-9c86-402e09fc814a\") " pod="openshift-monitoring/telemeter-client-5fc4d84cc6-jh48n"
May 25 11:19:56.566982 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:56.566840    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0df23aa9-66a4-415d-9c86-402e09fc814a-metrics-client-ca\") pod \"telemeter-client-5fc4d84cc6-jh48n\" (UID: \"0df23aa9-66a4-415d-9c86-402e09fc814a\") " pod="openshift-monitoring/telemeter-client-5fc4d84cc6-jh48n"
May 25 11:19:56.566982 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:56.566868    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0df23aa9-66a4-415d-9c86-402e09fc814a-serving-certs-ca-bundle\") pod \"telemeter-client-5fc4d84cc6-jh48n\" (UID: \"0df23aa9-66a4-415d-9c86-402e09fc814a\") " pod="openshift-monitoring/telemeter-client-5fc4d84cc6-jh48n"
May 25 11:19:56.566982 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:56.566909    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/0df23aa9-66a4-415d-9c86-402e09fc814a-telemeter-client-tls\") pod \"telemeter-client-5fc4d84cc6-jh48n\" (UID: \"0df23aa9-66a4-415d-9c86-402e09fc814a\") " pod="openshift-monitoring/telemeter-client-5fc4d84cc6-jh48n"
May 25 11:19:56.566982 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:56.566935    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/0df23aa9-66a4-415d-9c86-402e09fc814a-federate-client-tls\") pod \"telemeter-client-5fc4d84cc6-jh48n\" (UID: \"0df23aa9-66a4-415d-9c86-402e09fc814a\") " pod="openshift-monitoring/telemeter-client-5fc4d84cc6-jh48n"
May 25 11:19:56.566982 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:56.566955    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0df23aa9-66a4-415d-9c86-402e09fc814a-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5fc4d84cc6-jh48n\" (UID: \"0df23aa9-66a4-415d-9c86-402e09fc814a\") " pod="openshift-monitoring/telemeter-client-5fc4d84cc6-jh48n"
May 25 11:19:56.566982 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:56.566972    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/0df23aa9-66a4-415d-9c86-402e09fc814a-secret-telemeter-client\") pod \"telemeter-client-5fc4d84cc6-jh48n\" (UID: \"0df23aa9-66a4-415d-9c86-402e09fc814a\") " pod="openshift-monitoring/telemeter-client-5fc4d84cc6-jh48n"
May 25 11:19:56.567185 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:56.566990    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jvs2\" (UniqueName: \"kubernetes.io/projected/0df23aa9-66a4-415d-9c86-402e09fc814a-kube-api-access-9jvs2\") pod \"telemeter-client-5fc4d84cc6-jh48n\" (UID: \"0df23aa9-66a4-415d-9c86-402e09fc814a\") " pod="openshift-monitoring/telemeter-client-5fc4d84cc6-jh48n"
May 25 11:19:56.667564 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:56.667529    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0df23aa9-66a4-415d-9c86-402e09fc814a-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5fc4d84cc6-jh48n\" (UID: \"0df23aa9-66a4-415d-9c86-402e09fc814a\") " pod="openshift-monitoring/telemeter-client-5fc4d84cc6-jh48n"
May 25 11:19:56.667678 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:56.667570    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0df23aa9-66a4-415d-9c86-402e09fc814a-metrics-client-ca\") pod \"telemeter-client-5fc4d84cc6-jh48n\" (UID: \"0df23aa9-66a4-415d-9c86-402e09fc814a\") " pod="openshift-monitoring/telemeter-client-5fc4d84cc6-jh48n"
May 25 11:19:56.667678 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:56.667603    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0df23aa9-66a4-415d-9c86-402e09fc814a-serving-certs-ca-bundle\") pod \"telemeter-client-5fc4d84cc6-jh48n\" (UID: \"0df23aa9-66a4-415d-9c86-402e09fc814a\") " pod="openshift-monitoring/telemeter-client-5fc4d84cc6-jh48n"
May 25 11:19:56.667678 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:56.667650    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/0df23aa9-66a4-415d-9c86-402e09fc814a-telemeter-client-tls\") pod \"telemeter-client-5fc4d84cc6-jh48n\" (UID: \"0df23aa9-66a4-415d-9c86-402e09fc814a\") " pod="openshift-monitoring/telemeter-client-5fc4d84cc6-jh48n"
May 25 11:19:56.667841 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:56.667688    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/0df23aa9-66a4-415d-9c86-402e09fc814a-federate-client-tls\") pod \"telemeter-client-5fc4d84cc6-jh48n\" (UID: \"0df23aa9-66a4-415d-9c86-402e09fc814a\") " pod="openshift-monitoring/telemeter-client-5fc4d84cc6-jh48n"
May 25 11:19:56.667841 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:56.667713    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0df23aa9-66a4-415d-9c86-402e09fc814a-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5fc4d84cc6-jh48n\" (UID: \"0df23aa9-66a4-415d-9c86-402e09fc814a\") " pod="openshift-monitoring/telemeter-client-5fc4d84cc6-jh48n"
May 25 11:19:56.667841 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:56.667780    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/0df23aa9-66a4-415d-9c86-402e09fc814a-secret-telemeter-client\") pod \"telemeter-client-5fc4d84cc6-jh48n\" (UID: \"0df23aa9-66a4-415d-9c86-402e09fc814a\") " pod="openshift-monitoring/telemeter-client-5fc4d84cc6-jh48n"
May 25 11:19:56.667841 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:56.667822    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-9jvs2\" (UniqueName: \"kubernetes.io/projected/0df23aa9-66a4-415d-9c86-402e09fc814a-kube-api-access-9jvs2\") pod \"telemeter-client-5fc4d84cc6-jh48n\" (UID: \"0df23aa9-66a4-415d-9c86-402e09fc814a\") " pod="openshift-monitoring/telemeter-client-5fc4d84cc6-jh48n"
May 25 11:19:56.668384 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:56.668359    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0df23aa9-66a4-415d-9c86-402e09fc814a-serving-certs-ca-bundle\") pod \"telemeter-client-5fc4d84cc6-jh48n\" (UID: \"0df23aa9-66a4-415d-9c86-402e09fc814a\") " pod="openshift-monitoring/telemeter-client-5fc4d84cc6-jh48n"
May 25 11:19:56.668496 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:56.668416    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0df23aa9-66a4-415d-9c86-402e09fc814a-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5fc4d84cc6-jh48n\" (UID: \"0df23aa9-66a4-415d-9c86-402e09fc814a\") " pod="openshift-monitoring/telemeter-client-5fc4d84cc6-jh48n"
May 25 11:19:56.668496 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:56.668417    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0df23aa9-66a4-415d-9c86-402e09fc814a-metrics-client-ca\") pod \"telemeter-client-5fc4d84cc6-jh48n\" (UID: \"0df23aa9-66a4-415d-9c86-402e09fc814a\") " pod="openshift-monitoring/telemeter-client-5fc4d84cc6-jh48n"
May 25 11:19:56.670153 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:56.670132    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/0df23aa9-66a4-415d-9c86-402e09fc814a-federate-client-tls\") pod \"telemeter-client-5fc4d84cc6-jh48n\" (UID: \"0df23aa9-66a4-415d-9c86-402e09fc814a\") " pod="openshift-monitoring/telemeter-client-5fc4d84cc6-jh48n"
May 25 11:19:56.670245 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:56.670186    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0df23aa9-66a4-415d-9c86-402e09fc814a-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5fc4d84cc6-jh48n\" (UID: \"0df23aa9-66a4-415d-9c86-402e09fc814a\") " pod="openshift-monitoring/telemeter-client-5fc4d84cc6-jh48n"
May 25 11:19:56.670376 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:56.670353    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/0df23aa9-66a4-415d-9c86-402e09fc814a-secret-telemeter-client\") pod \"telemeter-client-5fc4d84cc6-jh48n\" (UID: \"0df23aa9-66a4-415d-9c86-402e09fc814a\") " pod="openshift-monitoring/telemeter-client-5fc4d84cc6-jh48n"
May 25 11:19:56.670464 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:56.670450    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/0df23aa9-66a4-415d-9c86-402e09fc814a-telemeter-client-tls\") pod \"telemeter-client-5fc4d84cc6-jh48n\" (UID: \"0df23aa9-66a4-415d-9c86-402e09fc814a\") " pod="openshift-monitoring/telemeter-client-5fc4d84cc6-jh48n"
May 25 11:19:56.675827 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:56.675810    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jvs2\" (UniqueName: \"kubernetes.io/projected/0df23aa9-66a4-415d-9c86-402e09fc814a-kube-api-access-9jvs2\") pod \"telemeter-client-5fc4d84cc6-jh48n\" (UID: \"0df23aa9-66a4-415d-9c86-402e09fc814a\") " pod="openshift-monitoring/telemeter-client-5fc4d84cc6-jh48n"
May 25 11:19:56.790897 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:56.790825    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5fc4d84cc6-jh48n"
May 25 11:19:56.932964 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:56.932934    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5fc4d84cc6-jh48n"]
May 25 11:19:56.935456 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:19:56.935425    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0df23aa9_66a4_415d_9c86_402e09fc814a.slice/crio-1f3a745f966cbe2ceb9c4e254c2d7f40d5f74b945f10049d4b2ce1427a17e735 WatchSource:0}: Error finding container 1f3a745f966cbe2ceb9c4e254c2d7f40d5f74b945f10049d4b2ce1427a17e735: Status 404 returned error can't find the container with id 1f3a745f966cbe2ceb9c4e254c2d7f40d5f74b945f10049d4b2ce1427a17e735
May 25 11:19:57.848381 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:57.848340    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5fc4d84cc6-jh48n" event={"ID":"0df23aa9-66a4-415d-9c86-402e09fc814a","Type":"ContainerStarted","Data":"1f3a745f966cbe2ceb9c4e254c2d7f40d5f74b945f10049d4b2ce1427a17e735"}
May 25 11:19:58.853717 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:58.853682    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5fc4d84cc6-jh48n" event={"ID":"0df23aa9-66a4-415d-9c86-402e09fc814a","Type":"ContainerStarted","Data":"521ce2fab8d3fc7b01e2973cf0fab491b05f01d83e675a96a234a35a0471b4b4"}
May 25 11:19:58.853717 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:58.853722    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5fc4d84cc6-jh48n" event={"ID":"0df23aa9-66a4-415d-9c86-402e09fc814a","Type":"ContainerStarted","Data":"08af742ea306bb4e8873c3d4bb37287daa133c4daf6a7f0f0716d575b260cad4"}
May 25 11:19:58.854102 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:58.853731    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5fc4d84cc6-jh48n" event={"ID":"0df23aa9-66a4-415d-9c86-402e09fc814a","Type":"ContainerStarted","Data":"5f6af8c025fcf3d970503e301c396aae894e5db00f2d5f7035e32bbef3b29ef8"}
May 25 11:19:58.882392 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:19:58.882352    2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-5fc4d84cc6-jh48n" podStartSLOduration=1.4284549389999999 podStartE2EDuration="2.882340026s" podCreationTimestamp="2026-05-25 11:19:56 +0000 UTC" firstStartedPulling="2026-05-25 11:19:56.937787229 +0000 UTC m=+170.482964810" lastFinishedPulling="2026-05-25 11:19:58.391672317 +0000 UTC m=+171.936849897" observedRunningTime="2026-05-25 11:19:58.880852553 +0000 UTC m=+172.426030157" watchObservedRunningTime="2026-05-25 11:19:58.882340026 +0000 UTC m=+172.427517628"
May 25 11:20:10.261285 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:20:10.261230    2561 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-77d77bf5f6-ggwqf"]
May 25 11:20:35.282771 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:20:35.282727    2561 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-console/console-77d77bf5f6-ggwqf" podUID="b9e4ef06-b5da-4a36-846a-f21769ac72d1" containerName="console" containerID="cri-o://8dfa963c42ac57cb9d0deca58cc4f6eb5fd09459fe7016d28eca7f16378dda8e" gracePeriod=15
May 25 11:20:35.529172 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:20:35.529150    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-77d77bf5f6-ggwqf_b9e4ef06-b5da-4a36-846a-f21769ac72d1/console/0.log"
May 25 11:20:35.529283 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:20:35.529209    2561 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-77d77bf5f6-ggwqf"
May 25 11:20:35.675906 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:20:35.675886    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ltbbv\" (UniqueName: \"kubernetes.io/projected/b9e4ef06-b5da-4a36-846a-f21769ac72d1-kube-api-access-ltbbv\") pod \"b9e4ef06-b5da-4a36-846a-f21769ac72d1\" (UID: \"b9e4ef06-b5da-4a36-846a-f21769ac72d1\") "
May 25 11:20:35.676019 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:20:35.675919    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b9e4ef06-b5da-4a36-846a-f21769ac72d1-console-oauth-config\") pod \"b9e4ef06-b5da-4a36-846a-f21769ac72d1\" (UID: \"b9e4ef06-b5da-4a36-846a-f21769ac72d1\") "
May 25 11:20:35.676019 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:20:35.675958    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b9e4ef06-b5da-4a36-846a-f21769ac72d1-oauth-serving-cert\") pod \"b9e4ef06-b5da-4a36-846a-f21769ac72d1\" (UID: \"b9e4ef06-b5da-4a36-846a-f21769ac72d1\") "
May 25 11:20:35.676099 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:20:35.676064    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b9e4ef06-b5da-4a36-846a-f21769ac72d1-console-serving-cert\") pod \"b9e4ef06-b5da-4a36-846a-f21769ac72d1\" (UID: \"b9e4ef06-b5da-4a36-846a-f21769ac72d1\") "
May 25 11:20:35.676149 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:20:35.676101    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b9e4ef06-b5da-4a36-846a-f21769ac72d1-console-config\") pod \"b9e4ef06-b5da-4a36-846a-f21769ac72d1\" (UID: \"b9e4ef06-b5da-4a36-846a-f21769ac72d1\") "
May 25 11:20:35.676149 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:20:35.676131    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b9e4ef06-b5da-4a36-846a-f21769ac72d1-service-ca\") pod \"b9e4ef06-b5da-4a36-846a-f21769ac72d1\" (UID: \"b9e4ef06-b5da-4a36-846a-f21769ac72d1\") "
May 25 11:20:35.676303 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:20:35.676146    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b9e4ef06-b5da-4a36-846a-f21769ac72d1-trusted-ca-bundle\") pod \"b9e4ef06-b5da-4a36-846a-f21769ac72d1\" (UID: \"b9e4ef06-b5da-4a36-846a-f21769ac72d1\") "
May 25 11:20:35.676463 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:20:35.676421    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9e4ef06-b5da-4a36-846a-f21769ac72d1-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "b9e4ef06-b5da-4a36-846a-f21769ac72d1" (UID: "b9e4ef06-b5da-4a36-846a-f21769ac72d1"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 11:20:35.676604 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:20:35.676575    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9e4ef06-b5da-4a36-846a-f21769ac72d1-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "b9e4ef06-b5da-4a36-846a-f21769ac72d1" (UID: "b9e4ef06-b5da-4a36-846a-f21769ac72d1"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 11:20:35.676604 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:20:35.676500    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9e4ef06-b5da-4a36-846a-f21769ac72d1-service-ca" (OuterVolumeSpecName: "service-ca") pod "b9e4ef06-b5da-4a36-846a-f21769ac72d1" (UID: "b9e4ef06-b5da-4a36-846a-f21769ac72d1"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 11:20:35.676741 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:20:35.676527    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9e4ef06-b5da-4a36-846a-f21769ac72d1-console-config" (OuterVolumeSpecName: "console-config") pod "b9e4ef06-b5da-4a36-846a-f21769ac72d1" (UID: "b9e4ef06-b5da-4a36-846a-f21769ac72d1"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 11:20:35.678005 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:20:35.677978    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9e4ef06-b5da-4a36-846a-f21769ac72d1-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "b9e4ef06-b5da-4a36-846a-f21769ac72d1" (UID: "b9e4ef06-b5da-4a36-846a-f21769ac72d1"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 11:20:35.678101 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:20:35.678070    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9e4ef06-b5da-4a36-846a-f21769ac72d1-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "b9e4ef06-b5da-4a36-846a-f21769ac72d1" (UID: "b9e4ef06-b5da-4a36-846a-f21769ac72d1"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 11:20:35.678240 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:20:35.678217    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9e4ef06-b5da-4a36-846a-f21769ac72d1-kube-api-access-ltbbv" (OuterVolumeSpecName: "kube-api-access-ltbbv") pod "b9e4ef06-b5da-4a36-846a-f21769ac72d1" (UID: "b9e4ef06-b5da-4a36-846a-f21769ac72d1"). InnerVolumeSpecName "kube-api-access-ltbbv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 11:20:35.777215 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:20:35.777181    2561 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b9e4ef06-b5da-4a36-846a-f21769ac72d1-console-serving-cert\") on node \"ip-10-0-136-227.ec2.internal\" DevicePath \"\""
May 25 11:20:35.777215 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:20:35.777214    2561 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b9e4ef06-b5da-4a36-846a-f21769ac72d1-console-config\") on node \"ip-10-0-136-227.ec2.internal\" DevicePath \"\""
May 25 11:20:35.777215 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:20:35.777223    2561 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b9e4ef06-b5da-4a36-846a-f21769ac72d1-service-ca\") on node \"ip-10-0-136-227.ec2.internal\" DevicePath \"\""
May 25 11:20:35.777389 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:20:35.777232    2561 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b9e4ef06-b5da-4a36-846a-f21769ac72d1-trusted-ca-bundle\") on node \"ip-10-0-136-227.ec2.internal\" DevicePath \"\""
May 25 11:20:35.777389 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:20:35.777241    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ltbbv\" (UniqueName: \"kubernetes.io/projected/b9e4ef06-b5da-4a36-846a-f21769ac72d1-kube-api-access-ltbbv\") on node \"ip-10-0-136-227.ec2.internal\" DevicePath \"\""
May 25 11:20:35.777389 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:20:35.777250    2561 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b9e4ef06-b5da-4a36-846a-f21769ac72d1-console-oauth-config\") on node \"ip-10-0-136-227.ec2.internal\" DevicePath \"\""
May 25 11:20:35.777389 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:20:35.777259    2561 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b9e4ef06-b5da-4a36-846a-f21769ac72d1-oauth-serving-cert\") on node \"ip-10-0-136-227.ec2.internal\" DevicePath \"\""
May 25 11:20:35.974410 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:20:35.974356    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-77d77bf5f6-ggwqf_b9e4ef06-b5da-4a36-846a-f21769ac72d1/console/0.log"
May 25 11:20:35.974410 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:20:35.974395    2561 generic.go:358] "Generic (PLEG): container finished" podID="b9e4ef06-b5da-4a36-846a-f21769ac72d1" containerID="8dfa963c42ac57cb9d0deca58cc4f6eb5fd09459fe7016d28eca7f16378dda8e" exitCode=2
May 25 11:20:35.974536 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:20:35.974429    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-77d77bf5f6-ggwqf" event={"ID":"b9e4ef06-b5da-4a36-846a-f21769ac72d1","Type":"ContainerDied","Data":"8dfa963c42ac57cb9d0deca58cc4f6eb5fd09459fe7016d28eca7f16378dda8e"}
May 25 11:20:35.974536 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:20:35.974467    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-77d77bf5f6-ggwqf" event={"ID":"b9e4ef06-b5da-4a36-846a-f21769ac72d1","Type":"ContainerDied","Data":"98ce06e0b39540021c1e2104a03ef1cd3367be27dd1a115728b13b676ac0abf4"}
May 25 11:20:35.974536 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:20:35.974468    2561 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-77d77bf5f6-ggwqf"
May 25 11:20:35.974536 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:20:35.974482    2561 scope.go:117] "RemoveContainer" containerID="8dfa963c42ac57cb9d0deca58cc4f6eb5fd09459fe7016d28eca7f16378dda8e"
May 25 11:20:35.982551 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:20:35.982520    2561 scope.go:117] "RemoveContainer" containerID="8dfa963c42ac57cb9d0deca58cc4f6eb5fd09459fe7016d28eca7f16378dda8e"
May 25 11:20:35.982848 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:20:35.982828    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8dfa963c42ac57cb9d0deca58cc4f6eb5fd09459fe7016d28eca7f16378dda8e\": container with ID starting with 8dfa963c42ac57cb9d0deca58cc4f6eb5fd09459fe7016d28eca7f16378dda8e not found: ID does not exist" containerID="8dfa963c42ac57cb9d0deca58cc4f6eb5fd09459fe7016d28eca7f16378dda8e"
May 25 11:20:35.982941 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:20:35.982854    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8dfa963c42ac57cb9d0deca58cc4f6eb5fd09459fe7016d28eca7f16378dda8e"} err="failed to get container status \"8dfa963c42ac57cb9d0deca58cc4f6eb5fd09459fe7016d28eca7f16378dda8e\": rpc error: code = NotFound desc = could not find container \"8dfa963c42ac57cb9d0deca58cc4f6eb5fd09459fe7016d28eca7f16378dda8e\": container with ID starting with 8dfa963c42ac57cb9d0deca58cc4f6eb5fd09459fe7016d28eca7f16378dda8e not found: ID does not exist"
May 25 11:20:35.996085 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:20:35.996062    2561 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-77d77bf5f6-ggwqf"]
May 25 11:20:35.999760 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:20:35.999738    2561 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-77d77bf5f6-ggwqf"]
May 25 11:20:37.116579 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:20:37.116544    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b9e4ef06-b5da-4a36-846a-f21769ac72d1" path="/var/lib/kubelet/pods/b9e4ef06-b5da-4a36-846a-f21769ac72d1/volumes"
May 25 11:22:06.980741 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:22:06.980714    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-9lj95_10a66344-3304-49a1-bbc1-b885800f5acf/console-operator/1.log"
May 25 11:22:06.982055 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:22:06.982024    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-9lj95_10a66344-3304-49a1-bbc1-b885800f5acf/console-operator/1.log"
May 25 11:22:06.988765 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:22:06.988746    2561 kubelet.go:1628] "Image garbage collection succeeded"
May 25 11:22:27.037724 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:22:27.037688    2561 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-operator-ffbb595cb-d68rr"]
May 25 11:22:27.040158 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:22:27.038135    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9e4ef06-b5da-4a36-846a-f21769ac72d1" containerName="console"
May 25 11:22:27.040158 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:22:27.038151    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9e4ef06-b5da-4a36-846a-f21769ac72d1" containerName="console"
May 25 11:22:27.040158 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:22:27.038207    2561 memory_manager.go:356] "RemoveStaleState removing state" podUID="b9e4ef06-b5da-4a36-846a-f21769ac72d1" containerName="console"
May 25 11:22:27.040992 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:22:27.040975    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-ffbb595cb-d68rr"
May 25 11:22:27.043544 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:22:27.043523    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-2ll8v\""
May 25 11:22:27.044244 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:22:27.044228    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-certs\""
May 25 11:22:27.044361 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:22:27.044310    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
May 25 11:22:27.044425 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:22:27.044359    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
May 25 11:22:27.044425 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:22:27.044414    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
May 25 11:22:27.044627 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:22:27.044610    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
May 25 11:22:27.053519 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:22:27.053499    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-ffbb595cb-d68rr"]
May 25 11:22:27.104896 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:22:27.104871    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/7b39dda4-21ee-4ee1-9a8b-d6042c965247-cabundle0\") pod \"keda-operator-ffbb595cb-d68rr\" (UID: \"7b39dda4-21ee-4ee1-9a8b-d6042c965247\") " pod="openshift-keda/keda-operator-ffbb595cb-d68rr"
May 25 11:22:27.105043 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:22:27.104930    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/7b39dda4-21ee-4ee1-9a8b-d6042c965247-certificates\") pod \"keda-operator-ffbb595cb-d68rr\" (UID: \"7b39dda4-21ee-4ee1-9a8b-d6042c965247\") " pod="openshift-keda/keda-operator-ffbb595cb-d68rr"
May 25 11:22:27.105043 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:22:27.104985    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzqdz\" (UniqueName: \"kubernetes.io/projected/7b39dda4-21ee-4ee1-9a8b-d6042c965247-kube-api-access-zzqdz\") pod \"keda-operator-ffbb595cb-d68rr\" (UID: \"7b39dda4-21ee-4ee1-9a8b-d6042c965247\") " pod="openshift-keda/keda-operator-ffbb595cb-d68rr"
May 25 11:22:27.205870 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:22:27.205835    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/7b39dda4-21ee-4ee1-9a8b-d6042c965247-certificates\") pod \"keda-operator-ffbb595cb-d68rr\" (UID: \"7b39dda4-21ee-4ee1-9a8b-d6042c965247\") " pod="openshift-keda/keda-operator-ffbb595cb-d68rr"
May 25 11:22:27.206007 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:22:27.205882    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-zzqdz\" (UniqueName: \"kubernetes.io/projected/7b39dda4-21ee-4ee1-9a8b-d6042c965247-kube-api-access-zzqdz\") pod \"keda-operator-ffbb595cb-d68rr\" (UID: \"7b39dda4-21ee-4ee1-9a8b-d6042c965247\") " pod="openshift-keda/keda-operator-ffbb595cb-d68rr"
May 25 11:22:27.206007 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:22:27.205918    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/7b39dda4-21ee-4ee1-9a8b-d6042c965247-cabundle0\") pod \"keda-operator-ffbb595cb-d68rr\" (UID: \"7b39dda4-21ee-4ee1-9a8b-d6042c965247\") " pod="openshift-keda/keda-operator-ffbb595cb-d68rr"
May 25 11:22:27.206007 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:22:27.205976    2561 projected.go:264] Couldn't get secret openshift-keda/keda-operator-certs: secret "keda-operator-certs" not found
May 25 11:22:27.206007 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:22:27.206000    2561 secret.go:281] references non-existent secret key: ca.crt
May 25 11:22:27.206007 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:22:27.206007    2561 projected.go:277] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
May 25 11:22:27.206206 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:22:27.206023    2561 projected.go:194] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-ffbb595cb-d68rr: [secret "keda-operator-certs" not found, references non-existent secret key: ca.crt]
May 25 11:22:27.206206 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:22:27.206076    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7b39dda4-21ee-4ee1-9a8b-d6042c965247-certificates podName:7b39dda4-21ee-4ee1-9a8b-d6042c965247 nodeName:}" failed. No retries permitted until 2026-05-25 11:22:27.706060923 +0000 UTC m=+321.251238509 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/7b39dda4-21ee-4ee1-9a8b-d6042c965247-certificates") pod "keda-operator-ffbb595cb-d68rr" (UID: "7b39dda4-21ee-4ee1-9a8b-d6042c965247") : [secret "keda-operator-certs" not found, references non-existent secret key: ca.crt]
May 25 11:22:27.206582 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:22:27.206563    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/7b39dda4-21ee-4ee1-9a8b-d6042c965247-cabundle0\") pod \"keda-operator-ffbb595cb-d68rr\" (UID: \"7b39dda4-21ee-4ee1-9a8b-d6042c965247\") " pod="openshift-keda/keda-operator-ffbb595cb-d68rr"
May 25 11:22:27.214563 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:22:27.214538    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzqdz\" (UniqueName: \"kubernetes.io/projected/7b39dda4-21ee-4ee1-9a8b-d6042c965247-kube-api-access-zzqdz\") pod \"keda-operator-ffbb595cb-d68rr\" (UID: \"7b39dda4-21ee-4ee1-9a8b-d6042c965247\") " pod="openshift-keda/keda-operator-ffbb595cb-d68rr"
May 25 11:22:27.286857 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:22:27.286831    2561 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-metrics-apiserver-7c9f485588-6bnwh"]
May 25 11:22:27.290426 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:22:27.290381    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-7c9f485588-6bnwh"
May 25 11:22:27.292788 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:22:27.292767    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-metrics-apiserver-certs\""
May 25 11:22:27.300414 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:22:27.300394    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-7c9f485588-6bnwh"]
May 25 11:22:27.407148 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:22:27.407122    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/9480b07c-7421-4c82-b7fd-a9a3ed4bbd40-certificates\") pod \"keda-metrics-apiserver-7c9f485588-6bnwh\" (UID: \"9480b07c-7421-4c82-b7fd-a9a3ed4bbd40\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-6bnwh"
May 25 11:22:27.407337 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:22:27.407152    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/9480b07c-7421-4c82-b7fd-a9a3ed4bbd40-temp-vol\") pod \"keda-metrics-apiserver-7c9f485588-6bnwh\" (UID: \"9480b07c-7421-4c82-b7fd-a9a3ed4bbd40\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-6bnwh"
May 25 11:22:27.407337 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:22:27.407258    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vq7bw\" (UniqueName: \"kubernetes.io/projected/9480b07c-7421-4c82-b7fd-a9a3ed4bbd40-kube-api-access-vq7bw\") pod \"keda-metrics-apiserver-7c9f485588-6bnwh\" (UID: \"9480b07c-7421-4c82-b7fd-a9a3ed4bbd40\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-6bnwh"
May 25 11:22:27.508494 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:22:27.508462    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-vq7bw\" (UniqueName: \"kubernetes.io/projected/9480b07c-7421-4c82-b7fd-a9a3ed4bbd40-kube-api-access-vq7bw\") pod \"keda-metrics-apiserver-7c9f485588-6bnwh\" (UID: \"9480b07c-7421-4c82-b7fd-a9a3ed4bbd40\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-6bnwh"
May 25 11:22:27.508644 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:22:27.508519    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/9480b07c-7421-4c82-b7fd-a9a3ed4bbd40-certificates\") pod \"keda-metrics-apiserver-7c9f485588-6bnwh\" (UID: \"9480b07c-7421-4c82-b7fd-a9a3ed4bbd40\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-6bnwh"
May 25 11:22:27.508644 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:22:27.508537    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/9480b07c-7421-4c82-b7fd-a9a3ed4bbd40-temp-vol\") pod \"keda-metrics-apiserver-7c9f485588-6bnwh\" (UID: \"9480b07c-7421-4c82-b7fd-a9a3ed4bbd40\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-6bnwh"
May 25 11:22:27.508719 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:22:27.508661    2561 secret.go:281] references non-existent secret key: tls.crt
May 25 11:22:27.508719 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:22:27.508681    2561 projected.go:277] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: tls.crt
May 25 11:22:27.508719 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:22:27.508700    2561 projected.go:194] Error preparing data for projected volume certificates for pod openshift-keda/keda-metrics-apiserver-7c9f485588-6bnwh: references non-existent secret key: tls.crt
May 25 11:22:27.508811 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:22:27.508754    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9480b07c-7421-4c82-b7fd-a9a3ed4bbd40-certificates podName:9480b07c-7421-4c82-b7fd-a9a3ed4bbd40 nodeName:}" failed. No retries permitted until 2026-05-25 11:22:28.00874048 +0000 UTC m=+321.553918060 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/9480b07c-7421-4c82-b7fd-a9a3ed4bbd40-certificates") pod "keda-metrics-apiserver-7c9f485588-6bnwh" (UID: "9480b07c-7421-4c82-b7fd-a9a3ed4bbd40") : references non-existent secret key: tls.crt
May 25 11:22:27.508902 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:22:27.508885    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/9480b07c-7421-4c82-b7fd-a9a3ed4bbd40-temp-vol\") pod \"keda-metrics-apiserver-7c9f485588-6bnwh\" (UID: \"9480b07c-7421-4c82-b7fd-a9a3ed4bbd40\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-6bnwh"
May 25 11:22:27.521556 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:22:27.521531    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vq7bw\" (UniqueName: \"kubernetes.io/projected/9480b07c-7421-4c82-b7fd-a9a3ed4bbd40-kube-api-access-vq7bw\") pod \"keda-metrics-apiserver-7c9f485588-6bnwh\" (UID: \"9480b07c-7421-4c82-b7fd-a9a3ed4bbd40\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-6bnwh"
May 25 11:22:27.524162 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:22:27.524141    2561 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-admission-cf49989db-5brl6"]
May 25 11:22:27.527375 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:22:27.527347    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-cf49989db-5brl6"
May 25 11:22:27.530162 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:22:27.530146    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-admission-webhooks-certs\""
May 25 11:22:27.556060 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:22:27.556007    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-cf49989db-5brl6"]
May 25 11:22:27.609773 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:22:27.609734    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fd5nh\" (UniqueName: \"kubernetes.io/projected/a457cbd8-6fca-4e8e-b70d-f733a6564bcb-kube-api-access-fd5nh\") pod \"keda-admission-cf49989db-5brl6\" (UID: \"a457cbd8-6fca-4e8e-b70d-f733a6564bcb\") " pod="openshift-keda/keda-admission-cf49989db-5brl6"
May 25 11:22:27.609948 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:22:27.609915    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/a457cbd8-6fca-4e8e-b70d-f733a6564bcb-certificates\") pod \"keda-admission-cf49989db-5brl6\" (UID: \"a457cbd8-6fca-4e8e-b70d-f733a6564bcb\") " pod="openshift-keda/keda-admission-cf49989db-5brl6"
May 25 11:22:27.710704 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:22:27.710667    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-fd5nh\" (UniqueName: \"kubernetes.io/projected/a457cbd8-6fca-4e8e-b70d-f733a6564bcb-kube-api-access-fd5nh\") pod \"keda-admission-cf49989db-5brl6\" (UID: \"a457cbd8-6fca-4e8e-b70d-f733a6564bcb\") " pod="openshift-keda/keda-admission-cf49989db-5brl6"
May 25 11:22:27.710872 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:22:27.710745    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/7b39dda4-21ee-4ee1-9a8b-d6042c965247-certificates\") pod \"keda-operator-ffbb595cb-d68rr\" (UID: \"7b39dda4-21ee-4ee1-9a8b-d6042c965247\") " pod="openshift-keda/keda-operator-ffbb595cb-d68rr"
May 25 11:22:27.710872 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:22:27.710806    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/a457cbd8-6fca-4e8e-b70d-f733a6564bcb-certificates\") pod \"keda-admission-cf49989db-5brl6\" (UID: \"a457cbd8-6fca-4e8e-b70d-f733a6564bcb\") " pod="openshift-keda/keda-admission-cf49989db-5brl6"
May 25 11:22:27.710985 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:22:27.710876    2561 secret.go:281] references non-existent secret key: ca.crt
May 25 11:22:27.710985 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:22:27.710895    2561 projected.go:277] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
May 25 11:22:27.710985 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:22:27.710905    2561 projected.go:194] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-ffbb595cb-d68rr: references non-existent secret key: ca.crt
May 25 11:22:27.710985 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:22:27.710956    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7b39dda4-21ee-4ee1-9a8b-d6042c965247-certificates podName:7b39dda4-21ee-4ee1-9a8b-d6042c965247 nodeName:}" failed. No retries permitted until 2026-05-25 11:22:28.710941788 +0000 UTC m=+322.256119389 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/7b39dda4-21ee-4ee1-9a8b-d6042c965247-certificates") pod "keda-operator-ffbb595cb-d68rr" (UID: "7b39dda4-21ee-4ee1-9a8b-d6042c965247") : references non-existent secret key: ca.crt
May 25 11:22:27.713261 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:22:27.713239    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/a457cbd8-6fca-4e8e-b70d-f733a6564bcb-certificates\") pod \"keda-admission-cf49989db-5brl6\" (UID: \"a457cbd8-6fca-4e8e-b70d-f733a6564bcb\") " pod="openshift-keda/keda-admission-cf49989db-5brl6"
May 25 11:22:27.721919 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:22:27.721899    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fd5nh\" (UniqueName: \"kubernetes.io/projected/a457cbd8-6fca-4e8e-b70d-f733a6564bcb-kube-api-access-fd5nh\") pod \"keda-admission-cf49989db-5brl6\" (UID: \"a457cbd8-6fca-4e8e-b70d-f733a6564bcb\") " pod="openshift-keda/keda-admission-cf49989db-5brl6"
May 25 11:22:27.837643 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:22:27.837607    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-cf49989db-5brl6"
May 25 11:22:27.972391 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:22:27.972366    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-cf49989db-5brl6"]
May 25 11:22:27.974827 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:22:27.974797    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda457cbd8_6fca_4e8e_b70d_f733a6564bcb.slice/crio-7c1258ed96c3ab08d74a6bc0f34c4202ccc482dd12ff25f983089b4a0aa0808e WatchSource:0}: Error finding container 7c1258ed96c3ab08d74a6bc0f34c4202ccc482dd12ff25f983089b4a0aa0808e: Status 404 returned error can't find the container with id 7c1258ed96c3ab08d74a6bc0f34c4202ccc482dd12ff25f983089b4a0aa0808e
May 25 11:22:27.976071 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:22:27.976055    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 25 11:22:28.013468 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:22:28.013442    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/9480b07c-7421-4c82-b7fd-a9a3ed4bbd40-certificates\") pod \"keda-metrics-apiserver-7c9f485588-6bnwh\" (UID: \"9480b07c-7421-4c82-b7fd-a9a3ed4bbd40\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-6bnwh"
May 25 11:22:28.015711 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:22:28.015693    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/9480b07c-7421-4c82-b7fd-a9a3ed4bbd40-certificates\") pod \"keda-metrics-apiserver-7c9f485588-6bnwh\" (UID: \"9480b07c-7421-4c82-b7fd-a9a3ed4bbd40\") " pod="openshift-keda/keda-metrics-apiserver-7c9f485588-6bnwh"
May 25 11:22:28.200749 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:22:28.200678    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-7c9f485588-6bnwh"
May 25 11:22:28.291198 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:22:28.291166    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-cf49989db-5brl6" event={"ID":"a457cbd8-6fca-4e8e-b70d-f733a6564bcb","Type":"ContainerStarted","Data":"7c1258ed96c3ab08d74a6bc0f34c4202ccc482dd12ff25f983089b4a0aa0808e"}
May 25 11:22:28.325588 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:22:28.325447    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-7c9f485588-6bnwh"]
May 25 11:22:28.328090 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:22:28.328063    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9480b07c_7421_4c82_b7fd_a9a3ed4bbd40.slice/crio-15608bfbbbe715464756b3ba7ed4e030a88c6d8eda14e73dcaa42afb19342b09 WatchSource:0}: Error finding container 15608bfbbbe715464756b3ba7ed4e030a88c6d8eda14e73dcaa42afb19342b09: Status 404 returned error can't find the container with id 15608bfbbbe715464756b3ba7ed4e030a88c6d8eda14e73dcaa42afb19342b09
May 25 11:22:28.720352 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:22:28.720312    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/7b39dda4-21ee-4ee1-9a8b-d6042c965247-certificates\") pod \"keda-operator-ffbb595cb-d68rr\" (UID: \"7b39dda4-21ee-4ee1-9a8b-d6042c965247\") " pod="openshift-keda/keda-operator-ffbb595cb-d68rr"
May 25 11:22:28.723339 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:22:28.723310    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/7b39dda4-21ee-4ee1-9a8b-d6042c965247-certificates\") pod \"keda-operator-ffbb595cb-d68rr\" (UID: \"7b39dda4-21ee-4ee1-9a8b-d6042c965247\") " pod="openshift-keda/keda-operator-ffbb595cb-d68rr"
May 25 11:22:28.851425 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:22:28.851387    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-ffbb595cb-d68rr"
May 25 11:22:29.015439 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:22:29.015412    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-ffbb595cb-d68rr"]
May 25 11:22:29.018607 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:22:29.018578    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b39dda4_21ee_4ee1_9a8b_d6042c965247.slice/crio-8e92bad084ee1b5a594836ff9b2c7f2413acf5b950f75a42fa4823757f459aa4 WatchSource:0}: Error finding container 8e92bad084ee1b5a594836ff9b2c7f2413acf5b950f75a42fa4823757f459aa4: Status 404 returned error can't find the container with id 8e92bad084ee1b5a594836ff9b2c7f2413acf5b950f75a42fa4823757f459aa4
May 25 11:22:29.298110 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:22:29.298016    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-ffbb595cb-d68rr" event={"ID":"7b39dda4-21ee-4ee1-9a8b-d6042c965247","Type":"ContainerStarted","Data":"8e92bad084ee1b5a594836ff9b2c7f2413acf5b950f75a42fa4823757f459aa4"}
May 25 11:22:29.299919 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:22:29.299861    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-7c9f485588-6bnwh" event={"ID":"9480b07c-7421-4c82-b7fd-a9a3ed4bbd40","Type":"ContainerStarted","Data":"15608bfbbbe715464756b3ba7ed4e030a88c6d8eda14e73dcaa42afb19342b09"}
May 25 11:22:31.309532 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:22:31.309489    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-cf49989db-5brl6" event={"ID":"a457cbd8-6fca-4e8e-b70d-f733a6564bcb","Type":"ContainerStarted","Data":"8e51b767476b4d40c9e19c31e2356e50353d943229cfa64ef56e10c15d5730eb"}
May 25 11:22:31.309965 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:22:31.309554    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-admission-cf49989db-5brl6"
May 25 11:22:31.330137 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:22:31.330093    2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-keda/keda-admission-cf49989db-5brl6" podStartSLOduration=1.760284368 podStartE2EDuration="4.330079771s" podCreationTimestamp="2026-05-25 11:22:27 +0000 UTC" firstStartedPulling="2026-05-25 11:22:27.976174308 +0000 UTC m=+321.521351888" lastFinishedPulling="2026-05-25 11:22:30.545969707 +0000 UTC m=+324.091147291" observedRunningTime="2026-05-25 11:22:31.328400384 +0000 UTC m=+324.873577988" watchObservedRunningTime="2026-05-25 11:22:31.330079771 +0000 UTC m=+324.875257373"
May 25 11:22:32.313243 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:22:32.313207    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-7c9f485588-6bnwh" event={"ID":"9480b07c-7421-4c82-b7fd-a9a3ed4bbd40","Type":"ContainerStarted","Data":"a15305ed42a023e7c5a93d8ddb53e4e3ad64c80ad7e01abf67518636c8206123"}
May 25 11:22:32.313728 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:22:32.313443    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-metrics-apiserver-7c9f485588-6bnwh"
May 25 11:22:32.331007 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:22:32.330951    2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-keda/keda-metrics-apiserver-7c9f485588-6bnwh" podStartSLOduration=1.903138594 podStartE2EDuration="5.330933023s" podCreationTimestamp="2026-05-25 11:22:27 +0000 UTC" firstStartedPulling="2026-05-25 11:22:28.329356248 +0000 UTC m=+321.874533832" lastFinishedPulling="2026-05-25 11:22:31.75715068 +0000 UTC m=+325.302328261" observedRunningTime="2026-05-25 11:22:32.330117345 +0000 UTC m=+325.875294950" watchObservedRunningTime="2026-05-25 11:22:32.330933023 +0000 UTC m=+325.876110627"
May 25 11:22:35.322935 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:22:35.322899    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-ffbb595cb-d68rr" event={"ID":"7b39dda4-21ee-4ee1-9a8b-d6042c965247","Type":"ContainerStarted","Data":"e4a8354cbaeaa06476e9a5c79adf6bf4f2e0ea9cdc7e5fa79f5f2b9b98c2c4ee"}
May 25 11:22:35.323315 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:22:35.323069    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-operator-ffbb595cb-d68rr"
May 25 11:22:35.352192 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:22:35.352146    2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-keda/keda-operator-ffbb595cb-d68rr" podStartSLOduration=3.028459113 podStartE2EDuration="8.352132858s" podCreationTimestamp="2026-05-25 11:22:27 +0000 UTC" firstStartedPulling="2026-05-25 11:22:29.020498994 +0000 UTC m=+322.565676577" lastFinishedPulling="2026-05-25 11:22:34.344172738 +0000 UTC m=+327.889350322" observedRunningTime="2026-05-25 11:22:35.350738625 +0000 UTC m=+328.895916227" watchObservedRunningTime="2026-05-25 11:22:35.352132858 +0000 UTC m=+328.897310462"
May 25 11:22:43.320183 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:22:43.320156    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-metrics-apiserver-7c9f485588-6bnwh"
May 25 11:22:52.315661 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:22:52.315630    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-admission-cf49989db-5brl6"
May 25 11:22:56.328804 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:22:56.328770    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-operator-ffbb595cb-d68rr"
May 25 11:23:33.245057 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:23:33.245028    2561 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-559cc6dd6f-p95h7"]
May 25 11:23:33.247072 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:23:33.247057    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-559cc6dd6f-p95h7"
May 25 11:23:33.249767 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:23:33.249747    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-tnnbt\""
May 25 11:23:33.250014 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:23:33.249998    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\""
May 25 11:23:33.250743 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:23:33.250724    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
May 25 11:23:33.250874 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:23:33.250745    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
May 25 11:23:33.259872 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:23:33.259850    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-559cc6dd6f-p95h7"]
May 25 11:23:33.284005 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:23:33.283982    2561 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-79f457656b-p72l9"]
May 25 11:23:33.285932 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:23:33.285918    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-79f457656b-p72l9"
May 25 11:23:33.288303 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:23:33.288286    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
May 25 11:23:33.288408 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:23:33.288311    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-g9pjq\""
May 25 11:23:33.297164 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:23:33.297143    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-79f457656b-p72l9"]
May 25 11:23:33.324576 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:23:33.324549    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxdvd\" (UniqueName: \"kubernetes.io/projected/efd241a1-f105-4fbb-96cd-6e196140a245-kube-api-access-sxdvd\") pod \"kserve-controller-manager-559cc6dd6f-p95h7\" (UID: \"efd241a1-f105-4fbb-96cd-6e196140a245\") " pod="kserve/kserve-controller-manager-559cc6dd6f-p95h7"
May 25 11:23:33.324682 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:23:33.324581    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtwtg\" (UniqueName: \"kubernetes.io/projected/e3cf2ca6-1c50-43b7-a6a3-7009420c743c-kube-api-access-wtwtg\") pod \"seaweedfs-79f457656b-p72l9\" (UID: \"e3cf2ca6-1c50-43b7-a6a3-7009420c743c\") " pod="kserve/seaweedfs-79f457656b-p72l9"
May 25 11:23:33.324682 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:23:33.324604    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/e3cf2ca6-1c50-43b7-a6a3-7009420c743c-data\") pod \"seaweedfs-79f457656b-p72l9\" (UID: \"e3cf2ca6-1c50-43b7-a6a3-7009420c743c\") " pod="kserve/seaweedfs-79f457656b-p72l9"
May 25 11:23:33.324768 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:23:33.324689    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/efd241a1-f105-4fbb-96cd-6e196140a245-cert\") pod \"kserve-controller-manager-559cc6dd6f-p95h7\" (UID: \"efd241a1-f105-4fbb-96cd-6e196140a245\") " pod="kserve/kserve-controller-manager-559cc6dd6f-p95h7"
May 25 11:23:33.425496 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:23:33.425475    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/efd241a1-f105-4fbb-96cd-6e196140a245-cert\") pod \"kserve-controller-manager-559cc6dd6f-p95h7\" (UID: \"efd241a1-f105-4fbb-96cd-6e196140a245\") " pod="kserve/kserve-controller-manager-559cc6dd6f-p95h7"
May 25 11:23:33.425594 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:23:33.425532    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-sxdvd\" (UniqueName: \"kubernetes.io/projected/efd241a1-f105-4fbb-96cd-6e196140a245-kube-api-access-sxdvd\") pod \"kserve-controller-manager-559cc6dd6f-p95h7\" (UID: \"efd241a1-f105-4fbb-96cd-6e196140a245\") " pod="kserve/kserve-controller-manager-559cc6dd6f-p95h7"
May 25 11:23:33.425594 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:23:33.425551    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-wtwtg\" (UniqueName: \"kubernetes.io/projected/e3cf2ca6-1c50-43b7-a6a3-7009420c743c-kube-api-access-wtwtg\") pod \"seaweedfs-79f457656b-p72l9\" (UID: \"e3cf2ca6-1c50-43b7-a6a3-7009420c743c\") " pod="kserve/seaweedfs-79f457656b-p72l9"
May 25 11:23:33.425594 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:23:33.425571    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/e3cf2ca6-1c50-43b7-a6a3-7009420c743c-data\") pod \"seaweedfs-79f457656b-p72l9\" (UID: \"e3cf2ca6-1c50-43b7-a6a3-7009420c743c\") " pod="kserve/seaweedfs-79f457656b-p72l9"
May 25 11:23:33.425751 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:23:33.425612    2561 secret.go:189] Couldn't get secret kserve/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found
May 25 11:23:33.425751 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:23:33.425663    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/efd241a1-f105-4fbb-96cd-6e196140a245-cert podName:efd241a1-f105-4fbb-96cd-6e196140a245 nodeName:}" failed. No retries permitted until 2026-05-25 11:23:33.925648755 +0000 UTC m=+387.470826336 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/efd241a1-f105-4fbb-96cd-6e196140a245-cert") pod "kserve-controller-manager-559cc6dd6f-p95h7" (UID: "efd241a1-f105-4fbb-96cd-6e196140a245") : secret "kserve-webhook-server-cert" not found
May 25 11:23:33.425863 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:23:33.425855    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/e3cf2ca6-1c50-43b7-a6a3-7009420c743c-data\") pod \"seaweedfs-79f457656b-p72l9\" (UID: \"e3cf2ca6-1c50-43b7-a6a3-7009420c743c\") " pod="kserve/seaweedfs-79f457656b-p72l9"
May 25 11:23:33.434156 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:23:33.434129    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtwtg\" (UniqueName: \"kubernetes.io/projected/e3cf2ca6-1c50-43b7-a6a3-7009420c743c-kube-api-access-wtwtg\") pod \"seaweedfs-79f457656b-p72l9\" (UID: \"e3cf2ca6-1c50-43b7-a6a3-7009420c743c\") " pod="kserve/seaweedfs-79f457656b-p72l9"
May 25 11:23:33.434241 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:23:33.434161    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxdvd\" (UniqueName: \"kubernetes.io/projected/efd241a1-f105-4fbb-96cd-6e196140a245-kube-api-access-sxdvd\") pod \"kserve-controller-manager-559cc6dd6f-p95h7\" (UID: \"efd241a1-f105-4fbb-96cd-6e196140a245\") " pod="kserve/kserve-controller-manager-559cc6dd6f-p95h7"
May 25 11:23:33.596016 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:23:33.595996    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-79f457656b-p72l9"
May 25 11:23:33.719217 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:23:33.719197    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-79f457656b-p72l9"]
May 25 11:23:33.722201 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:23:33.722173    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3cf2ca6_1c50_43b7_a6a3_7009420c743c.slice/crio-772e194e041abb264a91ff846eaa446ac5066ecccab04d0bb200ac1e8d8ef5b9 WatchSource:0}: Error finding container 772e194e041abb264a91ff846eaa446ac5066ecccab04d0bb200ac1e8d8ef5b9: Status 404 returned error can't find the container with id 772e194e041abb264a91ff846eaa446ac5066ecccab04d0bb200ac1e8d8ef5b9
May 25 11:23:33.929334 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:23:33.929232    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/efd241a1-f105-4fbb-96cd-6e196140a245-cert\") pod \"kserve-controller-manager-559cc6dd6f-p95h7\" (UID: \"efd241a1-f105-4fbb-96cd-6e196140a245\") " pod="kserve/kserve-controller-manager-559cc6dd6f-p95h7"
May 25 11:23:33.931496 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:23:33.931478    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/efd241a1-f105-4fbb-96cd-6e196140a245-cert\") pod \"kserve-controller-manager-559cc6dd6f-p95h7\" (UID: \"efd241a1-f105-4fbb-96cd-6e196140a245\") " pod="kserve/kserve-controller-manager-559cc6dd6f-p95h7"
May 25 11:23:34.157060 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:23:34.157035    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-559cc6dd6f-p95h7"
May 25 11:23:34.301110 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:23:34.300775    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-559cc6dd6f-p95h7"]
May 25 11:23:34.304024 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:23:34.303979    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefd241a1_f105_4fbb_96cd_6e196140a245.slice/crio-3317c08dfcf1007acb79ddfab2e4867c8cc834d5dadf8c8b85d143834fc03188 WatchSource:0}: Error finding container 3317c08dfcf1007acb79ddfab2e4867c8cc834d5dadf8c8b85d143834fc03188: Status 404 returned error can't find the container with id 3317c08dfcf1007acb79ddfab2e4867c8cc834d5dadf8c8b85d143834fc03188
May 25 11:23:34.501377 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:23:34.501305    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-79f457656b-p72l9" event={"ID":"e3cf2ca6-1c50-43b7-a6a3-7009420c743c","Type":"ContainerStarted","Data":"772e194e041abb264a91ff846eaa446ac5066ecccab04d0bb200ac1e8d8ef5b9"}
May 25 11:23:34.502399 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:23:34.502374    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-559cc6dd6f-p95h7" event={"ID":"efd241a1-f105-4fbb-96cd-6e196140a245","Type":"ContainerStarted","Data":"3317c08dfcf1007acb79ddfab2e4867c8cc834d5dadf8c8b85d143834fc03188"}
May 25 11:23:38.516258 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:23:38.516220    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-79f457656b-p72l9" event={"ID":"e3cf2ca6-1c50-43b7-a6a3-7009420c743c","Type":"ContainerStarted","Data":"0e5db5a8c309de15c58127351fa167bd28202e26737c8c30f6f6d6d30780c633"}
May 25 11:23:38.516258 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:23:38.516290    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-79f457656b-p72l9"
May 25 11:23:38.517582 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:23:38.517558    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-559cc6dd6f-p95h7" event={"ID":"efd241a1-f105-4fbb-96cd-6e196140a245","Type":"ContainerStarted","Data":"1722e31a8c5bf20eb64e8062747bec5330eee5b2faf79feb1eb3a4736655beb4"}
May 25 11:23:38.517693 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:23:38.517636    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-559cc6dd6f-p95h7"
May 25 11:23:38.533276 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:23:38.533235    2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve/seaweedfs-79f457656b-p72l9" podStartSLOduration=1.500387984 podStartE2EDuration="5.533224403s" podCreationTimestamp="2026-05-25 11:23:33 +0000 UTC" firstStartedPulling="2026-05-25 11:23:33.723801116 +0000 UTC m=+387.268978696" lastFinishedPulling="2026-05-25 11:23:37.756637531 +0000 UTC m=+391.301815115" observedRunningTime="2026-05-25 11:23:38.53236527 +0000 UTC m=+392.077542873" watchObservedRunningTime="2026-05-25 11:23:38.533224403 +0000 UTC m=+392.078402005"
May 25 11:23:38.548529 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:23:38.548493    2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve/kserve-controller-manager-559cc6dd6f-p95h7" podStartSLOduration=2.184568212 podStartE2EDuration="5.548482313s" podCreationTimestamp="2026-05-25 11:23:33 +0000 UTC" firstStartedPulling="2026-05-25 11:23:34.305428181 +0000 UTC m=+387.850605762" lastFinishedPulling="2026-05-25 11:23:37.669342282 +0000 UTC m=+391.214519863" observedRunningTime="2026-05-25 11:23:38.546670988 +0000 UTC m=+392.091848604" watchObservedRunningTime="2026-05-25 11:23:38.548482313 +0000 UTC m=+392.093659916"
May 25 11:23:44.523353 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:23:44.523318    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-79f457656b-p72l9"
May 25 11:24:09.178531 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:24:09.178502    2561 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-559cc6dd6f-p95h7"]
May 25 11:24:09.178982 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:24:09.178716    2561 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve/kserve-controller-manager-559cc6dd6f-p95h7" podUID="efd241a1-f105-4fbb-96cd-6e196140a245" containerName="manager" containerID="cri-o://1722e31a8c5bf20eb64e8062747bec5330eee5b2faf79feb1eb3a4736655beb4" gracePeriod=10
May 25 11:24:09.183611 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:24:09.183588    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-559cc6dd6f-p95h7"
May 25 11:24:09.247412 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:24:09.247378    2561 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-559cc6dd6f-bksd2"]
May 25 11:24:09.251493 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:24:09.251474    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-559cc6dd6f-bksd2"
May 25 11:24:09.267015 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:24:09.266989    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-559cc6dd6f-bksd2"]
May 25 11:24:09.405089 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:24:09.405064    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbfrn\" (UniqueName: \"kubernetes.io/projected/199e7c1d-de93-4bcb-98b9-5d91f052f6d4-kube-api-access-rbfrn\") pod \"kserve-controller-manager-559cc6dd6f-bksd2\" (UID: \"199e7c1d-de93-4bcb-98b9-5d91f052f6d4\") " pod="kserve/kserve-controller-manager-559cc6dd6f-bksd2"
May 25 11:24:09.405207 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:24:09.405156    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/199e7c1d-de93-4bcb-98b9-5d91f052f6d4-cert\") pod \"kserve-controller-manager-559cc6dd6f-bksd2\" (UID: \"199e7c1d-de93-4bcb-98b9-5d91f052f6d4\") " pod="kserve/kserve-controller-manager-559cc6dd6f-bksd2"
May 25 11:24:09.412913 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:24:09.412889    2561 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-559cc6dd6f-p95h7"
May 25 11:24:09.506527 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:24:09.506464    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/efd241a1-f105-4fbb-96cd-6e196140a245-cert\") pod \"efd241a1-f105-4fbb-96cd-6e196140a245\" (UID: \"efd241a1-f105-4fbb-96cd-6e196140a245\") "
May 25 11:24:09.506527 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:24:09.506499    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sxdvd\" (UniqueName: \"kubernetes.io/projected/efd241a1-f105-4fbb-96cd-6e196140a245-kube-api-access-sxdvd\") pod \"efd241a1-f105-4fbb-96cd-6e196140a245\" (UID: \"efd241a1-f105-4fbb-96cd-6e196140a245\") "
May 25 11:24:09.506718 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:24:09.506640    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-rbfrn\" (UniqueName: \"kubernetes.io/projected/199e7c1d-de93-4bcb-98b9-5d91f052f6d4-kube-api-access-rbfrn\") pod \"kserve-controller-manager-559cc6dd6f-bksd2\" (UID: \"199e7c1d-de93-4bcb-98b9-5d91f052f6d4\") " pod="kserve/kserve-controller-manager-559cc6dd6f-bksd2"
May 25 11:24:09.506718 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:24:09.506698    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/199e7c1d-de93-4bcb-98b9-5d91f052f6d4-cert\") pod \"kserve-controller-manager-559cc6dd6f-bksd2\" (UID: \"199e7c1d-de93-4bcb-98b9-5d91f052f6d4\") " pod="kserve/kserve-controller-manager-559cc6dd6f-bksd2"
May 25 11:24:09.508588 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:24:09.508560    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/efd241a1-f105-4fbb-96cd-6e196140a245-cert" (OuterVolumeSpecName: "cert") pod "efd241a1-f105-4fbb-96cd-6e196140a245" (UID: "efd241a1-f105-4fbb-96cd-6e196140a245"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 11:24:09.508691 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:24:09.508632    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efd241a1-f105-4fbb-96cd-6e196140a245-kube-api-access-sxdvd" (OuterVolumeSpecName: "kube-api-access-sxdvd") pod "efd241a1-f105-4fbb-96cd-6e196140a245" (UID: "efd241a1-f105-4fbb-96cd-6e196140a245"). InnerVolumeSpecName "kube-api-access-sxdvd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 11:24:09.508891 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:24:09.508874    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/199e7c1d-de93-4bcb-98b9-5d91f052f6d4-cert\") pod \"kserve-controller-manager-559cc6dd6f-bksd2\" (UID: \"199e7c1d-de93-4bcb-98b9-5d91f052f6d4\") " pod="kserve/kserve-controller-manager-559cc6dd6f-bksd2"
May 25 11:24:09.516185 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:24:09.516162    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbfrn\" (UniqueName: \"kubernetes.io/projected/199e7c1d-de93-4bcb-98b9-5d91f052f6d4-kube-api-access-rbfrn\") pod \"kserve-controller-manager-559cc6dd6f-bksd2\" (UID: \"199e7c1d-de93-4bcb-98b9-5d91f052f6d4\") " pod="kserve/kserve-controller-manager-559cc6dd6f-bksd2"
May 25 11:24:09.567790 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:24:09.567770    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-559cc6dd6f-bksd2"
May 25 11:24:09.607456 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:24:09.607432    2561 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/efd241a1-f105-4fbb-96cd-6e196140a245-cert\") on node \"ip-10-0-136-227.ec2.internal\" DevicePath \"\""
May 25 11:24:09.607456 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:24:09.607455    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sxdvd\" (UniqueName: \"kubernetes.io/projected/efd241a1-f105-4fbb-96cd-6e196140a245-kube-api-access-sxdvd\") on node \"ip-10-0-136-227.ec2.internal\" DevicePath \"\""
May 25 11:24:09.614092 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:24:09.614065    2561 generic.go:358] "Generic (PLEG): container finished" podID="efd241a1-f105-4fbb-96cd-6e196140a245" containerID="1722e31a8c5bf20eb64e8062747bec5330eee5b2faf79feb1eb3a4736655beb4" exitCode=0
May 25 11:24:09.614209 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:24:09.614189    2561 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-559cc6dd6f-p95h7"
May 25 11:24:09.614300 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:24:09.614185    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-559cc6dd6f-p95h7" event={"ID":"efd241a1-f105-4fbb-96cd-6e196140a245","Type":"ContainerDied","Data":"1722e31a8c5bf20eb64e8062747bec5330eee5b2faf79feb1eb3a4736655beb4"}
May 25 11:24:09.614346 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:24:09.614324    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-559cc6dd6f-p95h7" event={"ID":"efd241a1-f105-4fbb-96cd-6e196140a245","Type":"ContainerDied","Data":"3317c08dfcf1007acb79ddfab2e4867c8cc834d5dadf8c8b85d143834fc03188"}
May 25 11:24:09.614389 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:24:09.614361    2561 scope.go:117] "RemoveContainer" containerID="1722e31a8c5bf20eb64e8062747bec5330eee5b2faf79feb1eb3a4736655beb4"
May 25 11:24:09.624925 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:24:09.624909    2561 scope.go:117] "RemoveContainer" containerID="1722e31a8c5bf20eb64e8062747bec5330eee5b2faf79feb1eb3a4736655beb4"
May 25 11:24:09.625977 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:24:09.625309    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1722e31a8c5bf20eb64e8062747bec5330eee5b2faf79feb1eb3a4736655beb4\": container with ID starting with 1722e31a8c5bf20eb64e8062747bec5330eee5b2faf79feb1eb3a4736655beb4 not found: ID does not exist" containerID="1722e31a8c5bf20eb64e8062747bec5330eee5b2faf79feb1eb3a4736655beb4"
May 25 11:24:09.625977 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:24:09.625347    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1722e31a8c5bf20eb64e8062747bec5330eee5b2faf79feb1eb3a4736655beb4"} err="failed to get container status \"1722e31a8c5bf20eb64e8062747bec5330eee5b2faf79feb1eb3a4736655beb4\": rpc error: code = NotFound desc = could not find container \"1722e31a8c5bf20eb64e8062747bec5330eee5b2faf79feb1eb3a4736655beb4\": container with ID starting with 1722e31a8c5bf20eb64e8062747bec5330eee5b2faf79feb1eb3a4736655beb4 not found: ID does not exist"
May 25 11:24:09.642840 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:24:09.642818    2561 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-559cc6dd6f-p95h7"]
May 25 11:24:09.646384 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:24:09.646362    2561 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve/kserve-controller-manager-559cc6dd6f-p95h7"]
May 25 11:24:09.711067 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:24:09.711043    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-559cc6dd6f-bksd2"]
May 25 11:24:09.713858 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:24:09.713833    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod199e7c1d_de93_4bcb_98b9_5d91f052f6d4.slice/crio-8d18140a445188065e0b4045c177f99b97f43f14957f9ae4e3d70cdabd4d7625 WatchSource:0}: Error finding container 8d18140a445188065e0b4045c177f99b97f43f14957f9ae4e3d70cdabd4d7625: Status 404 returned error can't find the container with id 8d18140a445188065e0b4045c177f99b97f43f14957f9ae4e3d70cdabd4d7625
May 25 11:24:10.618209 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:24:10.618174    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-559cc6dd6f-bksd2" event={"ID":"199e7c1d-de93-4bcb-98b9-5d91f052f6d4","Type":"ContainerStarted","Data":"c8ff0d7d773de65c6e0ca9e9df21d289b5d92f3a9db70c8c2d2295084eace4de"}
May 25 11:24:10.618209 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:24:10.618210    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-559cc6dd6f-bksd2" event={"ID":"199e7c1d-de93-4bcb-98b9-5d91f052f6d4","Type":"ContainerStarted","Data":"8d18140a445188065e0b4045c177f99b97f43f14957f9ae4e3d70cdabd4d7625"}
May 25 11:24:10.618716 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:24:10.618290    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-559cc6dd6f-bksd2"
May 25 11:24:10.637819 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:24:10.637762    2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve/kserve-controller-manager-559cc6dd6f-bksd2" podStartSLOduration=1.042377967 podStartE2EDuration="1.6377492s" podCreationTimestamp="2026-05-25 11:24:09 +0000 UTC" firstStartedPulling="2026-05-25 11:24:09.715204365 +0000 UTC m=+423.260381946" lastFinishedPulling="2026-05-25 11:24:10.310575598 +0000 UTC m=+423.855753179" observedRunningTime="2026-05-25 11:24:10.635499391 +0000 UTC m=+424.180677005" watchObservedRunningTime="2026-05-25 11:24:10.6377492 +0000 UTC m=+424.182926802"
May 25 11:24:11.116973 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:24:11.116942    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="efd241a1-f105-4fbb-96cd-6e196140a245" path="/var/lib/kubelet/pods/efd241a1-f105-4fbb-96cd-6e196140a245/volumes"
May 25 11:24:41.627616 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:24:41.627531    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-559cc6dd6f-bksd2"
May 25 11:24:43.111399 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:24:43.111371    2561 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-696fc77849-8hh2g"]
May 25 11:24:43.111769 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:24:43.111700    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="efd241a1-f105-4fbb-96cd-6e196140a245" containerName="manager"
May 25 11:24:43.111769 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:24:43.111710    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="efd241a1-f105-4fbb-96cd-6e196140a245" containerName="manager"
May 25 11:24:43.111841 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:24:43.111776    2561 memory_manager.go:356] "RemoveStaleState removing state" podUID="efd241a1-f105-4fbb-96cd-6e196140a245" containerName="manager"
May 25 11:24:43.115017 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:24:43.114994    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-696fc77849-8hh2g"
May 25 11:24:43.117798 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:24:43.117779    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
May 25 11:24:43.117925 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:24:43.117844    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-dq5l6\""
May 25 11:24:43.126008 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:24:43.125986    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-696fc77849-8hh2g"]
May 25 11:24:43.157968 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:24:43.157936    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6flzt\" (UniqueName: \"kubernetes.io/projected/43d9a307-9385-49d0-b3cf-258c9ffdc164-kube-api-access-6flzt\") pod \"odh-model-controller-696fc77849-8hh2g\" (UID: \"43d9a307-9385-49d0-b3cf-258c9ffdc164\") " pod="kserve/odh-model-controller-696fc77849-8hh2g"
May 25 11:24:43.158128 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:24:43.158033    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/43d9a307-9385-49d0-b3cf-258c9ffdc164-cert\") pod \"odh-model-controller-696fc77849-8hh2g\" (UID: \"43d9a307-9385-49d0-b3cf-258c9ffdc164\") " pod="kserve/odh-model-controller-696fc77849-8hh2g"
May 25 11:24:43.258981 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:24:43.258949    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-6flzt\" (UniqueName: \"kubernetes.io/projected/43d9a307-9385-49d0-b3cf-258c9ffdc164-kube-api-access-6flzt\") pod \"odh-model-controller-696fc77849-8hh2g\" (UID: \"43d9a307-9385-49d0-b3cf-258c9ffdc164\") " pod="kserve/odh-model-controller-696fc77849-8hh2g"
May 25 11:24:43.259144 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:24:43.259004    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/43d9a307-9385-49d0-b3cf-258c9ffdc164-cert\") pod \"odh-model-controller-696fc77849-8hh2g\" (UID: \"43d9a307-9385-49d0-b3cf-258c9ffdc164\") " pod="kserve/odh-model-controller-696fc77849-8hh2g"
May 25 11:24:43.261421 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:24:43.261379    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/43d9a307-9385-49d0-b3cf-258c9ffdc164-cert\") pod \"odh-model-controller-696fc77849-8hh2g\" (UID: \"43d9a307-9385-49d0-b3cf-258c9ffdc164\") " pod="kserve/odh-model-controller-696fc77849-8hh2g"
May 25 11:24:43.270495 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:24:43.270477    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6flzt\" (UniqueName: \"kubernetes.io/projected/43d9a307-9385-49d0-b3cf-258c9ffdc164-kube-api-access-6flzt\") pod \"odh-model-controller-696fc77849-8hh2g\" (UID: \"43d9a307-9385-49d0-b3cf-258c9ffdc164\") " pod="kserve/odh-model-controller-696fc77849-8hh2g"
May 25 11:24:43.425872 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:24:43.425784    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-696fc77849-8hh2g"
May 25 11:24:43.550902 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:24:43.550872    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-696fc77849-8hh2g"]
May 25 11:24:43.553851 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:24:43.553822    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43d9a307_9385_49d0_b3cf_258c9ffdc164.slice/crio-c88421a940a2cf23684adffcde94dcb059af614c462c5323e8ac9025050c5328 WatchSource:0}: Error finding container c88421a940a2cf23684adffcde94dcb059af614c462c5323e8ac9025050c5328: Status 404 returned error can't find the container with id c88421a940a2cf23684adffcde94dcb059af614c462c5323e8ac9025050c5328
May 25 11:24:43.719751 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:24:43.719656    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-696fc77849-8hh2g" event={"ID":"43d9a307-9385-49d0-b3cf-258c9ffdc164","Type":"ContainerStarted","Data":"c88421a940a2cf23684adffcde94dcb059af614c462c5323e8ac9025050c5328"}
May 25 11:24:46.732805 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:24:46.732770    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-696fc77849-8hh2g" event={"ID":"43d9a307-9385-49d0-b3cf-258c9ffdc164","Type":"ContainerStarted","Data":"229cedd01f2582d3653b85738d2233877a1f50b87f563849b8e069bfc965b873"}
May 25 11:24:46.733164 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:24:46.732878    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-696fc77849-8hh2g"
May 25 11:24:46.747758 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:24:46.747560    2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve/odh-model-controller-696fc77849-8hh2g" podStartSLOduration=1.3197732420000001 podStartE2EDuration="3.747546849s" podCreationTimestamp="2026-05-25 11:24:43 +0000 UTC" firstStartedPulling="2026-05-25 11:24:43.555069197 +0000 UTC m=+457.100246778" lastFinishedPulling="2026-05-25 11:24:45.982842804 +0000 UTC m=+459.528020385" observedRunningTime="2026-05-25 11:24:46.747460195 +0000 UTC m=+460.292637800" watchObservedRunningTime="2026-05-25 11:24:46.747546849 +0000 UTC m=+460.292724452"
May 25 11:24:57.740852 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:24:57.740814    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-696fc77849-8hh2g"
May 25 11:25:09.806212 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:25:09.806181    2561 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-tls-custom-ddd4dbfd-lssrh"]
May 25 11:25:09.809694 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:25:09.809674    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-ddd4dbfd-lssrh"
May 25 11:25:09.812677 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:25:09.812655    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom-artifact\""
May 25 11:25:09.817872 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:25:09.817847    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-ddd4dbfd-lssrh"]
May 25 11:25:09.985325 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:25:09.985291    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xk4lt\" (UniqueName: \"kubernetes.io/projected/a8e0fb2a-7a5e-44d9-93f9-a4570e76381b-kube-api-access-xk4lt\") pod \"seaweedfs-tls-custom-ddd4dbfd-lssrh\" (UID: \"a8e0fb2a-7a5e-44d9-93f9-a4570e76381b\") " pod="kserve/seaweedfs-tls-custom-ddd4dbfd-lssrh"
May 25 11:25:09.985473 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:25:09.985336    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/a8e0fb2a-7a5e-44d9-93f9-a4570e76381b-data\") pod \"seaweedfs-tls-custom-ddd4dbfd-lssrh\" (UID: \"a8e0fb2a-7a5e-44d9-93f9-a4570e76381b\") " pod="kserve/seaweedfs-tls-custom-ddd4dbfd-lssrh"
May 25 11:25:10.085991 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:25:10.085967    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-xk4lt\" (UniqueName: \"kubernetes.io/projected/a8e0fb2a-7a5e-44d9-93f9-a4570e76381b-kube-api-access-xk4lt\") pod \"seaweedfs-tls-custom-ddd4dbfd-lssrh\" (UID: \"a8e0fb2a-7a5e-44d9-93f9-a4570e76381b\") " pod="kserve/seaweedfs-tls-custom-ddd4dbfd-lssrh"
May 25 11:25:10.086109 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:25:10.086004    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/a8e0fb2a-7a5e-44d9-93f9-a4570e76381b-data\") pod \"seaweedfs-tls-custom-ddd4dbfd-lssrh\" (UID: \"a8e0fb2a-7a5e-44d9-93f9-a4570e76381b\") " pod="kserve/seaweedfs-tls-custom-ddd4dbfd-lssrh"
May 25 11:25:10.086349 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:25:10.086335    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/a8e0fb2a-7a5e-44d9-93f9-a4570e76381b-data\") pod \"seaweedfs-tls-custom-ddd4dbfd-lssrh\" (UID: \"a8e0fb2a-7a5e-44d9-93f9-a4570e76381b\") " pod="kserve/seaweedfs-tls-custom-ddd4dbfd-lssrh"
May 25 11:25:10.096527 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:25:10.096508    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xk4lt\" (UniqueName: \"kubernetes.io/projected/a8e0fb2a-7a5e-44d9-93f9-a4570e76381b-kube-api-access-xk4lt\") pod \"seaweedfs-tls-custom-ddd4dbfd-lssrh\" (UID: \"a8e0fb2a-7a5e-44d9-93f9-a4570e76381b\") " pod="kserve/seaweedfs-tls-custom-ddd4dbfd-lssrh"
May 25 11:25:10.118535 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:25:10.118515    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-ddd4dbfd-lssrh"
May 25 11:25:10.243726 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:25:10.243696    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-ddd4dbfd-lssrh"]
May 25 11:25:10.248332 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:25:10.248295    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8e0fb2a_7a5e_44d9_93f9_a4570e76381b.slice/crio-c156dfcd8433aa27748ad74a6a668344dd6c4035f3ad26bc523265874f1d1b85 WatchSource:0}: Error finding container c156dfcd8433aa27748ad74a6a668344dd6c4035f3ad26bc523265874f1d1b85: Status 404 returned error can't find the container with id c156dfcd8433aa27748ad74a6a668344dd6c4035f3ad26bc523265874f1d1b85
May 25 11:25:10.812183 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:25:10.812082    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-ddd4dbfd-lssrh" event={"ID":"a8e0fb2a-7a5e-44d9-93f9-a4570e76381b","Type":"ContainerStarted","Data":"16b307c83a25db92a6e8158815c012c169d76c8cf0977dc11cf1f642c81db226"}
May 25 11:25:10.812183 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:25:10.812125    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-ddd4dbfd-lssrh" event={"ID":"a8e0fb2a-7a5e-44d9-93f9-a4570e76381b","Type":"ContainerStarted","Data":"c156dfcd8433aa27748ad74a6a668344dd6c4035f3ad26bc523265874f1d1b85"}
May 25 11:25:12.232075 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:25:12.232026    2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve/seaweedfs-tls-custom-ddd4dbfd-lssrh" podStartSLOduration=2.955157472 podStartE2EDuration="3.232009926s" podCreationTimestamp="2026-05-25 11:25:09 +0000 UTC" firstStartedPulling="2026-05-25 11:25:10.249916905 +0000 UTC m=+483.795094485" lastFinishedPulling="2026-05-25 11:25:10.526769357 +0000 UTC m=+484.071946939" observedRunningTime="2026-05-25 11:25:10.836429252 +0000 UTC m=+484.381606855" watchObservedRunningTime="2026-05-25 11:25:12.232009926 +0000 UTC m=+485.777187528"
May 25 11:25:12.232997 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:25:12.232979    2561 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve/seaweedfs-tls-custom-ddd4dbfd-lssrh"]
May 25 11:25:12.819329 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:25:12.819293    2561 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve/seaweedfs-tls-custom-ddd4dbfd-lssrh" podUID="a8e0fb2a-7a5e-44d9-93f9-a4570e76381b" containerName="seaweedfs-tls-custom" containerID="cri-o://16b307c83a25db92a6e8158815c012c169d76c8cf0977dc11cf1f642c81db226" gracePeriod=30
May 25 11:25:14.072214 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:25:14.072189    2561 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-ddd4dbfd-lssrh"
May 25 11:25:14.216526 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:25:14.216463    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/a8e0fb2a-7a5e-44d9-93f9-a4570e76381b-data\") pod \"a8e0fb2a-7a5e-44d9-93f9-a4570e76381b\" (UID: \"a8e0fb2a-7a5e-44d9-93f9-a4570e76381b\") "
May 25 11:25:14.216526 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:25:14.216504    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xk4lt\" (UniqueName: \"kubernetes.io/projected/a8e0fb2a-7a5e-44d9-93f9-a4570e76381b-kube-api-access-xk4lt\") pod \"a8e0fb2a-7a5e-44d9-93f9-a4570e76381b\" (UID: \"a8e0fb2a-7a5e-44d9-93f9-a4570e76381b\") "
May 25 11:25:14.217668 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:25:14.217639    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8e0fb2a-7a5e-44d9-93f9-a4570e76381b-data" (OuterVolumeSpecName: "data") pod "a8e0fb2a-7a5e-44d9-93f9-a4570e76381b" (UID: "a8e0fb2a-7a5e-44d9-93f9-a4570e76381b"). InnerVolumeSpecName "data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:25:14.218441 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:25:14.218422    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8e0fb2a-7a5e-44d9-93f9-a4570e76381b-kube-api-access-xk4lt" (OuterVolumeSpecName: "kube-api-access-xk4lt") pod "a8e0fb2a-7a5e-44d9-93f9-a4570e76381b" (UID: "a8e0fb2a-7a5e-44d9-93f9-a4570e76381b"). InnerVolumeSpecName "kube-api-access-xk4lt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 11:25:14.317590 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:25:14.317568    2561 reconciler_common.go:299] "Volume detached for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/a8e0fb2a-7a5e-44d9-93f9-a4570e76381b-data\") on node \"ip-10-0-136-227.ec2.internal\" DevicePath \"\""
May 25 11:25:14.317590 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:25:14.317588    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xk4lt\" (UniqueName: \"kubernetes.io/projected/a8e0fb2a-7a5e-44d9-93f9-a4570e76381b-kube-api-access-xk4lt\") on node \"ip-10-0-136-227.ec2.internal\" DevicePath \"\""
May 25 11:25:14.640638 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:25:14.640606    2561 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console/console-78c764b57c-98fm8"]
May 25 11:25:14.641091 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:25:14.641070    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8e0fb2a-7a5e-44d9-93f9-a4570e76381b" containerName="seaweedfs-tls-custom"
May 25 11:25:14.641145 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:25:14.641095    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8e0fb2a-7a5e-44d9-93f9-a4570e76381b" containerName="seaweedfs-tls-custom"
May 25 11:25:14.641190 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:25:14.641162    2561 memory_manager.go:356] "RemoveStaleState removing state" podUID="a8e0fb2a-7a5e-44d9-93f9-a4570e76381b" containerName="seaweedfs-tls-custom"
May 25 11:25:14.645593 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:25:14.645571    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-78c764b57c-98fm8"
May 25 11:25:14.648182 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:25:14.648157    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
May 25 11:25:14.648314 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:25:14.648294    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
May 25 11:25:14.649495 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:25:14.649305    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
May 25 11:25:14.649495 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:25:14.649327    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
May 25 11:25:14.649495 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:25:14.649335    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
May 25 11:25:14.649495 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:25:14.649367    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
May 25 11:25:14.649765 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:25:14.649644    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-wkb8g\""
May 25 11:25:14.649765 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:25:14.649674    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
May 25 11:25:14.655186 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:25:14.655166    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
May 25 11:25:14.660790 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:25:14.660766    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-78c764b57c-98fm8"]
May 25 11:25:14.821629 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:25:14.821587    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/db40b82d-e321-4af8-925c-710fff638c2d-trusted-ca-bundle\") pod \"console-78c764b57c-98fm8\" (UID: \"db40b82d-e321-4af8-925c-710fff638c2d\") " pod="openshift-console/console-78c764b57c-98fm8"
May 25 11:25:14.821629 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:25:14.821633    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/db40b82d-e321-4af8-925c-710fff638c2d-console-serving-cert\") pod \"console-78c764b57c-98fm8\" (UID: \"db40b82d-e321-4af8-925c-710fff638c2d\") " pod="openshift-console/console-78c764b57c-98fm8"
May 25 11:25:14.821829 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:25:14.821652    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pj2cq\" (UniqueName: \"kubernetes.io/projected/db40b82d-e321-4af8-925c-710fff638c2d-kube-api-access-pj2cq\") pod \"console-78c764b57c-98fm8\" (UID: \"db40b82d-e321-4af8-925c-710fff638c2d\") " pod="openshift-console/console-78c764b57c-98fm8"
May 25 11:25:14.821829 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:25:14.821757    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/db40b82d-e321-4af8-925c-710fff638c2d-console-config\") pod \"console-78c764b57c-98fm8\" (UID: \"db40b82d-e321-4af8-925c-710fff638c2d\") " pod="openshift-console/console-78c764b57c-98fm8"
May 25 11:25:14.821829 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:25:14.821791    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/db40b82d-e321-4af8-925c-710fff638c2d-oauth-serving-cert\") pod \"console-78c764b57c-98fm8\" (UID: \"db40b82d-e321-4af8-925c-710fff638c2d\") " pod="openshift-console/console-78c764b57c-98fm8"
May 25 11:25:14.821829 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:25:14.821822    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/db40b82d-e321-4af8-925c-710fff638c2d-service-ca\") pod \"console-78c764b57c-98fm8\" (UID: \"db40b82d-e321-4af8-925c-710fff638c2d\") " pod="openshift-console/console-78c764b57c-98fm8"
May 25 11:25:14.821968 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:25:14.821885    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/db40b82d-e321-4af8-925c-710fff638c2d-console-oauth-config\") pod \"console-78c764b57c-98fm8\" (UID: \"db40b82d-e321-4af8-925c-710fff638c2d\") " pod="openshift-console/console-78c764b57c-98fm8"
May 25 11:25:14.825321 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:25:14.825295    2561 generic.go:358] "Generic (PLEG): container finished" podID="a8e0fb2a-7a5e-44d9-93f9-a4570e76381b" containerID="16b307c83a25db92a6e8158815c012c169d76c8cf0977dc11cf1f642c81db226" exitCode=0
May 25 11:25:14.825446 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:25:14.825353    2561 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-ddd4dbfd-lssrh"
May 25 11:25:14.825446 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:25:14.825380    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-ddd4dbfd-lssrh" event={"ID":"a8e0fb2a-7a5e-44d9-93f9-a4570e76381b","Type":"ContainerDied","Data":"16b307c83a25db92a6e8158815c012c169d76c8cf0977dc11cf1f642c81db226"}
May 25 11:25:14.825446 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:25:14.825417    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-ddd4dbfd-lssrh" event={"ID":"a8e0fb2a-7a5e-44d9-93f9-a4570e76381b","Type":"ContainerDied","Data":"c156dfcd8433aa27748ad74a6a668344dd6c4035f3ad26bc523265874f1d1b85"}
May 25 11:25:14.825446 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:25:14.825434    2561 scope.go:117] "RemoveContainer" containerID="16b307c83a25db92a6e8158815c012c169d76c8cf0977dc11cf1f642c81db226"
May 25 11:25:14.835006 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:25:14.834992    2561 scope.go:117] "RemoveContainer" containerID="16b307c83a25db92a6e8158815c012c169d76c8cf0977dc11cf1f642c81db226"
May 25 11:25:14.835227 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:25:14.835209    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16b307c83a25db92a6e8158815c012c169d76c8cf0977dc11cf1f642c81db226\": container with ID starting with 16b307c83a25db92a6e8158815c012c169d76c8cf0977dc11cf1f642c81db226 not found: ID does not exist" containerID="16b307c83a25db92a6e8158815c012c169d76c8cf0977dc11cf1f642c81db226"
May 25 11:25:14.835284 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:25:14.835233    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16b307c83a25db92a6e8158815c012c169d76c8cf0977dc11cf1f642c81db226"} err="failed to get container status \"16b307c83a25db92a6e8158815c012c169d76c8cf0977dc11cf1f642c81db226\": rpc error: code = NotFound desc = could not find container \"16b307c83a25db92a6e8158815c012c169d76c8cf0977dc11cf1f642c81db226\": container with ID starting with 16b307c83a25db92a6e8158815c012c169d76c8cf0977dc11cf1f642c81db226 not found: ID does not exist"
May 25 11:25:14.849536 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:25:14.849509    2561 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve/seaweedfs-tls-custom-ddd4dbfd-lssrh"]
May 25 11:25:14.853970 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:25:14.853947    2561 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve/seaweedfs-tls-custom-ddd4dbfd-lssrh"]
May 25 11:25:14.888156 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:25:14.888126    2561 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-tls-custom-5c88b85bb7-mhx2w"]
May 25 11:25:14.892687 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:25:14.892638    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-5c88b85bb7-mhx2w"
May 25 11:25:14.895104 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:25:14.895085    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom-artifact\""
May 25 11:25:14.895204 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:25:14.895088    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom\""
May 25 11:25:14.900445 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:25:14.900422    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-5c88b85bb7-mhx2w"]
May 25 11:25:14.922781 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:25:14.922755    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/db40b82d-e321-4af8-925c-710fff638c2d-console-config\") pod \"console-78c764b57c-98fm8\" (UID: \"db40b82d-e321-4af8-925c-710fff638c2d\") " pod="openshift-console/console-78c764b57c-98fm8"
May 25 11:25:14.922910 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:25:14.922792    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/db40b82d-e321-4af8-925c-710fff638c2d-oauth-serving-cert\") pod \"console-78c764b57c-98fm8\" (UID: \"db40b82d-e321-4af8-925c-710fff638c2d\") " pod="openshift-console/console-78c764b57c-98fm8"
May 25 11:25:14.922910 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:25:14.922810    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/db40b82d-e321-4af8-925c-710fff638c2d-service-ca\") pod \"console-78c764b57c-98fm8\" (UID: \"db40b82d-e321-4af8-925c-710fff638c2d\") " pod="openshift-console/console-78c764b57c-98fm8"
May 25 11:25:14.922910 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:25:14.922836    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/db40b82d-e321-4af8-925c-710fff638c2d-console-oauth-config\") pod \"console-78c764b57c-98fm8\" (UID: \"db40b82d-e321-4af8-925c-710fff638c2d\") " pod="openshift-console/console-78c764b57c-98fm8"
May 25 11:25:14.922910 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:25:14.922875    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/db40b82d-e321-4af8-925c-710fff638c2d-trusted-ca-bundle\") pod \"console-78c764b57c-98fm8\" (UID: \"db40b82d-e321-4af8-925c-710fff638c2d\") " pod="openshift-console/console-78c764b57c-98fm8"
May 25 11:25:14.922910 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:25:14.922899    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/db40b82d-e321-4af8-925c-710fff638c2d-console-serving-cert\") pod \"console-78c764b57c-98fm8\" (UID: \"db40b82d-e321-4af8-925c-710fff638c2d\") " pod="openshift-console/console-78c764b57c-98fm8"
May 25 11:25:14.923165 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:25:14.922914    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-pj2cq\" (UniqueName: \"kubernetes.io/projected/db40b82d-e321-4af8-925c-710fff638c2d-kube-api-access-pj2cq\") pod \"console-78c764b57c-98fm8\" (UID: \"db40b82d-e321-4af8-925c-710fff638c2d\") " pod="openshift-console/console-78c764b57c-98fm8"
May 25 11:25:14.923566 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:25:14.923540    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/db40b82d-e321-4af8-925c-710fff638c2d-oauth-serving-cert\") pod \"console-78c764b57c-98fm8\" (UID: \"db40b82d-e321-4af8-925c-710fff638c2d\") " pod="openshift-console/console-78c764b57c-98fm8"
May 25 11:25:14.923660 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:25:14.923568    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/db40b82d-e321-4af8-925c-710fff638c2d-console-config\") pod \"console-78c764b57c-98fm8\" (UID: \"db40b82d-e321-4af8-925c-710fff638c2d\") " pod="openshift-console/console-78c764b57c-98fm8"
May 25 11:25:14.923660 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:25:14.923603    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/db40b82d-e321-4af8-925c-710fff638c2d-service-ca\") pod \"console-78c764b57c-98fm8\" (UID: \"db40b82d-e321-4af8-925c-710fff638c2d\") " pod="openshift-console/console-78c764b57c-98fm8"
May 25 11:25:14.923745 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:25:14.923727    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/db40b82d-e321-4af8-925c-710fff638c2d-trusted-ca-bundle\") pod \"console-78c764b57c-98fm8\" (UID: \"db40b82d-e321-4af8-925c-710fff638c2d\") " pod="openshift-console/console-78c764b57c-98fm8"
May 25 11:25:14.925262 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:25:14.925243    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/db40b82d-e321-4af8-925c-710fff638c2d-console-oauth-config\") pod \"console-78c764b57c-98fm8\" (UID: \"db40b82d-e321-4af8-925c-710fff638c2d\") " pod="openshift-console/console-78c764b57c-98fm8"
May 25 11:25:14.925451 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:25:14.925435    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/db40b82d-e321-4af8-925c-710fff638c2d-console-serving-cert\") pod \"console-78c764b57c-98fm8\" (UID: \"db40b82d-e321-4af8-925c-710fff638c2d\") " pod="openshift-console/console-78c764b57c-98fm8"
May 25 11:25:14.931085 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:25:14.931063    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pj2cq\" (UniqueName: \"kubernetes.io/projected/db40b82d-e321-4af8-925c-710fff638c2d-kube-api-access-pj2cq\") pod \"console-78c764b57c-98fm8\" (UID: \"db40b82d-e321-4af8-925c-710fff638c2d\") " pod="openshift-console/console-78c764b57c-98fm8"
May 25 11:25:14.956135 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:25:14.956115    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-78c764b57c-98fm8"
May 25 11:25:15.023739 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:25:15.023708    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dmds\" (UniqueName: \"kubernetes.io/projected/e9009c16-ff47-44bc-93c9-0deeeacf1c71-kube-api-access-9dmds\") pod \"seaweedfs-tls-custom-5c88b85bb7-mhx2w\" (UID: \"e9009c16-ff47-44bc-93c9-0deeeacf1c71\") " pod="kserve/seaweedfs-tls-custom-5c88b85bb7-mhx2w"
May 25 11:25:15.023875 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:25:15.023811    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"seaweedfs-tls-custom\" (UniqueName: \"kubernetes.io/projected/e9009c16-ff47-44bc-93c9-0deeeacf1c71-seaweedfs-tls-custom\") pod \"seaweedfs-tls-custom-5c88b85bb7-mhx2w\" (UID: \"e9009c16-ff47-44bc-93c9-0deeeacf1c71\") " pod="kserve/seaweedfs-tls-custom-5c88b85bb7-mhx2w"
May 25 11:25:15.023875 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:25:15.023843    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/e9009c16-ff47-44bc-93c9-0deeeacf1c71-data\") pod \"seaweedfs-tls-custom-5c88b85bb7-mhx2w\" (UID: \"e9009c16-ff47-44bc-93c9-0deeeacf1c71\") " pod="kserve/seaweedfs-tls-custom-5c88b85bb7-mhx2w"
May 25 11:25:15.082720 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:25:15.082696    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-78c764b57c-98fm8"]
May 25 11:25:15.085093 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:25:15.085060    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb40b82d_e321_4af8_925c_710fff638c2d.slice/crio-ba7fc383871bf579fea321d06e8b578b9da5b9b627bd19e7011f3a7549f6b522 WatchSource:0}: Error finding container ba7fc383871bf579fea321d06e8b578b9da5b9b627bd19e7011f3a7549f6b522: Status 404 returned error can't find the container with id ba7fc383871bf579fea321d06e8b578b9da5b9b627bd19e7011f3a7549f6b522
May 25 11:25:15.119830 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:25:15.119807    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a8e0fb2a-7a5e-44d9-93f9-a4570e76381b" path="/var/lib/kubelet/pods/a8e0fb2a-7a5e-44d9-93f9-a4570e76381b/volumes"
May 25 11:25:15.125088 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:25:15.125055    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-9dmds\" (UniqueName: \"kubernetes.io/projected/e9009c16-ff47-44bc-93c9-0deeeacf1c71-kube-api-access-9dmds\") pod \"seaweedfs-tls-custom-5c88b85bb7-mhx2w\" (UID: \"e9009c16-ff47-44bc-93c9-0deeeacf1c71\") " pod="kserve/seaweedfs-tls-custom-5c88b85bb7-mhx2w"
May 25 11:25:15.125189 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:25:15.125113    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"seaweedfs-tls-custom\" (UniqueName: \"kubernetes.io/projected/e9009c16-ff47-44bc-93c9-0deeeacf1c71-seaweedfs-tls-custom\") pod \"seaweedfs-tls-custom-5c88b85bb7-mhx2w\" (UID: \"e9009c16-ff47-44bc-93c9-0deeeacf1c71\") " pod="kserve/seaweedfs-tls-custom-5c88b85bb7-mhx2w"
May 25 11:25:15.125189 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:25:15.125132    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/e9009c16-ff47-44bc-93c9-0deeeacf1c71-data\") pod \"seaweedfs-tls-custom-5c88b85bb7-mhx2w\" (UID: \"e9009c16-ff47-44bc-93c9-0deeeacf1c71\") " pod="kserve/seaweedfs-tls-custom-5c88b85bb7-mhx2w"
May 25 11:25:15.125416 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:25:15.125400    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/e9009c16-ff47-44bc-93c9-0deeeacf1c71-data\") pod \"seaweedfs-tls-custom-5c88b85bb7-mhx2w\" (UID: \"e9009c16-ff47-44bc-93c9-0deeeacf1c71\") " pod="kserve/seaweedfs-tls-custom-5c88b85bb7-mhx2w"
May 25 11:25:15.127248 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:25:15.127232    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"seaweedfs-tls-custom\" (UniqueName: \"kubernetes.io/projected/e9009c16-ff47-44bc-93c9-0deeeacf1c71-seaweedfs-tls-custom\") pod \"seaweedfs-tls-custom-5c88b85bb7-mhx2w\" (UID: \"e9009c16-ff47-44bc-93c9-0deeeacf1c71\") " pod="kserve/seaweedfs-tls-custom-5c88b85bb7-mhx2w"
May 25 11:25:15.133510 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:25:15.133490    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dmds\" (UniqueName: \"kubernetes.io/projected/e9009c16-ff47-44bc-93c9-0deeeacf1c71-kube-api-access-9dmds\") pod \"seaweedfs-tls-custom-5c88b85bb7-mhx2w\" (UID: \"e9009c16-ff47-44bc-93c9-0deeeacf1c71\") " pod="kserve/seaweedfs-tls-custom-5c88b85bb7-mhx2w"
May 25 11:25:15.202208 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:25:15.202172    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-5c88b85bb7-mhx2w"
May 25 11:25:15.326423 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:25:15.326389    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-5c88b85bb7-mhx2w"]
May 25 11:25:15.329541 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:25:15.329513    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9009c16_ff47_44bc_93c9_0deeeacf1c71.slice/crio-5ca79e96f2764a126cf152a5d7ad0a74ae092cedbffb5fe4f186f9ed0c1dbd2c WatchSource:0}: Error finding container 5ca79e96f2764a126cf152a5d7ad0a74ae092cedbffb5fe4f186f9ed0c1dbd2c: Status 404 returned error can't find the container with id 5ca79e96f2764a126cf152a5d7ad0a74ae092cedbffb5fe4f186f9ed0c1dbd2c
May 25 11:25:15.829424 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:25:15.829387    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-5c88b85bb7-mhx2w" event={"ID":"e9009c16-ff47-44bc-93c9-0deeeacf1c71","Type":"ContainerStarted","Data":"8da898ba0ded75679c1499188cc76ebc4d1fece7603dbc3f655b19f3be619ea0"}
May 25 11:25:15.829588 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:25:15.829430    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-5c88b85bb7-mhx2w" event={"ID":"e9009c16-ff47-44bc-93c9-0deeeacf1c71","Type":"ContainerStarted","Data":"5ca79e96f2764a126cf152a5d7ad0a74ae092cedbffb5fe4f186f9ed0c1dbd2c"}
May 25 11:25:15.831552 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:25:15.831524    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-78c764b57c-98fm8" event={"ID":"db40b82d-e321-4af8-925c-710fff638c2d","Type":"ContainerStarted","Data":"195ed051ca3e5feab01cc15b62ad5d236a6441597d9b002ad804f7b33e5c16c0"}
May 25 11:25:15.831662 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:25:15.831558    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-78c764b57c-98fm8" event={"ID":"db40b82d-e321-4af8-925c-710fff638c2d","Type":"ContainerStarted","Data":"ba7fc383871bf579fea321d06e8b578b9da5b9b627bd19e7011f3a7549f6b522"}
May 25 11:25:15.849946 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:25:15.849860    2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve/seaweedfs-tls-custom-5c88b85bb7-mhx2w" podStartSLOduration=1.5936238139999999 podStartE2EDuration="1.849846565s" podCreationTimestamp="2026-05-25 11:25:14 +0000 UTC" firstStartedPulling="2026-05-25 11:25:15.330817174 +0000 UTC m=+488.875994755" lastFinishedPulling="2026-05-25 11:25:15.587039925 +0000 UTC m=+489.132217506" observedRunningTime="2026-05-25 11:25:15.847450559 +0000 UTC m=+489.392628162" watchObservedRunningTime="2026-05-25 11:25:15.849846565 +0000 UTC m=+489.395024167"
May 25 11:25:15.870322 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:25:15.870256    2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console/console-78c764b57c-98fm8" podStartSLOduration=1.870244794 podStartE2EDuration="1.870244794s" podCreationTimestamp="2026-05-25 11:25:14 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 11:25:15.868685851 +0000 UTC m=+489.413863457" watchObservedRunningTime="2026-05-25 11:25:15.870244794 +0000 UTC m=+489.415422396"
May 25 11:25:24.956445 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:25:24.956400    2561 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-78c764b57c-98fm8"
May 25 11:25:24.956903 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:25:24.956492    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-78c764b57c-98fm8"
May 25 11:25:24.961027 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:25:24.961003    2561 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-78c764b57c-98fm8"
May 25 11:25:25.872785 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:25:25.872757    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-78c764b57c-98fm8"
May 25 11:25:50.130688 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:25:50.130655    2561 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5d8f79df8c-lp8w8"]
May 25 11:25:50.135524 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:25:50.135502    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5d8f79df8c-lp8w8"
May 25 11:25:50.139603 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:25:50.139582    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
May 25 11:25:50.139720 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:25:50.139700    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-predictor-serving-cert\""
May 25 11:25:50.139838 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:25:50.139821    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
May 25 11:25:50.140074 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:25:50.140058    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-kube-rbac-proxy-sar-config\""
May 25 11:25:50.140655 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:25:50.140640    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-svs62\""
May 25 11:25:50.151535 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:25:50.151511    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5d8f79df8c-lp8w8"]
May 25 11:25:50.189717 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:25:50.189690    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3d609a8b-9235-486b-a281-5d40e5b330d9-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-5d8f79df8c-lp8w8\" (UID: \"3d609a8b-9235-486b-a281-5d40e5b330d9\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5d8f79df8c-lp8w8"
May 25 11:25:50.189857 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:25:50.189737    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-sklearn-batcher-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3d609a8b-9235-486b-a281-5d40e5b330d9-isvc-sklearn-batcher-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-5d8f79df8c-lp8w8\" (UID: \"3d609a8b-9235-486b-a281-5d40e5b330d9\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5d8f79df8c-lp8w8"
May 25 11:25:50.189857 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:25:50.189778    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbhvh\" (UniqueName: \"kubernetes.io/projected/3d609a8b-9235-486b-a281-5d40e5b330d9-kube-api-access-sbhvh\") pod \"isvc-sklearn-batcher-predictor-5d8f79df8c-lp8w8\" (UID: \"3d609a8b-9235-486b-a281-5d40e5b330d9\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5d8f79df8c-lp8w8"
May 25 11:25:50.189857 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:25:50.189804    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3d609a8b-9235-486b-a281-5d40e5b330d9-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-5d8f79df8c-lp8w8\" (UID: \"3d609a8b-9235-486b-a281-5d40e5b330d9\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5d8f79df8c-lp8w8"
May 25 11:25:50.290729 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:25:50.290689    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3d609a8b-9235-486b-a281-5d40e5b330d9-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-5d8f79df8c-lp8w8\" (UID: \"3d609a8b-9235-486b-a281-5d40e5b330d9\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5d8f79df8c-lp8w8"
May 25 11:25:50.290924 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:25:50.290740    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-sklearn-batcher-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3d609a8b-9235-486b-a281-5d40e5b330d9-isvc-sklearn-batcher-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-5d8f79df8c-lp8w8\" (UID: \"3d609a8b-9235-486b-a281-5d40e5b330d9\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5d8f79df8c-lp8w8"
May 25 11:25:50.290924 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:25:50.290775    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-sbhvh\" (UniqueName: \"kubernetes.io/projected/3d609a8b-9235-486b-a281-5d40e5b330d9-kube-api-access-sbhvh\") pod \"isvc-sklearn-batcher-predictor-5d8f79df8c-lp8w8\" (UID: \"3d609a8b-9235-486b-a281-5d40e5b330d9\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5d8f79df8c-lp8w8"
May 25 11:25:50.290924 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:25:50.290801    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3d609a8b-9235-486b-a281-5d40e5b330d9-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-5d8f79df8c-lp8w8\" (UID: \"3d609a8b-9235-486b-a281-5d40e5b330d9\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5d8f79df8c-lp8w8"
May 25 11:25:50.291323 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:25:50.291294    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3d609a8b-9235-486b-a281-5d40e5b330d9-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-5d8f79df8c-lp8w8\" (UID: \"3d609a8b-9235-486b-a281-5d40e5b330d9\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5d8f79df8c-lp8w8"
May 25 11:25:50.291584 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:25:50.291560    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-sklearn-batcher-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3d609a8b-9235-486b-a281-5d40e5b330d9-isvc-sklearn-batcher-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-5d8f79df8c-lp8w8\" (UID: \"3d609a8b-9235-486b-a281-5d40e5b330d9\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5d8f79df8c-lp8w8"
May 25 11:25:50.293227 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:25:50.293206    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3d609a8b-9235-486b-a281-5d40e5b330d9-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-5d8f79df8c-lp8w8\" (UID: \"3d609a8b-9235-486b-a281-5d40e5b330d9\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5d8f79df8c-lp8w8"
May 25 11:25:50.299188 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:25:50.299157    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbhvh\" (UniqueName: \"kubernetes.io/projected/3d609a8b-9235-486b-a281-5d40e5b330d9-kube-api-access-sbhvh\") pod \"isvc-sklearn-batcher-predictor-5d8f79df8c-lp8w8\" (UID: \"3d609a8b-9235-486b-a281-5d40e5b330d9\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5d8f79df8c-lp8w8"
May 25 11:25:50.446300 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:25:50.446196    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5d8f79df8c-lp8w8"
May 25 11:25:50.578908 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:25:50.578883    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5d8f79df8c-lp8w8"]
May 25 11:25:50.581183 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:25:50.581153    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d609a8b_9235_486b_a281_5d40e5b330d9.slice/crio-7d39cd64837e003a196f74e642828503a026b55146e2b55aa767089c1fa210f1 WatchSource:0}: Error finding container 7d39cd64837e003a196f74e642828503a026b55146e2b55aa767089c1fa210f1: Status 404 returned error can't find the container with id 7d39cd64837e003a196f74e642828503a026b55146e2b55aa767089c1fa210f1
May 25 11:25:50.953565 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:25:50.953537    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5d8f79df8c-lp8w8" event={"ID":"3d609a8b-9235-486b-a281-5d40e5b330d9","Type":"ContainerStarted","Data":"7d39cd64837e003a196f74e642828503a026b55146e2b55aa767089c1fa210f1"}
May 25 11:25:53.965312 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:25:53.965202    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5d8f79df8c-lp8w8" event={"ID":"3d609a8b-9235-486b-a281-5d40e5b330d9","Type":"ContainerStarted","Data":"deb363619350e414fa08f6ccbb2635e73872590f31e595c8b1658187d0999ad5"}
May 25 11:25:57.979449 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:25:57.979358    2561 generic.go:358] "Generic (PLEG): container finished" podID="3d609a8b-9235-486b-a281-5d40e5b330d9" containerID="deb363619350e414fa08f6ccbb2635e73872590f31e595c8b1658187d0999ad5" exitCode=0
May 25 11:25:57.979449 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:25:57.979434    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5d8f79df8c-lp8w8" event={"ID":"3d609a8b-9235-486b-a281-5d40e5b330d9","Type":"ContainerDied","Data":"deb363619350e414fa08f6ccbb2635e73872590f31e595c8b1658187d0999ad5"}
May 25 11:26:11.038289 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:26:11.038220    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5d8f79df8c-lp8w8" event={"ID":"3d609a8b-9235-486b-a281-5d40e5b330d9","Type":"ContainerStarted","Data":"8b026f018dc67dfd6b052be11c4c52636c942bfe5099a42a6aa69e8a33e8bbd8"}
May 25 11:26:14.051543 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:26:14.051507    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5d8f79df8c-lp8w8" event={"ID":"3d609a8b-9235-486b-a281-5d40e5b330d9","Type":"ContainerStarted","Data":"eeb47edb240875fa4a5223bd0f5d58f26fdc43e2866e94f3738e25c2d11a3722"}
May 25 11:26:16.060788 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:26:16.060714    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5d8f79df8c-lp8w8" event={"ID":"3d609a8b-9235-486b-a281-5d40e5b330d9","Type":"ContainerStarted","Data":"50f92ac7258951f4a7e4b3c73571670a6bf1c9039fac382ce9c9b9bbc537c0fa"}
May 25 11:26:16.061321 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:26:16.060885    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5d8f79df8c-lp8w8"
May 25 11:26:16.061321 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:26:16.060925    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5d8f79df8c-lp8w8"
May 25 11:26:16.061321 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:26:16.061038    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5d8f79df8c-lp8w8"
May 25 11:26:16.062441 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:26:16.062400    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5d8f79df8c-lp8w8" podUID="3d609a8b-9235-486b-a281-5d40e5b330d9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
May 25 11:26:16.063040 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:26:16.063019    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5d8f79df8c-lp8w8" podUID="3d609a8b-9235-486b-a281-5d40e5b330d9" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 11:26:16.085485 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:26:16.085439    2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5d8f79df8c-lp8w8" podStartSLOduration=0.870649263 podStartE2EDuration="26.085427605s" podCreationTimestamp="2026-05-25 11:25:50 +0000 UTC" firstStartedPulling="2026-05-25 11:25:50.583135886 +0000 UTC m=+524.128313467" lastFinishedPulling="2026-05-25 11:26:15.797914212 +0000 UTC m=+549.343091809" observedRunningTime="2026-05-25 11:26:16.083151582 +0000 UTC m=+549.628329186" watchObservedRunningTime="2026-05-25 11:26:16.085427605 +0000 UTC m=+549.630605207"
May 25 11:26:17.064230 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:26:17.064188    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5d8f79df8c-lp8w8" podUID="3d609a8b-9235-486b-a281-5d40e5b330d9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
May 25 11:26:17.064798 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:26:17.064446    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5d8f79df8c-lp8w8" podUID="3d609a8b-9235-486b-a281-5d40e5b330d9" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 11:26:22.068629 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:26:22.068597    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5d8f79df8c-lp8w8"
May 25 11:26:22.069300 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:26:22.069256    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5d8f79df8c-lp8w8" podUID="3d609a8b-9235-486b-a281-5d40e5b330d9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
May 25 11:26:22.069705 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:26:22.069675    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5d8f79df8c-lp8w8" podUID="3d609a8b-9235-486b-a281-5d40e5b330d9" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 11:26:32.069800 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:26:32.069752    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5d8f79df8c-lp8w8" podUID="3d609a8b-9235-486b-a281-5d40e5b330d9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
May 25 11:26:32.070187 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:26:32.070144    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5d8f79df8c-lp8w8" podUID="3d609a8b-9235-486b-a281-5d40e5b330d9" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 11:26:42.069412 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:26:42.069362    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5d8f79df8c-lp8w8" podUID="3d609a8b-9235-486b-a281-5d40e5b330d9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
May 25 11:26:42.069841 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:26:42.069815    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5d8f79df8c-lp8w8" podUID="3d609a8b-9235-486b-a281-5d40e5b330d9" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 11:26:52.069136 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:26:52.069096    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5d8f79df8c-lp8w8" podUID="3d609a8b-9235-486b-a281-5d40e5b330d9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
May 25 11:26:52.069794 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:26:52.069773    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5d8f79df8c-lp8w8" podUID="3d609a8b-9235-486b-a281-5d40e5b330d9" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 11:27:02.069578 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:27:02.069529    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5d8f79df8c-lp8w8" podUID="3d609a8b-9235-486b-a281-5d40e5b330d9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
May 25 11:27:02.069977 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:27:02.069840    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5d8f79df8c-lp8w8" podUID="3d609a8b-9235-486b-a281-5d40e5b330d9" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 11:27:07.014022 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:27:07.013995    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-9lj95_10a66344-3304-49a1-bbc1-b885800f5acf/console-operator/1.log"
May 25 11:27:07.014488 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:27:07.013995    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-9lj95_10a66344-3304-49a1-bbc1-b885800f5acf/console-operator/1.log"
May 25 11:27:12.069333 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:27:12.069286    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5d8f79df8c-lp8w8" podUID="3d609a8b-9235-486b-a281-5d40e5b330d9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
May 25 11:27:12.069792 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:27:12.069707    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5d8f79df8c-lp8w8" podUID="3d609a8b-9235-486b-a281-5d40e5b330d9" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 11:27:22.069484 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:27:22.069450    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5d8f79df8c-lp8w8"
May 25 11:27:22.069990 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:27:22.069970    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5d8f79df8c-lp8w8"
May 25 11:27:35.096808 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:27:35.096769    2561 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5d8f79df8c-lp8w8"]
May 25 11:27:35.098647 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:27:35.097158    2561 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5d8f79df8c-lp8w8" podUID="3d609a8b-9235-486b-a281-5d40e5b330d9" containerName="kserve-container" containerID="cri-o://8b026f018dc67dfd6b052be11c4c52636c942bfe5099a42a6aa69e8a33e8bbd8" gracePeriod=30
May 25 11:27:35.098647 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:27:35.097201    2561 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5d8f79df8c-lp8w8" podUID="3d609a8b-9235-486b-a281-5d40e5b330d9" containerName="kube-rbac-proxy" containerID="cri-o://eeb47edb240875fa4a5223bd0f5d58f26fdc43e2866e94f3738e25c2d11a3722" gracePeriod=30
May 25 11:27:35.098647 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:27:35.097203    2561 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5d8f79df8c-lp8w8" podUID="3d609a8b-9235-486b-a281-5d40e5b330d9" containerName="agent" containerID="cri-o://50f92ac7258951f4a7e4b3c73571670a6bf1c9039fac382ce9c9b9bbc537c0fa" gracePeriod=30
May 25 11:27:35.223665 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:27:35.223630    2561 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5899b869f4-t7jx7"]
May 25 11:27:35.226310 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:27:35.226292    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5899b869f4-t7jx7"
May 25 11:27:35.230086 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:27:35.230060    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config\""
May 25 11:27:35.230185 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:27:35.230085    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-custom-predictor-serving-cert\""
May 25 11:27:35.246678 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:27:35.246652    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5899b869f4-t7jx7"]
May 25 11:27:35.316674 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:27:35.316644    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4fdf6bf7-80d8-4af3-81d7-1c9dde107499-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-5899b869f4-t7jx7\" (UID: \"4fdf6bf7-80d8-4af3-81d7-1c9dde107499\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5899b869f4-t7jx7"
May 25 11:27:35.316849 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:27:35.316799    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4fdf6bf7-80d8-4af3-81d7-1c9dde107499-isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-5899b869f4-t7jx7\" (UID: \"4fdf6bf7-80d8-4af3-81d7-1c9dde107499\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5899b869f4-t7jx7"
May 25 11:27:35.316849 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:27:35.316833    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4fdf6bf7-80d8-4af3-81d7-1c9dde107499-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-5899b869f4-t7jx7\" (UID: \"4fdf6bf7-80d8-4af3-81d7-1c9dde107499\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5899b869f4-t7jx7"
May 25 11:27:35.316954 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:27:35.316859    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pmd5\" (UniqueName: \"kubernetes.io/projected/4fdf6bf7-80d8-4af3-81d7-1c9dde107499-kube-api-access-6pmd5\") pod \"isvc-sklearn-batcher-custom-predictor-5899b869f4-t7jx7\" (UID: \"4fdf6bf7-80d8-4af3-81d7-1c9dde107499\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5899b869f4-t7jx7"
May 25 11:27:35.347392 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:27:35.347313    2561 generic.go:358] "Generic (PLEG): container finished" podID="3d609a8b-9235-486b-a281-5d40e5b330d9" containerID="eeb47edb240875fa4a5223bd0f5d58f26fdc43e2866e94f3738e25c2d11a3722" exitCode=2
May 25 11:27:35.347392 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:27:35.347376    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5d8f79df8c-lp8w8" event={"ID":"3d609a8b-9235-486b-a281-5d40e5b330d9","Type":"ContainerDied","Data":"eeb47edb240875fa4a5223bd0f5d58f26fdc43e2866e94f3738e25c2d11a3722"}
May 25 11:27:35.418153 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:27:35.418117    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4fdf6bf7-80d8-4af3-81d7-1c9dde107499-isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-5899b869f4-t7jx7\" (UID: \"4fdf6bf7-80d8-4af3-81d7-1c9dde107499\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5899b869f4-t7jx7"
May 25 11:27:35.418153 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:27:35.418154    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4fdf6bf7-80d8-4af3-81d7-1c9dde107499-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-5899b869f4-t7jx7\" (UID: \"4fdf6bf7-80d8-4af3-81d7-1c9dde107499\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5899b869f4-t7jx7"
May 25 11:27:35.418424 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:27:35.418179    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-6pmd5\" (UniqueName: \"kubernetes.io/projected/4fdf6bf7-80d8-4af3-81d7-1c9dde107499-kube-api-access-6pmd5\") pod \"isvc-sklearn-batcher-custom-predictor-5899b869f4-t7jx7\" (UID: \"4fdf6bf7-80d8-4af3-81d7-1c9dde107499\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5899b869f4-t7jx7"
May 25 11:27:35.418424 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:27:35.418201    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4fdf6bf7-80d8-4af3-81d7-1c9dde107499-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-5899b869f4-t7jx7\" (UID: \"4fdf6bf7-80d8-4af3-81d7-1c9dde107499\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5899b869f4-t7jx7"
May 25 11:27:35.418563 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:27:35.418440    2561 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-serving-cert: secret "isvc-sklearn-batcher-custom-predictor-serving-cert" not found
May 25 11:27:35.418563 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:27:35.418518    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4fdf6bf7-80d8-4af3-81d7-1c9dde107499-proxy-tls podName:4fdf6bf7-80d8-4af3-81d7-1c9dde107499 nodeName:}" failed. No retries permitted until 2026-05-25 11:27:35.918495715 +0000 UTC m=+629.463673297 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/4fdf6bf7-80d8-4af3-81d7-1c9dde107499-proxy-tls") pod "isvc-sklearn-batcher-custom-predictor-5899b869f4-t7jx7" (UID: "4fdf6bf7-80d8-4af3-81d7-1c9dde107499") : secret "isvc-sklearn-batcher-custom-predictor-serving-cert" not found
May 25 11:27:35.418689 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:27:35.418587    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4fdf6bf7-80d8-4af3-81d7-1c9dde107499-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-5899b869f4-t7jx7\" (UID: \"4fdf6bf7-80d8-4af3-81d7-1c9dde107499\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5899b869f4-t7jx7"
May 25 11:27:35.418800 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:27:35.418778    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4fdf6bf7-80d8-4af3-81d7-1c9dde107499-isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-5899b869f4-t7jx7\" (UID: \"4fdf6bf7-80d8-4af3-81d7-1c9dde107499\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5899b869f4-t7jx7"
May 25 11:27:35.429863 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:27:35.429838    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pmd5\" (UniqueName: \"kubernetes.io/projected/4fdf6bf7-80d8-4af3-81d7-1c9dde107499-kube-api-access-6pmd5\") pod \"isvc-sklearn-batcher-custom-predictor-5899b869f4-t7jx7\" (UID: \"4fdf6bf7-80d8-4af3-81d7-1c9dde107499\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5899b869f4-t7jx7"
May 25 11:27:35.923221 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:27:35.923182    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4fdf6bf7-80d8-4af3-81d7-1c9dde107499-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-5899b869f4-t7jx7\" (UID: \"4fdf6bf7-80d8-4af3-81d7-1c9dde107499\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5899b869f4-t7jx7"
May 25 11:27:35.925567 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:27:35.925538    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4fdf6bf7-80d8-4af3-81d7-1c9dde107499-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-5899b869f4-t7jx7\" (UID: \"4fdf6bf7-80d8-4af3-81d7-1c9dde107499\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5899b869f4-t7jx7"
May 25 11:27:36.136243 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:27:36.136202    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5899b869f4-t7jx7"
May 25 11:27:36.269176 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:27:36.269098    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5899b869f4-t7jx7"]
May 25 11:27:36.272608 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:27:36.272573    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fdf6bf7_80d8_4af3_81d7_1c9dde107499.slice/crio-038d0b75d474e13b828d9d7503926d9039f519ee66b9c243175f752434d6a700 WatchSource:0}: Error finding container 038d0b75d474e13b828d9d7503926d9039f519ee66b9c243175f752434d6a700: Status 404 returned error can't find the container with id 038d0b75d474e13b828d9d7503926d9039f519ee66b9c243175f752434d6a700
May 25 11:27:36.274885 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:27:36.274868    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 25 11:27:36.352596 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:27:36.352562    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5899b869f4-t7jx7" event={"ID":"4fdf6bf7-80d8-4af3-81d7-1c9dde107499","Type":"ContainerStarted","Data":"87b9909983bb29fc43b15558528e0f6970bd2558853ab06f9495d8a3355c2888"}
May 25 11:27:36.352596 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:27:36.352599    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5899b869f4-t7jx7" event={"ID":"4fdf6bf7-80d8-4af3-81d7-1c9dde107499","Type":"ContainerStarted","Data":"038d0b75d474e13b828d9d7503926d9039f519ee66b9c243175f752434d6a700"}
May 25 11:27:37.064700 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:27:37.064650    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5d8f79df8c-lp8w8" podUID="3d609a8b-9235-486b-a281-5d40e5b330d9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.36:8643/healthz\": dial tcp 10.132.0.36:8643: connect: connection refused"
May 25 11:27:39.365121 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:27:39.365084    2561 generic.go:358] "Generic (PLEG): container finished" podID="3d609a8b-9235-486b-a281-5d40e5b330d9" containerID="8b026f018dc67dfd6b052be11c4c52636c942bfe5099a42a6aa69e8a33e8bbd8" exitCode=0
May 25 11:27:39.365535 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:27:39.365160    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5d8f79df8c-lp8w8" event={"ID":"3d609a8b-9235-486b-a281-5d40e5b330d9","Type":"ContainerDied","Data":"8b026f018dc67dfd6b052be11c4c52636c942bfe5099a42a6aa69e8a33e8bbd8"}
May 25 11:27:40.369333 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:27:40.369302    2561 generic.go:358] "Generic (PLEG): container finished" podID="4fdf6bf7-80d8-4af3-81d7-1c9dde107499" containerID="87b9909983bb29fc43b15558528e0f6970bd2558853ab06f9495d8a3355c2888" exitCode=0
May 25 11:27:40.369710 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:27:40.369345    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5899b869f4-t7jx7" event={"ID":"4fdf6bf7-80d8-4af3-81d7-1c9dde107499","Type":"ContainerDied","Data":"87b9909983bb29fc43b15558528e0f6970bd2558853ab06f9495d8a3355c2888"}
May 25 11:27:41.375006 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:27:41.374970    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5899b869f4-t7jx7" event={"ID":"4fdf6bf7-80d8-4af3-81d7-1c9dde107499","Type":"ContainerStarted","Data":"90101b777368c00ace927ac891dc0baafcef4fdd9042fcfdd5738c1146b7a392"}
May 25 11:27:41.375006 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:27:41.375010    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5899b869f4-t7jx7" event={"ID":"4fdf6bf7-80d8-4af3-81d7-1c9dde107499","Type":"ContainerStarted","Data":"06f546da1701c04f608494cf51b3eff5933fe908911f966947d54c84a54dcf3a"}
May 25 11:27:41.375452 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:27:41.375022    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5899b869f4-t7jx7" event={"ID":"4fdf6bf7-80d8-4af3-81d7-1c9dde107499","Type":"ContainerStarted","Data":"f11f529775c30248fb282b9aa620fd4cd608f05cef453c03e59cedc88ba0a75c"}
May 25 11:27:41.375452 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:27:41.375292    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5899b869f4-t7jx7"
May 25 11:27:41.375452 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:27:41.375421    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5899b869f4-t7jx7"
May 25 11:27:41.376759 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:27:41.376735    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5899b869f4-t7jx7" podUID="4fdf6bf7-80d8-4af3-81d7-1c9dde107499" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:5000: connect: connection refused"
May 25 11:27:41.395407 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:27:41.395370    2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5899b869f4-t7jx7" podStartSLOduration=6.395357482 podStartE2EDuration="6.395357482s" podCreationTimestamp="2026-05-25 11:27:35 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 11:27:41.394487238 +0000 UTC m=+634.939664877" watchObservedRunningTime="2026-05-25 11:27:41.395357482 +0000 UTC m=+634.940535085"
May 25 11:27:42.064430 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:27:42.064377    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5d8f79df8c-lp8w8" podUID="3d609a8b-9235-486b-a281-5d40e5b330d9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.36:8643/healthz\": dial tcp 10.132.0.36:8643: connect: connection refused"
May 25 11:27:42.069823 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:27:42.069795    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5d8f79df8c-lp8w8" podUID="3d609a8b-9235-486b-a281-5d40e5b330d9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
May 25 11:27:42.070441 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:27:42.070417    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5d8f79df8c-lp8w8" podUID="3d609a8b-9235-486b-a281-5d40e5b330d9" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 11:27:42.378513 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:27:42.378467    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5899b869f4-t7jx7" podUID="4fdf6bf7-80d8-4af3-81d7-1c9dde107499" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:5000: connect: connection refused"
May 25 11:27:42.378513 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:27:42.378498    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5899b869f4-t7jx7"
May 25 11:27:42.379584 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:27:42.379561    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5899b869f4-t7jx7" podUID="4fdf6bf7-80d8-4af3-81d7-1c9dde107499" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 11:27:43.381585 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:27:43.381542    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5899b869f4-t7jx7" podUID="4fdf6bf7-80d8-4af3-81d7-1c9dde107499" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:5000: connect: connection refused"
May 25 11:27:43.382037 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:27:43.382021    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5899b869f4-t7jx7" podUID="4fdf6bf7-80d8-4af3-81d7-1c9dde107499" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 11:27:47.064449 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:27:47.064410    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5d8f79df8c-lp8w8" podUID="3d609a8b-9235-486b-a281-5d40e5b330d9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.36:8643/healthz\": dial tcp 10.132.0.36:8643: connect: connection refused"
May 25 11:27:47.064886 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:27:47.064530    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5d8f79df8c-lp8w8"
May 25 11:27:48.386114 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:27:48.386084    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5899b869f4-t7jx7"
May 25 11:27:48.386614 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:27:48.386589    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5899b869f4-t7jx7" podUID="4fdf6bf7-80d8-4af3-81d7-1c9dde107499" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:5000: connect: connection refused"
May 25 11:27:48.386827 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:27:48.386805    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5899b869f4-t7jx7" podUID="4fdf6bf7-80d8-4af3-81d7-1c9dde107499" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 11:27:52.064579 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:27:52.064529    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5d8f79df8c-lp8w8" podUID="3d609a8b-9235-486b-a281-5d40e5b330d9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.36:8643/healthz\": dial tcp 10.132.0.36:8643: connect: connection refused"
May 25 11:27:52.069917 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:27:52.069883    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5d8f79df8c-lp8w8" podUID="3d609a8b-9235-486b-a281-5d40e5b330d9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
May 25 11:27:52.070320 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:27:52.070291    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5d8f79df8c-lp8w8" podUID="3d609a8b-9235-486b-a281-5d40e5b330d9" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 11:27:57.064667 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:27:57.064629    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5d8f79df8c-lp8w8" podUID="3d609a8b-9235-486b-a281-5d40e5b330d9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.36:8643/healthz\": dial tcp 10.132.0.36:8643: connect: connection refused"
May 25 11:27:58.386907 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:27:58.386864    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5899b869f4-t7jx7" podUID="4fdf6bf7-80d8-4af3-81d7-1c9dde107499" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:5000: connect: connection refused"
May 25 11:27:58.387363 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:27:58.387239    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5899b869f4-t7jx7" podUID="4fdf6bf7-80d8-4af3-81d7-1c9dde107499" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 11:28:02.064942 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:28:02.064897    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5d8f79df8c-lp8w8" podUID="3d609a8b-9235-486b-a281-5d40e5b330d9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.36:8643/healthz\": dial tcp 10.132.0.36:8643: connect: connection refused"
May 25 11:28:02.070118 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:28:02.070078    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5d8f79df8c-lp8w8" podUID="3d609a8b-9235-486b-a281-5d40e5b330d9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
May 25 11:28:02.070260 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:28:02.070209    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5d8f79df8c-lp8w8"
May 25 11:28:02.070453 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:28:02.070435    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5d8f79df8c-lp8w8" podUID="3d609a8b-9235-486b-a281-5d40e5b330d9" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 11:28:02.070536 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:28:02.070525    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5d8f79df8c-lp8w8"
May 25 11:28:05.253947 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:28:05.253925    2561 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5d8f79df8c-lp8w8"
May 25 11:28:05.292958 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:28:05.292922    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sbhvh\" (UniqueName: \"kubernetes.io/projected/3d609a8b-9235-486b-a281-5d40e5b330d9-kube-api-access-sbhvh\") pod \"3d609a8b-9235-486b-a281-5d40e5b330d9\" (UID: \"3d609a8b-9235-486b-a281-5d40e5b330d9\") "
May 25 11:28:05.293141 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:28:05.292980    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3d609a8b-9235-486b-a281-5d40e5b330d9-kserve-provision-location\") pod \"3d609a8b-9235-486b-a281-5d40e5b330d9\" (UID: \"3d609a8b-9235-486b-a281-5d40e5b330d9\") "
May 25 11:28:05.293141 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:28:05.293023    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3d609a8b-9235-486b-a281-5d40e5b330d9-proxy-tls\") pod \"3d609a8b-9235-486b-a281-5d40e5b330d9\" (UID: \"3d609a8b-9235-486b-a281-5d40e5b330d9\") "
May 25 11:28:05.293141 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:28:05.293068    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-sklearn-batcher-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3d609a8b-9235-486b-a281-5d40e5b330d9-isvc-sklearn-batcher-kube-rbac-proxy-sar-config\") pod \"3d609a8b-9235-486b-a281-5d40e5b330d9\" (UID: \"3d609a8b-9235-486b-a281-5d40e5b330d9\") "
May 25 11:28:05.293510 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:28:05.293437    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d609a8b-9235-486b-a281-5d40e5b330d9-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3d609a8b-9235-486b-a281-5d40e5b330d9" (UID: "3d609a8b-9235-486b-a281-5d40e5b330d9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:28:05.293630 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:28:05.293599    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d609a8b-9235-486b-a281-5d40e5b330d9-isvc-sklearn-batcher-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-sklearn-batcher-kube-rbac-proxy-sar-config") pod "3d609a8b-9235-486b-a281-5d40e5b330d9" (UID: "3d609a8b-9235-486b-a281-5d40e5b330d9"). InnerVolumeSpecName "isvc-sklearn-batcher-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 11:28:05.295436 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:28:05.295409    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d609a8b-9235-486b-a281-5d40e5b330d9-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3d609a8b-9235-486b-a281-5d40e5b330d9" (UID: "3d609a8b-9235-486b-a281-5d40e5b330d9"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 11:28:05.295621 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:28:05.295595    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d609a8b-9235-486b-a281-5d40e5b330d9-kube-api-access-sbhvh" (OuterVolumeSpecName: "kube-api-access-sbhvh") pod "3d609a8b-9235-486b-a281-5d40e5b330d9" (UID: "3d609a8b-9235-486b-a281-5d40e5b330d9"). InnerVolumeSpecName "kube-api-access-sbhvh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 11:28:05.394656 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:28:05.394569    2561 reconciler_common.go:299] "Volume detached for volume \"isvc-sklearn-batcher-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3d609a8b-9235-486b-a281-5d40e5b330d9-isvc-sklearn-batcher-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-227.ec2.internal\" DevicePath \"\""
May 25 11:28:05.394656 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:28:05.394601    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sbhvh\" (UniqueName: \"kubernetes.io/projected/3d609a8b-9235-486b-a281-5d40e5b330d9-kube-api-access-sbhvh\") on node \"ip-10-0-136-227.ec2.internal\" DevicePath \"\""
May 25 11:28:05.394656 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:28:05.394614    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3d609a8b-9235-486b-a281-5d40e5b330d9-kserve-provision-location\") on node \"ip-10-0-136-227.ec2.internal\" DevicePath \"\""
May 25 11:28:05.394656 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:28:05.394627    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3d609a8b-9235-486b-a281-5d40e5b330d9-proxy-tls\") on node \"ip-10-0-136-227.ec2.internal\" DevicePath \"\""
May 25 11:28:05.458574 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:28:05.458542    2561 generic.go:358] "Generic (PLEG): container finished" podID="3d609a8b-9235-486b-a281-5d40e5b330d9" containerID="50f92ac7258951f4a7e4b3c73571670a6bf1c9039fac382ce9c9b9bbc537c0fa" exitCode=0
May 25 11:28:05.458731 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:28:05.458605    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5d8f79df8c-lp8w8" event={"ID":"3d609a8b-9235-486b-a281-5d40e5b330d9","Type":"ContainerDied","Data":"50f92ac7258951f4a7e4b3c73571670a6bf1c9039fac382ce9c9b9bbc537c0fa"}
May 25 11:28:05.458731 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:28:05.458634    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5d8f79df8c-lp8w8" event={"ID":"3d609a8b-9235-486b-a281-5d40e5b330d9","Type":"ContainerDied","Data":"7d39cd64837e003a196f74e642828503a026b55146e2b55aa767089c1fa210f1"}
May 25 11:28:05.458731 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:28:05.458637    2561 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5d8f79df8c-lp8w8"
May 25 11:28:05.458731 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:28:05.458649    2561 scope.go:117] "RemoveContainer" containerID="50f92ac7258951f4a7e4b3c73571670a6bf1c9039fac382ce9c9b9bbc537c0fa"
May 25 11:28:05.467825 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:28:05.467804    2561 scope.go:117] "RemoveContainer" containerID="eeb47edb240875fa4a5223bd0f5d58f26fdc43e2866e94f3738e25c2d11a3722"
May 25 11:28:05.475099 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:28:05.475083    2561 scope.go:117] "RemoveContainer" containerID="8b026f018dc67dfd6b052be11c4c52636c942bfe5099a42a6aa69e8a33e8bbd8"
May 25 11:28:05.481504 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:28:05.481479    2561 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5d8f79df8c-lp8w8"]
May 25 11:28:05.483187 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:28:05.483165    2561 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-5d8f79df8c-lp8w8"]
May 25 11:28:05.483823 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:28:05.483809    2561 scope.go:117] "RemoveContainer" containerID="deb363619350e414fa08f6ccbb2635e73872590f31e595c8b1658187d0999ad5"
May 25 11:28:05.492592 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:28:05.492562    2561 scope.go:117] "RemoveContainer" containerID="50f92ac7258951f4a7e4b3c73571670a6bf1c9039fac382ce9c9b9bbc537c0fa"
May 25 11:28:05.492823 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:28:05.492804    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50f92ac7258951f4a7e4b3c73571670a6bf1c9039fac382ce9c9b9bbc537c0fa\": container with ID starting with 50f92ac7258951f4a7e4b3c73571670a6bf1c9039fac382ce9c9b9bbc537c0fa not found: ID does not exist" containerID="50f92ac7258951f4a7e4b3c73571670a6bf1c9039fac382ce9c9b9bbc537c0fa"
May 25 11:28:05.492886 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:28:05.492832    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50f92ac7258951f4a7e4b3c73571670a6bf1c9039fac382ce9c9b9bbc537c0fa"} err="failed to get container status \"50f92ac7258951f4a7e4b3c73571670a6bf1c9039fac382ce9c9b9bbc537c0fa\": rpc error: code = NotFound desc = could not find container \"50f92ac7258951f4a7e4b3c73571670a6bf1c9039fac382ce9c9b9bbc537c0fa\": container with ID starting with 50f92ac7258951f4a7e4b3c73571670a6bf1c9039fac382ce9c9b9bbc537c0fa not found: ID does not exist"
May 25 11:28:05.492886 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:28:05.492852    2561 scope.go:117] "RemoveContainer" containerID="eeb47edb240875fa4a5223bd0f5d58f26fdc43e2866e94f3738e25c2d11a3722"
May 25 11:28:05.493091 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:28:05.493068    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eeb47edb240875fa4a5223bd0f5d58f26fdc43e2866e94f3738e25c2d11a3722\": container with ID starting with eeb47edb240875fa4a5223bd0f5d58f26fdc43e2866e94f3738e25c2d11a3722 not found: ID does not exist" containerID="eeb47edb240875fa4a5223bd0f5d58f26fdc43e2866e94f3738e25c2d11a3722"
May 25 11:28:05.493154 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:28:05.493101    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eeb47edb240875fa4a5223bd0f5d58f26fdc43e2866e94f3738e25c2d11a3722"} err="failed to get container status \"eeb47edb240875fa4a5223bd0f5d58f26fdc43e2866e94f3738e25c2d11a3722\": rpc error: code = NotFound desc = could not find container \"eeb47edb240875fa4a5223bd0f5d58f26fdc43e2866e94f3738e25c2d11a3722\": container with ID starting with eeb47edb240875fa4a5223bd0f5d58f26fdc43e2866e94f3738e25c2d11a3722 not found: ID does not exist"
May 25 11:28:05.493154 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:28:05.493125    2561 scope.go:117] "RemoveContainer" containerID="8b026f018dc67dfd6b052be11c4c52636c942bfe5099a42a6aa69e8a33e8bbd8"
May 25 11:28:05.493383 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:28:05.493365    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b026f018dc67dfd6b052be11c4c52636c942bfe5099a42a6aa69e8a33e8bbd8\": container with ID starting with 8b026f018dc67dfd6b052be11c4c52636c942bfe5099a42a6aa69e8a33e8bbd8 not found: ID does not exist" containerID="8b026f018dc67dfd6b052be11c4c52636c942bfe5099a42a6aa69e8a33e8bbd8"
May 25 11:28:05.493471 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:28:05.493390    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b026f018dc67dfd6b052be11c4c52636c942bfe5099a42a6aa69e8a33e8bbd8"} err="failed to get container status \"8b026f018dc67dfd6b052be11c4c52636c942bfe5099a42a6aa69e8a33e8bbd8\": rpc error: code = NotFound desc = could not find container \"8b026f018dc67dfd6b052be11c4c52636c942bfe5099a42a6aa69e8a33e8bbd8\": container with ID starting with 8b026f018dc67dfd6b052be11c4c52636c942bfe5099a42a6aa69e8a33e8bbd8 not found: ID does not exist"
May 25 11:28:05.493471 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:28:05.493409    2561 scope.go:117] "RemoveContainer" containerID="deb363619350e414fa08f6ccbb2635e73872590f31e595c8b1658187d0999ad5"
May 25 11:28:05.493627 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:28:05.493610    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"deb363619350e414fa08f6ccbb2635e73872590f31e595c8b1658187d0999ad5\": container with ID starting with deb363619350e414fa08f6ccbb2635e73872590f31e595c8b1658187d0999ad5 not found: ID does not exist" containerID="deb363619350e414fa08f6ccbb2635e73872590f31e595c8b1658187d0999ad5"
May 25 11:28:05.493667 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:28:05.493632    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"deb363619350e414fa08f6ccbb2635e73872590f31e595c8b1658187d0999ad5"} err="failed to get container status \"deb363619350e414fa08f6ccbb2635e73872590f31e595c8b1658187d0999ad5\": rpc error: code = NotFound desc = could not find container \"deb363619350e414fa08f6ccbb2635e73872590f31e595c8b1658187d0999ad5\": container with ID starting with deb363619350e414fa08f6ccbb2635e73872590f31e595c8b1658187d0999ad5 not found: ID does not exist"
May 25 11:28:07.116991 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:28:07.116961    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3d609a8b-9235-486b-a281-5d40e5b330d9" path="/var/lib/kubelet/pods/3d609a8b-9235-486b-a281-5d40e5b330d9/volumes"
May 25 11:28:08.387493 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:28:08.387453    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5899b869f4-t7jx7" podUID="4fdf6bf7-80d8-4af3-81d7-1c9dde107499" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:5000: connect: connection refused"
May 25 11:28:08.387956 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:28:08.387932    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5899b869f4-t7jx7" podUID="4fdf6bf7-80d8-4af3-81d7-1c9dde107499" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 11:28:18.387033 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:28:18.386994    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5899b869f4-t7jx7" podUID="4fdf6bf7-80d8-4af3-81d7-1c9dde107499" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:5000: connect: connection refused"
May 25 11:28:18.387537 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:28:18.387514    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5899b869f4-t7jx7" podUID="4fdf6bf7-80d8-4af3-81d7-1c9dde107499" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 11:28:28.387200 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:28:28.387159    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5899b869f4-t7jx7" podUID="4fdf6bf7-80d8-4af3-81d7-1c9dde107499" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:5000: connect: connection refused"
May 25 11:28:28.387634 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:28:28.387609    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5899b869f4-t7jx7" podUID="4fdf6bf7-80d8-4af3-81d7-1c9dde107499" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 11:28:38.387173 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:28:38.387129    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5899b869f4-t7jx7" podUID="4fdf6bf7-80d8-4af3-81d7-1c9dde107499" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:5000: connect: connection refused"
May 25 11:28:38.387738 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:28:38.387685    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5899b869f4-t7jx7" podUID="4fdf6bf7-80d8-4af3-81d7-1c9dde107499" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 11:28:48.387384 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:28:48.387350    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5899b869f4-t7jx7"
May 25 11:28:48.387835 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:28:48.387521    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5899b869f4-t7jx7"
May 25 11:29:00.277683 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:29:00.277593    2561 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5899b869f4-t7jx7"]
May 25 11:29:00.278329 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:29:00.278241    2561 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5899b869f4-t7jx7" podUID="4fdf6bf7-80d8-4af3-81d7-1c9dde107499" containerName="kserve-container" containerID="cri-o://f11f529775c30248fb282b9aa620fd4cd608f05cef453c03e59cedc88ba0a75c" gracePeriod=30
May 25 11:29:00.278329 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:29:00.278289    2561 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5899b869f4-t7jx7" podUID="4fdf6bf7-80d8-4af3-81d7-1c9dde107499" containerName="kube-rbac-proxy" containerID="cri-o://06f546da1701c04f608494cf51b3eff5933fe908911f966947d54c84a54dcf3a" gracePeriod=30
May 25 11:29:00.278602 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:29:00.278556    2561 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5899b869f4-t7jx7" podUID="4fdf6bf7-80d8-4af3-81d7-1c9dde107499" containerName="agent" containerID="cri-o://90101b777368c00ace927ac891dc0baafcef4fdd9042fcfdd5738c1146b7a392" gracePeriod=30
May 25 11:29:00.654981 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:29:00.654944    2561 generic.go:358] "Generic (PLEG): container finished" podID="4fdf6bf7-80d8-4af3-81d7-1c9dde107499" containerID="06f546da1701c04f608494cf51b3eff5933fe908911f966947d54c84a54dcf3a" exitCode=2
May 25 11:29:00.655147 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:29:00.654984    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5899b869f4-t7jx7" event={"ID":"4fdf6bf7-80d8-4af3-81d7-1c9dde107499","Type":"ContainerDied","Data":"06f546da1701c04f608494cf51b3eff5933fe908911f966947d54c84a54dcf3a"}
May 25 11:29:03.382329 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:29:03.382243    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5899b869f4-t7jx7" podUID="4fdf6bf7-80d8-4af3-81d7-1c9dde107499" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.37:8643/healthz\": dial tcp 10.132.0.37:8643: connect: connection refused"
May 25 11:29:04.672455 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:29:04.672363    2561 generic.go:358] "Generic (PLEG): container finished" podID="4fdf6bf7-80d8-4af3-81d7-1c9dde107499" containerID="f11f529775c30248fb282b9aa620fd4cd608f05cef453c03e59cedc88ba0a75c" exitCode=0
May 25 11:29:04.672455 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:29:04.672439    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5899b869f4-t7jx7" event={"ID":"4fdf6bf7-80d8-4af3-81d7-1c9dde107499","Type":"ContainerDied","Data":"f11f529775c30248fb282b9aa620fd4cd608f05cef453c03e59cedc88ba0a75c"}
May 25 11:29:08.382217 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:29:08.382169    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5899b869f4-t7jx7" podUID="4fdf6bf7-80d8-4af3-81d7-1c9dde107499" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.37:8643/healthz\": dial tcp 10.132.0.37:8643: connect: connection refused"
May 25 11:29:08.386524 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:29:08.386493    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5899b869f4-t7jx7" podUID="4fdf6bf7-80d8-4af3-81d7-1c9dde107499" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:5000: connect: connection refused"
May 25 11:29:08.386790 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:29:08.386771    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5899b869f4-t7jx7" podUID="4fdf6bf7-80d8-4af3-81d7-1c9dde107499" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 11:29:13.382703 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:29:13.382662    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5899b869f4-t7jx7" podUID="4fdf6bf7-80d8-4af3-81d7-1c9dde107499" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.37:8643/healthz\": dial tcp 10.132.0.37:8643: connect: connection refused"
May 25 11:29:13.383161 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:29:13.382795    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5899b869f4-t7jx7"
May 25 11:29:18.382205 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:29:18.382159    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5899b869f4-t7jx7" podUID="4fdf6bf7-80d8-4af3-81d7-1c9dde107499" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.37:8643/healthz\": dial tcp 10.132.0.37:8643: connect: connection refused"
May 25 11:29:18.386480 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:29:18.386452    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5899b869f4-t7jx7" podUID="4fdf6bf7-80d8-4af3-81d7-1c9dde107499" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:5000: connect: connection refused"
May 25 11:29:18.386808 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:29:18.386784    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5899b869f4-t7jx7" podUID="4fdf6bf7-80d8-4af3-81d7-1c9dde107499" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 11:29:20.411391 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:29:20.411359    2561 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-58ff4b6bc4-mqbl4"]
May 25 11:29:20.411752 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:29:20.411742    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d609a8b-9235-486b-a281-5d40e5b330d9" containerName="kserve-container"
May 25 11:29:20.411793 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:29:20.411755    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d609a8b-9235-486b-a281-5d40e5b330d9" containerName="kserve-container"
May 25 11:29:20.411793 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:29:20.411766    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d609a8b-9235-486b-a281-5d40e5b330d9" containerName="agent"
May 25 11:29:20.411793 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:29:20.411771    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d609a8b-9235-486b-a281-5d40e5b330d9" containerName="agent"
May 25 11:29:20.411793 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:29:20.411778    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d609a8b-9235-486b-a281-5d40e5b330d9" containerName="kube-rbac-proxy"
May 25 11:29:20.411793 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:29:20.411785    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d609a8b-9235-486b-a281-5d40e5b330d9" containerName="kube-rbac-proxy"
May 25 11:29:20.411793 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:29:20.411795    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d609a8b-9235-486b-a281-5d40e5b330d9" containerName="storage-initializer"
May 25 11:29:20.411975 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:29:20.411800    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d609a8b-9235-486b-a281-5d40e5b330d9" containerName="storage-initializer"
May 25 11:29:20.411975 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:29:20.411855    2561 memory_manager.go:356] "RemoveStaleState removing state" podUID="3d609a8b-9235-486b-a281-5d40e5b330d9" containerName="kube-rbac-proxy"
May 25 11:29:20.411975 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:29:20.411863    2561 memory_manager.go:356] "RemoveStaleState removing state" podUID="3d609a8b-9235-486b-a281-5d40e5b330d9" containerName="kserve-container"
May 25 11:29:20.411975 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:29:20.411871    2561 memory_manager.go:356] "RemoveStaleState removing state" podUID="3d609a8b-9235-486b-a281-5d40e5b330d9" containerName="agent"
May 25 11:29:20.415486 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:29:20.415463    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-58ff4b6bc4-mqbl4"
May 25 11:29:20.418102 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:29:20.418069    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-predictor-serving-cert\""
May 25 11:29:20.418229 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:29:20.418170    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-kube-rbac-proxy-sar-config\""
May 25 11:29:20.426322 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:29:20.426297    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-58ff4b6bc4-mqbl4"]
May 25 11:29:20.563886 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:29:20.563824    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-logger-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4f034c8d-77f5-4e58-9314-6771e1ec46f8-isvc-logger-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-58ff4b6bc4-mqbl4\" (UID: \"4f034c8d-77f5-4e58-9314-6771e1ec46f8\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-58ff4b6bc4-mqbl4"
May 25 11:29:20.564047 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:29:20.563896    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fs6b\" (UniqueName: \"kubernetes.io/projected/4f034c8d-77f5-4e58-9314-6771e1ec46f8-kube-api-access-8fs6b\") pod \"isvc-logger-predictor-58ff4b6bc4-mqbl4\" (UID: \"4f034c8d-77f5-4e58-9314-6771e1ec46f8\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-58ff4b6bc4-mqbl4"
May 25 11:29:20.564047 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:29:20.563933    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4f034c8d-77f5-4e58-9314-6771e1ec46f8-proxy-tls\") pod \"isvc-logger-predictor-58ff4b6bc4-mqbl4\" (UID: \"4f034c8d-77f5-4e58-9314-6771e1ec46f8\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-58ff4b6bc4-mqbl4"
May 25 11:29:20.564047 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:29:20.563959    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4f034c8d-77f5-4e58-9314-6771e1ec46f8-kserve-provision-location\") pod \"isvc-logger-predictor-58ff4b6bc4-mqbl4\" (UID: \"4f034c8d-77f5-4e58-9314-6771e1ec46f8\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-58ff4b6bc4-mqbl4"
May 25 11:29:20.665361 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:29:20.665241    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-logger-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4f034c8d-77f5-4e58-9314-6771e1ec46f8-isvc-logger-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-58ff4b6bc4-mqbl4\" (UID: \"4f034c8d-77f5-4e58-9314-6771e1ec46f8\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-58ff4b6bc4-mqbl4"
May 25 11:29:20.665361 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:29:20.665319    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-8fs6b\" (UniqueName: \"kubernetes.io/projected/4f034c8d-77f5-4e58-9314-6771e1ec46f8-kube-api-access-8fs6b\") pod \"isvc-logger-predictor-58ff4b6bc4-mqbl4\" (UID: \"4f034c8d-77f5-4e58-9314-6771e1ec46f8\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-58ff4b6bc4-mqbl4"
May 25 11:29:20.665361 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:29:20.665361    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4f034c8d-77f5-4e58-9314-6771e1ec46f8-proxy-tls\") pod \"isvc-logger-predictor-58ff4b6bc4-mqbl4\" (UID: \"4f034c8d-77f5-4e58-9314-6771e1ec46f8\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-58ff4b6bc4-mqbl4"
May 25 11:29:20.665648 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:29:20.665391    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4f034c8d-77f5-4e58-9314-6771e1ec46f8-kserve-provision-location\") pod \"isvc-logger-predictor-58ff4b6bc4-mqbl4\" (UID: \"4f034c8d-77f5-4e58-9314-6771e1ec46f8\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-58ff4b6bc4-mqbl4"
May 25 11:29:20.665804 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:29:20.665775    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4f034c8d-77f5-4e58-9314-6771e1ec46f8-kserve-provision-location\") pod \"isvc-logger-predictor-58ff4b6bc4-mqbl4\" (UID: \"4f034c8d-77f5-4e58-9314-6771e1ec46f8\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-58ff4b6bc4-mqbl4"
May 25 11:29:20.666094 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:29:20.666070    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-logger-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4f034c8d-77f5-4e58-9314-6771e1ec46f8-isvc-logger-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-58ff4b6bc4-mqbl4\" (UID: \"4f034c8d-77f5-4e58-9314-6771e1ec46f8\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-58ff4b6bc4-mqbl4"
May 25 11:29:20.667769 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:29:20.667739    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4f034c8d-77f5-4e58-9314-6771e1ec46f8-proxy-tls\") pod \"isvc-logger-predictor-58ff4b6bc4-mqbl4\" (UID: \"4f034c8d-77f5-4e58-9314-6771e1ec46f8\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-58ff4b6bc4-mqbl4"
May 25 11:29:20.674459 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:29:20.674435    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fs6b\" (UniqueName: \"kubernetes.io/projected/4f034c8d-77f5-4e58-9314-6771e1ec46f8-kube-api-access-8fs6b\") pod \"isvc-logger-predictor-58ff4b6bc4-mqbl4\" (UID: \"4f034c8d-77f5-4e58-9314-6771e1ec46f8\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-58ff4b6bc4-mqbl4"
May 25 11:29:20.727597 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:29:20.727564    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-58ff4b6bc4-mqbl4"
May 25 11:29:20.865321 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:29:20.864378    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-58ff4b6bc4-mqbl4"]
May 25 11:29:21.730222 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:29:21.730186    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-58ff4b6bc4-mqbl4" event={"ID":"4f034c8d-77f5-4e58-9314-6771e1ec46f8","Type":"ContainerStarted","Data":"d648e847aa12a82774aeb5e9b52b4f70dcf6dffdeee5c448b540b786316de2fa"}
May 25 11:29:21.730222 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:29:21.730223    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-58ff4b6bc4-mqbl4" event={"ID":"4f034c8d-77f5-4e58-9314-6771e1ec46f8","Type":"ContainerStarted","Data":"9100a89d1e51f4949ddca9f829b7c9a0b83a9d95842fe2b1c98edd124a483a28"}
May 25 11:29:23.382008 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:29:23.381970    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5899b869f4-t7jx7" podUID="4fdf6bf7-80d8-4af3-81d7-1c9dde107499" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.37:8643/healthz\": dial tcp 10.132.0.37:8643: connect: connection refused"
May 25 11:29:24.740498 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:29:24.740466    2561 generic.go:358] "Generic (PLEG): container finished" podID="4f034c8d-77f5-4e58-9314-6771e1ec46f8" containerID="d648e847aa12a82774aeb5e9b52b4f70dcf6dffdeee5c448b540b786316de2fa" exitCode=0
May 25 11:29:24.740837 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:29:24.740546    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-58ff4b6bc4-mqbl4" event={"ID":"4f034c8d-77f5-4e58-9314-6771e1ec46f8","Type":"ContainerDied","Data":"d648e847aa12a82774aeb5e9b52b4f70dcf6dffdeee5c448b540b786316de2fa"}
May 25 11:29:25.746609 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:29:25.746571    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-58ff4b6bc4-mqbl4" event={"ID":"4f034c8d-77f5-4e58-9314-6771e1ec46f8","Type":"ContainerStarted","Data":"2fe6d6b7aa85977d0397945321e1d16b931f5ead979503c7146445bb06d7072d"}
May 25 11:29:25.747083 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:29:25.746619    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-58ff4b6bc4-mqbl4" event={"ID":"4f034c8d-77f5-4e58-9314-6771e1ec46f8","Type":"ContainerStarted","Data":"964bc25d94330ee03c6574b75471104618479fe6a6903ef14464d9d26f03212d"}
May 25 11:29:25.747083 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:29:25.746634    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-58ff4b6bc4-mqbl4" event={"ID":"4f034c8d-77f5-4e58-9314-6771e1ec46f8","Type":"ContainerStarted","Data":"b9c724643585568052d802c9c21df510e4fcedb438052f0bd7890a06cb5a4b97"}
May 25 11:29:25.747083 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:29:25.746917    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-58ff4b6bc4-mqbl4"
May 25 11:29:25.775089 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:29:25.775038    2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-logger-predictor-58ff4b6bc4-mqbl4" podStartSLOduration=5.775022209 podStartE2EDuration="5.775022209s" podCreationTimestamp="2026-05-25 11:29:20 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 11:29:25.773843236 +0000 UTC m=+739.319020841" watchObservedRunningTime="2026-05-25 11:29:25.775022209 +0000 UTC m=+739.320199811"
May 25 11:29:26.749889 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:29:26.749858    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-58ff4b6bc4-mqbl4"
May 25 11:29:26.750293 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:29:26.749900    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-58ff4b6bc4-mqbl4"
May 25 11:29:26.751627 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:29:26.751590    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-58ff4b6bc4-mqbl4" podUID="4f034c8d-77f5-4e58-9314-6771e1ec46f8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
May 25 11:29:26.752311 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:29:26.752288    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-58ff4b6bc4-mqbl4" podUID="4f034c8d-77f5-4e58-9314-6771e1ec46f8" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 11:29:27.753726 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:29:27.753687    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-58ff4b6bc4-mqbl4" podUID="4f034c8d-77f5-4e58-9314-6771e1ec46f8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
May 25 11:29:27.754185 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:29:27.754090    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-58ff4b6bc4-mqbl4" podUID="4f034c8d-77f5-4e58-9314-6771e1ec46f8" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 11:29:28.382315 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:29:28.382236    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5899b869f4-t7jx7" podUID="4fdf6bf7-80d8-4af3-81d7-1c9dde107499" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.37:8643/healthz\": dial tcp 10.132.0.37:8643: connect: connection refused"
May 25 11:29:28.386702 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:29:28.386661    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5899b869f4-t7jx7" podUID="4fdf6bf7-80d8-4af3-81d7-1c9dde107499" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:5000: connect: connection refused"
May 25 11:29:28.387097 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:29:28.387069    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5899b869f4-t7jx7" podUID="4fdf6bf7-80d8-4af3-81d7-1c9dde107499" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 11:29:28.387097 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:29:28.387086    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5899b869f4-t7jx7"
May 25 11:29:28.387295 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:29:28.387167    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5899b869f4-t7jx7"
May 25 11:29:30.431731 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:29:30.431707    2561 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5899b869f4-t7jx7"
May 25 11:29:30.455295 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:29:30.455169    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4fdf6bf7-80d8-4af3-81d7-1c9dde107499-proxy-tls\") pod \"4fdf6bf7-80d8-4af3-81d7-1c9dde107499\" (UID: \"4fdf6bf7-80d8-4af3-81d7-1c9dde107499\") "
May 25 11:29:30.456000 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:29:30.455457    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4fdf6bf7-80d8-4af3-81d7-1c9dde107499-isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config\") pod \"4fdf6bf7-80d8-4af3-81d7-1c9dde107499\" (UID: \"4fdf6bf7-80d8-4af3-81d7-1c9dde107499\") "
May 25 11:29:30.456000 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:29:30.455530    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6pmd5\" (UniqueName: \"kubernetes.io/projected/4fdf6bf7-80d8-4af3-81d7-1c9dde107499-kube-api-access-6pmd5\") pod \"4fdf6bf7-80d8-4af3-81d7-1c9dde107499\" (UID: \"4fdf6bf7-80d8-4af3-81d7-1c9dde107499\") "
May 25 11:29:30.456000 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:29:30.455615    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4fdf6bf7-80d8-4af3-81d7-1c9dde107499-kserve-provision-location\") pod \"4fdf6bf7-80d8-4af3-81d7-1c9dde107499\" (UID: \"4fdf6bf7-80d8-4af3-81d7-1c9dde107499\") "
May 25 11:29:30.456000 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:29:30.455797    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4fdf6bf7-80d8-4af3-81d7-1c9dde107499-isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config") pod "4fdf6bf7-80d8-4af3-81d7-1c9dde107499" (UID: "4fdf6bf7-80d8-4af3-81d7-1c9dde107499"). InnerVolumeSpecName "isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 11:29:30.456000 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:29:30.455906    2561 reconciler_common.go:299] "Volume detached for volume \"isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4fdf6bf7-80d8-4af3-81d7-1c9dde107499-isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-227.ec2.internal\" DevicePath \"\""
May 25 11:29:30.456450 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:29:30.456420    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fdf6bf7-80d8-4af3-81d7-1c9dde107499-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4fdf6bf7-80d8-4af3-81d7-1c9dde107499" (UID: "4fdf6bf7-80d8-4af3-81d7-1c9dde107499"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:29:30.457680 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:29:30.457646    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4fdf6bf7-80d8-4af3-81d7-1c9dde107499-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4fdf6bf7-80d8-4af3-81d7-1c9dde107499" (UID: "4fdf6bf7-80d8-4af3-81d7-1c9dde107499"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 11:29:30.457991 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:29:30.457971    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fdf6bf7-80d8-4af3-81d7-1c9dde107499-kube-api-access-6pmd5" (OuterVolumeSpecName: "kube-api-access-6pmd5") pod "4fdf6bf7-80d8-4af3-81d7-1c9dde107499" (UID: "4fdf6bf7-80d8-4af3-81d7-1c9dde107499"). InnerVolumeSpecName "kube-api-access-6pmd5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 11:29:30.557106 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:29:30.557022    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6pmd5\" (UniqueName: \"kubernetes.io/projected/4fdf6bf7-80d8-4af3-81d7-1c9dde107499-kube-api-access-6pmd5\") on node \"ip-10-0-136-227.ec2.internal\" DevicePath \"\""
May 25 11:29:30.557106 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:29:30.557051    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4fdf6bf7-80d8-4af3-81d7-1c9dde107499-kserve-provision-location\") on node \"ip-10-0-136-227.ec2.internal\" DevicePath \"\""
May 25 11:29:30.557106 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:29:30.557061    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4fdf6bf7-80d8-4af3-81d7-1c9dde107499-proxy-tls\") on node \"ip-10-0-136-227.ec2.internal\" DevicePath \"\""
May 25 11:29:30.768861 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:29:30.768827    2561 generic.go:358] "Generic (PLEG): container finished" podID="4fdf6bf7-80d8-4af3-81d7-1c9dde107499" containerID="90101b777368c00ace927ac891dc0baafcef4fdd9042fcfdd5738c1146b7a392" exitCode=0
May 25 11:29:30.769061 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:29:30.768916    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5899b869f4-t7jx7" event={"ID":"4fdf6bf7-80d8-4af3-81d7-1c9dde107499","Type":"ContainerDied","Data":"90101b777368c00ace927ac891dc0baafcef4fdd9042fcfdd5738c1146b7a392"}
May 25 11:29:30.769061 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:29:30.768941    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5899b869f4-t7jx7" event={"ID":"4fdf6bf7-80d8-4af3-81d7-1c9dde107499","Type":"ContainerDied","Data":"038d0b75d474e13b828d9d7503926d9039f519ee66b9c243175f752434d6a700"}
May 25 11:29:30.769061 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:29:30.768964    2561 scope.go:117] "RemoveContainer" containerID="90101b777368c00ace927ac891dc0baafcef4fdd9042fcfdd5738c1146b7a392"
May 25 11:29:30.769061 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:29:30.768996    2561 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5899b869f4-t7jx7"
May 25 11:29:30.777486 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:29:30.777468    2561 scope.go:117] "RemoveContainer" containerID="06f546da1701c04f608494cf51b3eff5933fe908911f966947d54c84a54dcf3a"
May 25 11:29:30.785008 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:29:30.784992    2561 scope.go:117] "RemoveContainer" containerID="f11f529775c30248fb282b9aa620fd4cd608f05cef453c03e59cedc88ba0a75c"
May 25 11:29:30.792859 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:29:30.792836    2561 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5899b869f4-t7jx7"]
May 25 11:29:30.793100 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:29:30.793081    2561 scope.go:117] "RemoveContainer" containerID="87b9909983bb29fc43b15558528e0f6970bd2558853ab06f9495d8a3355c2888"
May 25 11:29:30.797686 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:29:30.797664    2561 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5899b869f4-t7jx7"]
May 25 11:29:30.800863 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:29:30.800848    2561 scope.go:117] "RemoveContainer" containerID="90101b777368c00ace927ac891dc0baafcef4fdd9042fcfdd5738c1146b7a392"
May 25 11:29:30.801067 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:29:30.801050    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90101b777368c00ace927ac891dc0baafcef4fdd9042fcfdd5738c1146b7a392\": container with ID starting with 90101b777368c00ace927ac891dc0baafcef4fdd9042fcfdd5738c1146b7a392 not found: ID does not exist" containerID="90101b777368c00ace927ac891dc0baafcef4fdd9042fcfdd5738c1146b7a392"
May 25 11:29:30.801106 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:29:30.801075    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90101b777368c00ace927ac891dc0baafcef4fdd9042fcfdd5738c1146b7a392"} err="failed to get container status \"90101b777368c00ace927ac891dc0baafcef4fdd9042fcfdd5738c1146b7a392\": rpc error: code = NotFound desc = could not find container \"90101b777368c00ace927ac891dc0baafcef4fdd9042fcfdd5738c1146b7a392\": container with ID starting with 90101b777368c00ace927ac891dc0baafcef4fdd9042fcfdd5738c1146b7a392 not found: ID does not exist"
May 25 11:29:30.801106 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:29:30.801096    2561 scope.go:117] "RemoveContainer" containerID="06f546da1701c04f608494cf51b3eff5933fe908911f966947d54c84a54dcf3a"
May 25 11:29:30.801350 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:29:30.801313    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06f546da1701c04f608494cf51b3eff5933fe908911f966947d54c84a54dcf3a\": container with ID starting with 06f546da1701c04f608494cf51b3eff5933fe908911f966947d54c84a54dcf3a not found: ID does not exist" containerID="06f546da1701c04f608494cf51b3eff5933fe908911f966947d54c84a54dcf3a"
May 25 11:29:30.801400 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:29:30.801355    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06f546da1701c04f608494cf51b3eff5933fe908911f966947d54c84a54dcf3a"} err="failed to get container status \"06f546da1701c04f608494cf51b3eff5933fe908911f966947d54c84a54dcf3a\": rpc error: code = NotFound desc = could not find container \"06f546da1701c04f608494cf51b3eff5933fe908911f966947d54c84a54dcf3a\": container with ID starting with 06f546da1701c04f608494cf51b3eff5933fe908911f966947d54c84a54dcf3a not found: ID does not exist"
May 25 11:29:30.801400 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:29:30.801368    2561 scope.go:117] "RemoveContainer" containerID="f11f529775c30248fb282b9aa620fd4cd608f05cef453c03e59cedc88ba0a75c"
May 25 11:29:30.801589 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:29:30.801572    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f11f529775c30248fb282b9aa620fd4cd608f05cef453c03e59cedc88ba0a75c\": container with ID starting with f11f529775c30248fb282b9aa620fd4cd608f05cef453c03e59cedc88ba0a75c not found: ID does not exist" containerID="f11f529775c30248fb282b9aa620fd4cd608f05cef453c03e59cedc88ba0a75c"
May 25 11:29:30.801631 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:29:30.801596    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f11f529775c30248fb282b9aa620fd4cd608f05cef453c03e59cedc88ba0a75c"} err="failed to get container status \"f11f529775c30248fb282b9aa620fd4cd608f05cef453c03e59cedc88ba0a75c\": rpc error: code = NotFound desc = could not find container \"f11f529775c30248fb282b9aa620fd4cd608f05cef453c03e59cedc88ba0a75c\": container with ID starting with f11f529775c30248fb282b9aa620fd4cd608f05cef453c03e59cedc88ba0a75c not found: ID does not exist"
May 25 11:29:30.801631 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:29:30.801612    2561 scope.go:117] "RemoveContainer" containerID="87b9909983bb29fc43b15558528e0f6970bd2558853ab06f9495d8a3355c2888"
May 25 11:29:30.801833 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:29:30.801816    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87b9909983bb29fc43b15558528e0f6970bd2558853ab06f9495d8a3355c2888\": container with ID starting with 87b9909983bb29fc43b15558528e0f6970bd2558853ab06f9495d8a3355c2888 not found: ID does not exist" containerID="87b9909983bb29fc43b15558528e0f6970bd2558853ab06f9495d8a3355c2888"
May 25 11:29:30.801885 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:29:30.801836    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87b9909983bb29fc43b15558528e0f6970bd2558853ab06f9495d8a3355c2888"} err="failed to get container status \"87b9909983bb29fc43b15558528e0f6970bd2558853ab06f9495d8a3355c2888\": rpc error: code = NotFound desc = could not find container \"87b9909983bb29fc43b15558528e0f6970bd2558853ab06f9495d8a3355c2888\": container with ID starting with 87b9909983bb29fc43b15558528e0f6970bd2558853ab06f9495d8a3355c2888 not found: ID does not exist"
May 25 11:29:31.118434 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:29:31.118391    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4fdf6bf7-80d8-4af3-81d7-1c9dde107499" path="/var/lib/kubelet/pods/4fdf6bf7-80d8-4af3-81d7-1c9dde107499/volumes"
May 25 11:29:32.758244 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:29:32.758217    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-58ff4b6bc4-mqbl4"
May 25 11:29:32.758792 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:29:32.758753    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-58ff4b6bc4-mqbl4" podUID="4f034c8d-77f5-4e58-9314-6771e1ec46f8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
May 25 11:29:32.759195 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:29:32.759172    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-58ff4b6bc4-mqbl4" podUID="4f034c8d-77f5-4e58-9314-6771e1ec46f8" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 11:29:42.759232 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:29:42.759186    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-58ff4b6bc4-mqbl4" podUID="4f034c8d-77f5-4e58-9314-6771e1ec46f8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
May 25 11:29:42.759674 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:29:42.759615    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-58ff4b6bc4-mqbl4" podUID="4f034c8d-77f5-4e58-9314-6771e1ec46f8" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 11:29:52.759013 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:29:52.758968    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-58ff4b6bc4-mqbl4" podUID="4f034c8d-77f5-4e58-9314-6771e1ec46f8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
May 25 11:29:52.759606 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:29:52.759510    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-58ff4b6bc4-mqbl4" podUID="4f034c8d-77f5-4e58-9314-6771e1ec46f8" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 11:30:02.759519 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:30:02.759474    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-58ff4b6bc4-mqbl4" podUID="4f034c8d-77f5-4e58-9314-6771e1ec46f8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
May 25 11:30:02.760023 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:30:02.759953    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-58ff4b6bc4-mqbl4" podUID="4f034c8d-77f5-4e58-9314-6771e1ec46f8" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 11:30:12.758742 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:30:12.758699    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-58ff4b6bc4-mqbl4" podUID="4f034c8d-77f5-4e58-9314-6771e1ec46f8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
May 25 11:30:12.759175 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:30:12.759153    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-58ff4b6bc4-mqbl4" podUID="4f034c8d-77f5-4e58-9314-6771e1ec46f8" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 11:30:22.759370 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:30:22.759328    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-58ff4b6bc4-mqbl4" podUID="4f034c8d-77f5-4e58-9314-6771e1ec46f8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
May 25 11:30:22.759790 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:30:22.759761    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-58ff4b6bc4-mqbl4" podUID="4f034c8d-77f5-4e58-9314-6771e1ec46f8" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 11:30:32.760339 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:30:32.760248    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-58ff4b6bc4-mqbl4"
May 25 11:30:32.760843 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:30:32.760494    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-58ff4b6bc4-mqbl4"
May 25 11:30:45.814686 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:30:45.814651    2561 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-58ff4b6bc4-mqbl4"]
May 25 11:30:45.815071 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:30:45.815036    2561 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-58ff4b6bc4-mqbl4" podUID="4f034c8d-77f5-4e58-9314-6771e1ec46f8" containerName="kserve-container" containerID="cri-o://b9c724643585568052d802c9c21df510e4fcedb438052f0bd7890a06cb5a4b97" gracePeriod=30
May 25 11:30:45.815144 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:30:45.815070    2561 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-58ff4b6bc4-mqbl4" podUID="4f034c8d-77f5-4e58-9314-6771e1ec46f8" containerName="agent" containerID="cri-o://2fe6d6b7aa85977d0397945321e1d16b931f5ead979503c7146445bb06d7072d" gracePeriod=30
May 25 11:30:45.815144 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:30:45.815070    2561 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-58ff4b6bc4-mqbl4" podUID="4f034c8d-77f5-4e58-9314-6771e1ec46f8" containerName="kube-rbac-proxy" containerID="cri-o://964bc25d94330ee03c6574b75471104618479fe6a6903ef14464d9d26f03212d" gracePeriod=30
May 25 11:30:45.846106 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:30:45.846075    2561 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-w9l9h"]
May 25 11:30:45.846469 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:30:45.846456    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4fdf6bf7-80d8-4af3-81d7-1c9dde107499" containerName="agent"
May 25 11:30:45.846532 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:30:45.846471    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="4fdf6bf7-80d8-4af3-81d7-1c9dde107499" containerName="agent"
May 25 11:30:45.846532 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:30:45.846480    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4fdf6bf7-80d8-4af3-81d7-1c9dde107499" containerName="kserve-container"
May 25 11:30:45.846532 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:30:45.846486    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="4fdf6bf7-80d8-4af3-81d7-1c9dde107499" containerName="kserve-container"
May 25 11:30:45.846532 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:30:45.846501    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4fdf6bf7-80d8-4af3-81d7-1c9dde107499" containerName="storage-initializer"
May 25 11:30:45.846532 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:30:45.846506    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="4fdf6bf7-80d8-4af3-81d7-1c9dde107499" containerName="storage-initializer"
May 25 11:30:45.846532 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:30:45.846518    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4fdf6bf7-80d8-4af3-81d7-1c9dde107499" containerName="kube-rbac-proxy"
May 25 11:30:45.846532 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:30:45.846523    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="4fdf6bf7-80d8-4af3-81d7-1c9dde107499" containerName="kube-rbac-proxy"
May 25 11:30:45.846742 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:30:45.846589    2561 memory_manager.go:356] "RemoveStaleState removing state" podUID="4fdf6bf7-80d8-4af3-81d7-1c9dde107499" containerName="kube-rbac-proxy"
May 25 11:30:45.846742 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:30:45.846598    2561 memory_manager.go:356] "RemoveStaleState removing state" podUID="4fdf6bf7-80d8-4af3-81d7-1c9dde107499" containerName="kserve-container"
May 25 11:30:45.846742 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:30:45.846606    2561 memory_manager.go:356] "RemoveStaleState removing state" podUID="4fdf6bf7-80d8-4af3-81d7-1c9dde107499" containerName="agent"
May 25 11:30:45.850022 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:30:45.850003    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-w9l9h"
May 25 11:30:45.852459 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:30:45.852433    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-kube-rbac-proxy-sar-config\""
May 25 11:30:45.852568 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:30:45.852471    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-predictor-serving-cert\""
May 25 11:30:45.860599 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:30:45.860576    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-w9l9h"]
May 25 11:30:45.966062 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:30:45.966035    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cbf3dfb7-c560-4f58-a46f-65e5e3eb38aa-kserve-provision-location\") pod \"isvc-lightgbm-predictor-bdf964bd-w9l9h\" (UID: \"cbf3dfb7-c560-4f58-a46f-65e5e3eb38aa\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-w9l9h"
May 25 11:30:45.966203 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:30:45.966167    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vld7\" (UniqueName: \"kubernetes.io/projected/cbf3dfb7-c560-4f58-a46f-65e5e3eb38aa-kube-api-access-4vld7\") pod \"isvc-lightgbm-predictor-bdf964bd-w9l9h\" (UID: \"cbf3dfb7-c560-4f58-a46f-65e5e3eb38aa\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-w9l9h"
May 25 11:30:45.966305 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:30:45.966230    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-lightgbm-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cbf3dfb7-c560-4f58-a46f-65e5e3eb38aa-isvc-lightgbm-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-predictor-bdf964bd-w9l9h\" (UID: \"cbf3dfb7-c560-4f58-a46f-65e5e3eb38aa\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-w9l9h"
May 25 11:30:45.966370 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:30:45.966306    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cbf3dfb7-c560-4f58-a46f-65e5e3eb38aa-proxy-tls\") pod \"isvc-lightgbm-predictor-bdf964bd-w9l9h\" (UID: \"cbf3dfb7-c560-4f58-a46f-65e5e3eb38aa\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-w9l9h"
May 25 11:30:46.027583 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:30:46.027557    2561 generic.go:358] "Generic (PLEG): container finished" podID="4f034c8d-77f5-4e58-9314-6771e1ec46f8" containerID="964bc25d94330ee03c6574b75471104618479fe6a6903ef14464d9d26f03212d" exitCode=2
May 25 11:30:46.027696 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:30:46.027624    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-58ff4b6bc4-mqbl4" event={"ID":"4f034c8d-77f5-4e58-9314-6771e1ec46f8","Type":"ContainerDied","Data":"964bc25d94330ee03c6574b75471104618479fe6a6903ef14464d9d26f03212d"}
May 25 11:30:46.067132 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:30:46.067109    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-4vld7\" (UniqueName: \"kubernetes.io/projected/cbf3dfb7-c560-4f58-a46f-65e5e3eb38aa-kube-api-access-4vld7\") pod \"isvc-lightgbm-predictor-bdf964bd-w9l9h\" (UID: \"cbf3dfb7-c560-4f58-a46f-65e5e3eb38aa\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-w9l9h"
May 25 11:30:46.067214 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:30:46.067145    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-lightgbm-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cbf3dfb7-c560-4f58-a46f-65e5e3eb38aa-isvc-lightgbm-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-predictor-bdf964bd-w9l9h\" (UID: \"cbf3dfb7-c560-4f58-a46f-65e5e3eb38aa\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-w9l9h"
May 25 11:30:46.067214 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:30:46.067200    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cbf3dfb7-c560-4f58-a46f-65e5e3eb38aa-proxy-tls\") pod \"isvc-lightgbm-predictor-bdf964bd-w9l9h\" (UID: \"cbf3dfb7-c560-4f58-a46f-65e5e3eb38aa\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-w9l9h"
May 25 11:30:46.067351 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:30:46.067244    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cbf3dfb7-c560-4f58-a46f-65e5e3eb38aa-kserve-provision-location\") pod \"isvc-lightgbm-predictor-bdf964bd-w9l9h\" (UID: \"cbf3dfb7-c560-4f58-a46f-65e5e3eb38aa\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-w9l9h"
May 25 11:30:46.067630 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:30:46.067606    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cbf3dfb7-c560-4f58-a46f-65e5e3eb38aa-kserve-provision-location\") pod \"isvc-lightgbm-predictor-bdf964bd-w9l9h\" (UID: \"cbf3dfb7-c560-4f58-a46f-65e5e3eb38aa\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-w9l9h"
May 25 11:30:46.067815 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:30:46.067793    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-lightgbm-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cbf3dfb7-c560-4f58-a46f-65e5e3eb38aa-isvc-lightgbm-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-predictor-bdf964bd-w9l9h\" (UID: \"cbf3dfb7-c560-4f58-a46f-65e5e3eb38aa\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-w9l9h"
May 25 11:30:46.069625 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:30:46.069605    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cbf3dfb7-c560-4f58-a46f-65e5e3eb38aa-proxy-tls\") pod \"isvc-lightgbm-predictor-bdf964bd-w9l9h\" (UID: \"cbf3dfb7-c560-4f58-a46f-65e5e3eb38aa\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-w9l9h"
May 25 11:30:46.075810 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:30:46.075783    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vld7\" (UniqueName: \"kubernetes.io/projected/cbf3dfb7-c560-4f58-a46f-65e5e3eb38aa-kube-api-access-4vld7\") pod \"isvc-lightgbm-predictor-bdf964bd-w9l9h\" (UID: \"cbf3dfb7-c560-4f58-a46f-65e5e3eb38aa\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-w9l9h"
May 25 11:30:46.160050 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:30:46.160030    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-w9l9h"
May 25 11:30:46.310751 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:30:46.310728    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-w9l9h"]
May 25 11:30:46.313257 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:30:46.313231    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbf3dfb7_c560_4f58_a46f_65e5e3eb38aa.slice/crio-79d90de54b6727715b3b477642b8111c89000bbaca4247444e9c708fa6973e31 WatchSource:0}: Error finding container 79d90de54b6727715b3b477642b8111c89000bbaca4247444e9c708fa6973e31: Status 404 returned error can't find the container with id 79d90de54b6727715b3b477642b8111c89000bbaca4247444e9c708fa6973e31
May 25 11:30:47.032044 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:30:47.032009    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-w9l9h" event={"ID":"cbf3dfb7-c560-4f58-a46f-65e5e3eb38aa","Type":"ContainerStarted","Data":"457a81bd72cd61bee5383aac067bc1de93087715bceda1dc4515987f353f7e49"}
May 25 11:30:47.032044 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:30:47.032048    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-w9l9h" event={"ID":"cbf3dfb7-c560-4f58-a46f-65e5e3eb38aa","Type":"ContainerStarted","Data":"79d90de54b6727715b3b477642b8111c89000bbaca4247444e9c708fa6973e31"}
May 25 11:30:47.754406 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:30:47.754364    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-58ff4b6bc4-mqbl4" podUID="4f034c8d-77f5-4e58-9314-6771e1ec46f8" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.38:8643/healthz\": dial tcp 10.132.0.38:8643: connect: connection refused"
May 25 11:30:50.044245 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:30:50.044218    2561 generic.go:358] "Generic (PLEG): container finished" podID="4f034c8d-77f5-4e58-9314-6771e1ec46f8" containerID="b9c724643585568052d802c9c21df510e4fcedb438052f0bd7890a06cb5a4b97" exitCode=0
May 25 11:30:50.044695 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:30:50.044256    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-58ff4b6bc4-mqbl4" event={"ID":"4f034c8d-77f5-4e58-9314-6771e1ec46f8","Type":"ContainerDied","Data":"b9c724643585568052d802c9c21df510e4fcedb438052f0bd7890a06cb5a4b97"}
May 25 11:30:51.049263 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:30:51.049226    2561 generic.go:358] "Generic (PLEG): container finished" podID="cbf3dfb7-c560-4f58-a46f-65e5e3eb38aa" containerID="457a81bd72cd61bee5383aac067bc1de93087715bceda1dc4515987f353f7e49" exitCode=0
May 25 11:30:51.049633 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:30:51.049291    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-w9l9h" event={"ID":"cbf3dfb7-c560-4f58-a46f-65e5e3eb38aa","Type":"ContainerDied","Data":"457a81bd72cd61bee5383aac067bc1de93087715bceda1dc4515987f353f7e49"}
May 25 11:30:52.754233 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:30:52.754182    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-58ff4b6bc4-mqbl4" podUID="4f034c8d-77f5-4e58-9314-6771e1ec46f8" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.38:8643/healthz\": dial tcp 10.132.0.38:8643: connect: connection refused"
May 25 11:30:52.759655 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:30:52.759615    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-58ff4b6bc4-mqbl4" podUID="4f034c8d-77f5-4e58-9314-6771e1ec46f8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
May 25 11:30:52.760256 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:30:52.760178    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-58ff4b6bc4-mqbl4" podUID="4f034c8d-77f5-4e58-9314-6771e1ec46f8" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 11:30:57.754629 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:30:57.754585    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-58ff4b6bc4-mqbl4" podUID="4f034c8d-77f5-4e58-9314-6771e1ec46f8" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.38:8643/healthz\": dial tcp 10.132.0.38:8643: connect: connection refused"
May 25 11:30:57.755052 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:30:57.754743    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-58ff4b6bc4-mqbl4"
May 25 11:30:58.081606 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:30:58.081577    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-w9l9h" event={"ID":"cbf3dfb7-c560-4f58-a46f-65e5e3eb38aa","Type":"ContainerStarted","Data":"b6fb06873cd8ee6f841568831a02cc8fe5850075e651196839e1f8804b8f05b6"}
May 25 11:30:58.081756 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:30:58.081613    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-w9l9h" event={"ID":"cbf3dfb7-c560-4f58-a46f-65e5e3eb38aa","Type":"ContainerStarted","Data":"b5ddb399ad304a3f3bd484c6b93e31c7d8571540c930c6ab8687942f994517da"}
May 25 11:30:58.081874 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:30:58.081856    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-w9l9h"
May 25 11:30:58.102698 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:30:58.102651    2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-w9l9h" podStartSLOduration=6.90793591 podStartE2EDuration="13.102640181s" podCreationTimestamp="2026-05-25 11:30:45 +0000 UTC" firstStartedPulling="2026-05-25 11:30:51.050497193 +0000 UTC m=+824.595674774" lastFinishedPulling="2026-05-25 11:30:57.245201449 +0000 UTC m=+830.790379045" observedRunningTime="2026-05-25 11:30:58.101312891 +0000 UTC m=+831.646490495" watchObservedRunningTime="2026-05-25 11:30:58.102640181 +0000 UTC m=+831.647817783"
May 25 11:30:59.085056 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:30:59.085026    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-w9l9h"
May 25 11:30:59.086320 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:30:59.086290    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-w9l9h" podUID="cbf3dfb7-c560-4f58-a46f-65e5e3eb38aa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
May 25 11:31:00.087934 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:31:00.087893    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-w9l9h" podUID="cbf3dfb7-c560-4f58-a46f-65e5e3eb38aa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
May 25 11:31:02.754515 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:31:02.754471    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-58ff4b6bc4-mqbl4" podUID="4f034c8d-77f5-4e58-9314-6771e1ec46f8" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.38:8643/healthz\": dial tcp 10.132.0.38:8643: connect: connection refused"
May 25 11:31:02.758841 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:31:02.758814    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-58ff4b6bc4-mqbl4" podUID="4f034c8d-77f5-4e58-9314-6771e1ec46f8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
May 25 11:31:02.759163 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:31:02.759139    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-58ff4b6bc4-mqbl4" podUID="4f034c8d-77f5-4e58-9314-6771e1ec46f8" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 11:31:05.092510 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:31:05.092479    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-w9l9h"
May 25 11:31:05.093154 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:31:05.093123    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-w9l9h" podUID="cbf3dfb7-c560-4f58-a46f-65e5e3eb38aa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
May 25 11:31:07.754801 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:31:07.754758    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-58ff4b6bc4-mqbl4" podUID="4f034c8d-77f5-4e58-9314-6771e1ec46f8" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.38:8643/healthz\": dial tcp 10.132.0.38:8643: connect: connection refused"
May 25 11:31:12.754523 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:31:12.754483    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-58ff4b6bc4-mqbl4" podUID="4f034c8d-77f5-4e58-9314-6771e1ec46f8" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.38:8643/healthz\": dial tcp 10.132.0.38:8643: connect: connection refused"
May 25 11:31:12.758810 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:31:12.758784    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-58ff4b6bc4-mqbl4" podUID="4f034c8d-77f5-4e58-9314-6771e1ec46f8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
May 25 11:31:12.758939 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:31:12.758918    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-58ff4b6bc4-mqbl4"
May 25 11:31:12.759154 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:31:12.759125    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-58ff4b6bc4-mqbl4" podUID="4f034c8d-77f5-4e58-9314-6771e1ec46f8" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 11:31:12.759243 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:31:12.759201    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-58ff4b6bc4-mqbl4"
May 25 11:31:15.093190 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:31:15.093150    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-w9l9h" podUID="cbf3dfb7-c560-4f58-a46f-65e5e3eb38aa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
May 25 11:31:15.977343 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:31:15.977320    2561 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-58ff4b6bc4-mqbl4"
May 25 11:31:16.011590 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:31:16.011565    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8fs6b\" (UniqueName: \"kubernetes.io/projected/4f034c8d-77f5-4e58-9314-6771e1ec46f8-kube-api-access-8fs6b\") pod \"4f034c8d-77f5-4e58-9314-6771e1ec46f8\" (UID: \"4f034c8d-77f5-4e58-9314-6771e1ec46f8\") "
May 25 11:31:16.011728 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:31:16.011619    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-logger-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4f034c8d-77f5-4e58-9314-6771e1ec46f8-isvc-logger-kube-rbac-proxy-sar-config\") pod \"4f034c8d-77f5-4e58-9314-6771e1ec46f8\" (UID: \"4f034c8d-77f5-4e58-9314-6771e1ec46f8\") "
May 25 11:31:16.011728 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:31:16.011663    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4f034c8d-77f5-4e58-9314-6771e1ec46f8-kserve-provision-location\") pod \"4f034c8d-77f5-4e58-9314-6771e1ec46f8\" (UID: \"4f034c8d-77f5-4e58-9314-6771e1ec46f8\") "
May 25 11:31:16.011860 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:31:16.011735    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4f034c8d-77f5-4e58-9314-6771e1ec46f8-proxy-tls\") pod \"4f034c8d-77f5-4e58-9314-6771e1ec46f8\" (UID: \"4f034c8d-77f5-4e58-9314-6771e1ec46f8\") "
May 25 11:31:16.012154 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:31:16.012125    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f034c8d-77f5-4e58-9314-6771e1ec46f8-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4f034c8d-77f5-4e58-9314-6771e1ec46f8" (UID: "4f034c8d-77f5-4e58-9314-6771e1ec46f8"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:31:16.012331 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:31:16.012309    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f034c8d-77f5-4e58-9314-6771e1ec46f8-isvc-logger-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-logger-kube-rbac-proxy-sar-config") pod "4f034c8d-77f5-4e58-9314-6771e1ec46f8" (UID: "4f034c8d-77f5-4e58-9314-6771e1ec46f8"). InnerVolumeSpecName "isvc-logger-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 11:31:16.013723 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:31:16.013689    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f034c8d-77f5-4e58-9314-6771e1ec46f8-kube-api-access-8fs6b" (OuterVolumeSpecName: "kube-api-access-8fs6b") pod "4f034c8d-77f5-4e58-9314-6771e1ec46f8" (UID: "4f034c8d-77f5-4e58-9314-6771e1ec46f8"). InnerVolumeSpecName "kube-api-access-8fs6b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 11:31:16.013846 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:31:16.013829    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f034c8d-77f5-4e58-9314-6771e1ec46f8-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4f034c8d-77f5-4e58-9314-6771e1ec46f8" (UID: "4f034c8d-77f5-4e58-9314-6771e1ec46f8"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 11:31:16.112633 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:31:16.112564    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8fs6b\" (UniqueName: \"kubernetes.io/projected/4f034c8d-77f5-4e58-9314-6771e1ec46f8-kube-api-access-8fs6b\") on node \"ip-10-0-136-227.ec2.internal\" DevicePath \"\""
May 25 11:31:16.112633 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:31:16.112601    2561 reconciler_common.go:299] "Volume detached for volume \"isvc-logger-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4f034c8d-77f5-4e58-9314-6771e1ec46f8-isvc-logger-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-227.ec2.internal\" DevicePath \"\""
May 25 11:31:16.112633 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:31:16.112613    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4f034c8d-77f5-4e58-9314-6771e1ec46f8-kserve-provision-location\") on node \"ip-10-0-136-227.ec2.internal\" DevicePath \"\""
May 25 11:31:16.112633 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:31:16.112622    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4f034c8d-77f5-4e58-9314-6771e1ec46f8-proxy-tls\") on node \"ip-10-0-136-227.ec2.internal\" DevicePath \"\""
May 25 11:31:16.152550 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:31:16.152524    2561 generic.go:358] "Generic (PLEG): container finished" podID="4f034c8d-77f5-4e58-9314-6771e1ec46f8" containerID="2fe6d6b7aa85977d0397945321e1d16b931f5ead979503c7146445bb06d7072d" exitCode=0
May 25 11:31:16.152657 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:31:16.152556    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-58ff4b6bc4-mqbl4" event={"ID":"4f034c8d-77f5-4e58-9314-6771e1ec46f8","Type":"ContainerDied","Data":"2fe6d6b7aa85977d0397945321e1d16b931f5ead979503c7146445bb06d7072d"}
May 25 11:31:16.152657 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:31:16.152593    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-58ff4b6bc4-mqbl4" event={"ID":"4f034c8d-77f5-4e58-9314-6771e1ec46f8","Type":"ContainerDied","Data":"9100a89d1e51f4949ddca9f829b7c9a0b83a9d95842fe2b1c98edd124a483a28"}
May 25 11:31:16.152657 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:31:16.152611    2561 scope.go:117] "RemoveContainer" containerID="2fe6d6b7aa85977d0397945321e1d16b931f5ead979503c7146445bb06d7072d"
May 25 11:31:16.152657 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:31:16.152634    2561 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-58ff4b6bc4-mqbl4"
May 25 11:31:16.160742 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:31:16.160707    2561 scope.go:117] "RemoveContainer" containerID="964bc25d94330ee03c6574b75471104618479fe6a6903ef14464d9d26f03212d"
May 25 11:31:16.168020 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:31:16.168005    2561 scope.go:117] "RemoveContainer" containerID="b9c724643585568052d802c9c21df510e4fcedb438052f0bd7890a06cb5a4b97"
May 25 11:31:16.175764 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:31:16.175747    2561 scope.go:117] "RemoveContainer" containerID="d648e847aa12a82774aeb5e9b52b4f70dcf6dffdeee5c448b540b786316de2fa"
May 25 11:31:16.176177 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:31:16.176158    2561 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-58ff4b6bc4-mqbl4"]
May 25 11:31:16.183237 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:31:16.183212    2561 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-58ff4b6bc4-mqbl4"]
May 25 11:31:16.183618 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:31:16.183602    2561 scope.go:117] "RemoveContainer" containerID="2fe6d6b7aa85977d0397945321e1d16b931f5ead979503c7146445bb06d7072d"
May 25 11:31:16.183850 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:31:16.183831    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2fe6d6b7aa85977d0397945321e1d16b931f5ead979503c7146445bb06d7072d\": container with ID starting with 2fe6d6b7aa85977d0397945321e1d16b931f5ead979503c7146445bb06d7072d not found: ID does not exist" containerID="2fe6d6b7aa85977d0397945321e1d16b931f5ead979503c7146445bb06d7072d"
May 25 11:31:16.183921 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:31:16.183863    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2fe6d6b7aa85977d0397945321e1d16b931f5ead979503c7146445bb06d7072d"} err="failed to get container status \"2fe6d6b7aa85977d0397945321e1d16b931f5ead979503c7146445bb06d7072d\": rpc error: code = NotFound desc = could not find container \"2fe6d6b7aa85977d0397945321e1d16b931f5ead979503c7146445bb06d7072d\": container with ID starting with 2fe6d6b7aa85977d0397945321e1d16b931f5ead979503c7146445bb06d7072d not found: ID does not exist"
May 25 11:31:16.183921 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:31:16.183887    2561 scope.go:117] "RemoveContainer" containerID="964bc25d94330ee03c6574b75471104618479fe6a6903ef14464d9d26f03212d"
May 25 11:31:16.184124 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:31:16.184106    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"964bc25d94330ee03c6574b75471104618479fe6a6903ef14464d9d26f03212d\": container with ID starting with 964bc25d94330ee03c6574b75471104618479fe6a6903ef14464d9d26f03212d not found: ID does not exist" containerID="964bc25d94330ee03c6574b75471104618479fe6a6903ef14464d9d26f03212d"
May 25 11:31:16.184167 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:31:16.184131    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"964bc25d94330ee03c6574b75471104618479fe6a6903ef14464d9d26f03212d"} err="failed to get container status \"964bc25d94330ee03c6574b75471104618479fe6a6903ef14464d9d26f03212d\": rpc error: code = NotFound desc = could not find container \"964bc25d94330ee03c6574b75471104618479fe6a6903ef14464d9d26f03212d\": container with ID starting with 964bc25d94330ee03c6574b75471104618479fe6a6903ef14464d9d26f03212d not found: ID does not exist"
May 25 11:31:16.184167 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:31:16.184150    2561 scope.go:117] "RemoveContainer" containerID="b9c724643585568052d802c9c21df510e4fcedb438052f0bd7890a06cb5a4b97"
May 25 11:31:16.184343 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:31:16.184330    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9c724643585568052d802c9c21df510e4fcedb438052f0bd7890a06cb5a4b97\": container with ID starting with b9c724643585568052d802c9c21df510e4fcedb438052f0bd7890a06cb5a4b97 not found: ID does not exist" containerID="b9c724643585568052d802c9c21df510e4fcedb438052f0bd7890a06cb5a4b97"
May 25 11:31:16.184397 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:31:16.184346    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9c724643585568052d802c9c21df510e4fcedb438052f0bd7890a06cb5a4b97"} err="failed to get container status \"b9c724643585568052d802c9c21df510e4fcedb438052f0bd7890a06cb5a4b97\": rpc error: code = NotFound desc = could not find container \"b9c724643585568052d802c9c21df510e4fcedb438052f0bd7890a06cb5a4b97\": container with ID starting with b9c724643585568052d802c9c21df510e4fcedb438052f0bd7890a06cb5a4b97 not found: ID does not exist"
May 25 11:31:16.184397 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:31:16.184363    2561 scope.go:117] "RemoveContainer" containerID="d648e847aa12a82774aeb5e9b52b4f70dcf6dffdeee5c448b540b786316de2fa"
May 25 11:31:16.184561 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:31:16.184546    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d648e847aa12a82774aeb5e9b52b4f70dcf6dffdeee5c448b540b786316de2fa\": container with ID starting with d648e847aa12a82774aeb5e9b52b4f70dcf6dffdeee5c448b540b786316de2fa not found: ID does not exist" containerID="d648e847aa12a82774aeb5e9b52b4f70dcf6dffdeee5c448b540b786316de2fa"
May 25 11:31:16.184600 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:31:16.184563    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d648e847aa12a82774aeb5e9b52b4f70dcf6dffdeee5c448b540b786316de2fa"} err="failed to get container status \"d648e847aa12a82774aeb5e9b52b4f70dcf6dffdeee5c448b540b786316de2fa\": rpc error: code = NotFound desc = could not find container \"d648e847aa12a82774aeb5e9b52b4f70dcf6dffdeee5c448b540b786316de2fa\": container with ID starting with d648e847aa12a82774aeb5e9b52b4f70dcf6dffdeee5c448b540b786316de2fa not found: ID does not exist"
May 25 11:31:17.119057 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:31:17.119028    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4f034c8d-77f5-4e58-9314-6771e1ec46f8" path="/var/lib/kubelet/pods/4f034c8d-77f5-4e58-9314-6771e1ec46f8/volumes"
May 25 11:31:25.093915 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:31:25.093873    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-w9l9h" podUID="cbf3dfb7-c560-4f58-a46f-65e5e3eb38aa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
May 25 11:31:35.093673 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:31:35.093635    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-w9l9h" podUID="cbf3dfb7-c560-4f58-a46f-65e5e3eb38aa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
May 25 11:31:45.094024 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:31:45.093987    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-w9l9h" podUID="cbf3dfb7-c560-4f58-a46f-65e5e3eb38aa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
May 25 11:31:55.093427 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:31:55.093388    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-w9l9h" podUID="cbf3dfb7-c560-4f58-a46f-65e5e3eb38aa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
May 25 11:32:05.093451 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:32:05.093370    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-w9l9h" podUID="cbf3dfb7-c560-4f58-a46f-65e5e3eb38aa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
May 25 11:32:07.037410 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:32:07.037384    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-9lj95_10a66344-3304-49a1-bbc1-b885800f5acf/console-operator/1.log"
May 25 11:32:07.039573 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:32:07.039551    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-9lj95_10a66344-3304-49a1-bbc1-b885800f5acf/console-operator/1.log"
May 25 11:32:15.094335 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:32:15.094306    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-w9l9h"
May 25 11:32:15.788739 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:32:15.788700    2561 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-w9l9h"]
May 25 11:32:15.789110 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:32:15.789060    2561 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-w9l9h" podUID="cbf3dfb7-c560-4f58-a46f-65e5e3eb38aa" containerName="kube-rbac-proxy" containerID="cri-o://b6fb06873cd8ee6f841568831a02cc8fe5850075e651196839e1f8804b8f05b6" gracePeriod=30
May 25 11:32:15.789211 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:32:15.789033    2561 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-w9l9h" podUID="cbf3dfb7-c560-4f58-a46f-65e5e3eb38aa" containerName="kserve-container" containerID="cri-o://b5ddb399ad304a3f3bd484c6b93e31c7d8571540c930c6ab8687942f994517da" gracePeriod=30
May 25 11:32:15.934884 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:32:15.934851    2561 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-r9xxl"]
May 25 11:32:15.935253 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:32:15.935239    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f034c8d-77f5-4e58-9314-6771e1ec46f8" containerName="kserve-container"
May 25 11:32:15.935338 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:32:15.935255    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f034c8d-77f5-4e58-9314-6771e1ec46f8" containerName="kserve-container"
May 25 11:32:15.935338 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:32:15.935262    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f034c8d-77f5-4e58-9314-6771e1ec46f8" containerName="kube-rbac-proxy"
May 25 11:32:15.935338 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:32:15.935282    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f034c8d-77f5-4e58-9314-6771e1ec46f8" containerName="kube-rbac-proxy"
May 25 11:32:15.935338 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:32:15.935291    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f034c8d-77f5-4e58-9314-6771e1ec46f8" containerName="agent"
May 25 11:32:15.935338 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:32:15.935296    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f034c8d-77f5-4e58-9314-6771e1ec46f8" containerName="agent"
May 25 11:32:15.935338 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:32:15.935315    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f034c8d-77f5-4e58-9314-6771e1ec46f8" containerName="storage-initializer"
May 25 11:32:15.935338 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:32:15.935320    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f034c8d-77f5-4e58-9314-6771e1ec46f8" containerName="storage-initializer"
May 25 11:32:15.935585 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:32:15.935374    2561 memory_manager.go:356] "RemoveStaleState removing state" podUID="4f034c8d-77f5-4e58-9314-6771e1ec46f8" containerName="kserve-container"
May 25 11:32:15.935585 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:32:15.935384    2561 memory_manager.go:356] "RemoveStaleState removing state" podUID="4f034c8d-77f5-4e58-9314-6771e1ec46f8" containerName="kube-rbac-proxy"
May 25 11:32:15.935585 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:32:15.935391    2561 memory_manager.go:356] "RemoveStaleState removing state" podUID="4f034c8d-77f5-4e58-9314-6771e1ec46f8" containerName="agent"
May 25 11:32:15.938575 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:32:15.938559    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-r9xxl"
May 25 11:32:15.941230 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:32:15.941205    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-runtime-kube-rbac-proxy-sar-config\""
May 25 11:32:15.941373 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:32:15.941216    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-runtime-predictor-serving-cert\""
May 25 11:32:15.948515 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:32:15.948489    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-r9xxl"]
May 25 11:32:16.010088 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:32:16.010050    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnpcf\" (UniqueName: \"kubernetes.io/projected/791fa6a6-1afd-4d66-bf37-846c657cc7ee-kube-api-access-pnpcf\") pod \"isvc-lightgbm-runtime-predictor-749c4f6d58-r9xxl\" (UID: \"791fa6a6-1afd-4d66-bf37-846c657cc7ee\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-r9xxl"
May 25 11:32:16.010088 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:32:16.010092    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/791fa6a6-1afd-4d66-bf37-846c657cc7ee-kserve-provision-location\") pod \"isvc-lightgbm-runtime-predictor-749c4f6d58-r9xxl\" (UID: \"791fa6a6-1afd-4d66-bf37-846c657cc7ee\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-r9xxl"
May 25 11:32:16.010316 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:32:16.010119    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-lightgbm-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/791fa6a6-1afd-4d66-bf37-846c657cc7ee-isvc-lightgbm-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-runtime-predictor-749c4f6d58-r9xxl\" (UID: \"791fa6a6-1afd-4d66-bf37-846c657cc7ee\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-r9xxl"
May 25 11:32:16.010316 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:32:16.010182    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/791fa6a6-1afd-4d66-bf37-846c657cc7ee-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-749c4f6d58-r9xxl\" (UID: \"791fa6a6-1afd-4d66-bf37-846c657cc7ee\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-r9xxl"
May 25 11:32:16.111066 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:32:16.111028    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/791fa6a6-1afd-4d66-bf37-846c657cc7ee-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-749c4f6d58-r9xxl\" (UID: \"791fa6a6-1afd-4d66-bf37-846c657cc7ee\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-r9xxl"
May 25 11:32:16.111482 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:32:16.111110    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-pnpcf\" (UniqueName: \"kubernetes.io/projected/791fa6a6-1afd-4d66-bf37-846c657cc7ee-kube-api-access-pnpcf\") pod \"isvc-lightgbm-runtime-predictor-749c4f6d58-r9xxl\" (UID: \"791fa6a6-1afd-4d66-bf37-846c657cc7ee\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-r9xxl"
May 25 11:32:16.111482 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:32:16.111129    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/791fa6a6-1afd-4d66-bf37-846c657cc7ee-kserve-provision-location\") pod \"isvc-lightgbm-runtime-predictor-749c4f6d58-r9xxl\" (UID: \"791fa6a6-1afd-4d66-bf37-846c657cc7ee\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-r9xxl"
May 25 11:32:16.111482 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:32:16.111152    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-lightgbm-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/791fa6a6-1afd-4d66-bf37-846c657cc7ee-isvc-lightgbm-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-runtime-predictor-749c4f6d58-r9xxl\" (UID: \"791fa6a6-1afd-4d66-bf37-846c657cc7ee\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-r9xxl"
May 25 11:32:16.111606 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:32:16.111587    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/791fa6a6-1afd-4d66-bf37-846c657cc7ee-kserve-provision-location\") pod \"isvc-lightgbm-runtime-predictor-749c4f6d58-r9xxl\" (UID: \"791fa6a6-1afd-4d66-bf37-846c657cc7ee\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-r9xxl"
May 25 11:32:16.111825 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:32:16.111809    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-lightgbm-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/791fa6a6-1afd-4d66-bf37-846c657cc7ee-isvc-lightgbm-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-runtime-predictor-749c4f6d58-r9xxl\" (UID: \"791fa6a6-1afd-4d66-bf37-846c657cc7ee\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-r9xxl"
May 25 11:32:16.113604 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:32:16.113582    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/791fa6a6-1afd-4d66-bf37-846c657cc7ee-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-749c4f6d58-r9xxl\" (UID: \"791fa6a6-1afd-4d66-bf37-846c657cc7ee\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-r9xxl"
May 25 11:32:16.119857 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:32:16.119836    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnpcf\" (UniqueName: \"kubernetes.io/projected/791fa6a6-1afd-4d66-bf37-846c657cc7ee-kube-api-access-pnpcf\") pod \"isvc-lightgbm-runtime-predictor-749c4f6d58-r9xxl\" (UID: \"791fa6a6-1afd-4d66-bf37-846c657cc7ee\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-r9xxl"
May 25 11:32:16.250505 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:32:16.250473    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-r9xxl"
May 25 11:32:16.358816 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:32:16.358789    2561 generic.go:358] "Generic (PLEG): container finished" podID="cbf3dfb7-c560-4f58-a46f-65e5e3eb38aa" containerID="b6fb06873cd8ee6f841568831a02cc8fe5850075e651196839e1f8804b8f05b6" exitCode=2
May 25 11:32:16.358978 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:32:16.358874    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-w9l9h" event={"ID":"cbf3dfb7-c560-4f58-a46f-65e5e3eb38aa","Type":"ContainerDied","Data":"b6fb06873cd8ee6f841568831a02cc8fe5850075e651196839e1f8804b8f05b6"}
May 25 11:32:16.380923 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:32:16.380900    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-r9xxl"]
May 25 11:32:16.382684 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:32:16.382656    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod791fa6a6_1afd_4d66_bf37_846c657cc7ee.slice/crio-9c318b7eee89c2a0403b179a8f4f81d198b8a7eb9a57f5e598ab1c0b9c2bbf6a WatchSource:0}: Error finding container 9c318b7eee89c2a0403b179a8f4f81d198b8a7eb9a57f5e598ab1c0b9c2bbf6a: Status 404 returned error can't find the container with id 9c318b7eee89c2a0403b179a8f4f81d198b8a7eb9a57f5e598ab1c0b9c2bbf6a
May 25 11:32:17.363667 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:32:17.363626    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-r9xxl" event={"ID":"791fa6a6-1afd-4d66-bf37-846c657cc7ee","Type":"ContainerStarted","Data":"2e6264478ac3fb0fba7f236a6e6db24410a30f2c9412cf450712b3db57d13166"}
May 25 11:32:17.363667 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:32:17.363661    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-r9xxl" event={"ID":"791fa6a6-1afd-4d66-bf37-846c657cc7ee","Type":"ContainerStarted","Data":"9c318b7eee89c2a0403b179a8f4f81d198b8a7eb9a57f5e598ab1c0b9c2bbf6a"}
May 25 11:32:19.698006 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:32:19.697976    2561 cadvisor_stats_provider.go:525] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbf3dfb7_c560_4f58_a46f_65e5e3eb38aa.slice/crio-b5ddb399ad304a3f3bd484c6b93e31c7d8571540c930c6ab8687942f994517da.scope\": RecentStats: unable to find data in memory cache]"
May 25 11:32:19.839917 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:32:19.839896    2561 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-w9l9h"
May 25 11:32:19.942191 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:32:19.942123    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4vld7\" (UniqueName: \"kubernetes.io/projected/cbf3dfb7-c560-4f58-a46f-65e5e3eb38aa-kube-api-access-4vld7\") pod \"cbf3dfb7-c560-4f58-a46f-65e5e3eb38aa\" (UID: \"cbf3dfb7-c560-4f58-a46f-65e5e3eb38aa\") "
May 25 11:32:19.942364 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:32:19.942218    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-lightgbm-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cbf3dfb7-c560-4f58-a46f-65e5e3eb38aa-isvc-lightgbm-kube-rbac-proxy-sar-config\") pod \"cbf3dfb7-c560-4f58-a46f-65e5e3eb38aa\" (UID: \"cbf3dfb7-c560-4f58-a46f-65e5e3eb38aa\") "
May 25 11:32:19.942364 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:32:19.942263    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cbf3dfb7-c560-4f58-a46f-65e5e3eb38aa-kserve-provision-location\") pod \"cbf3dfb7-c560-4f58-a46f-65e5e3eb38aa\" (UID: \"cbf3dfb7-c560-4f58-a46f-65e5e3eb38aa\") "
May 25 11:32:19.942364 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:32:19.942328    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cbf3dfb7-c560-4f58-a46f-65e5e3eb38aa-proxy-tls\") pod \"cbf3dfb7-c560-4f58-a46f-65e5e3eb38aa\" (UID: \"cbf3dfb7-c560-4f58-a46f-65e5e3eb38aa\") "
May 25 11:32:19.942574 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:32:19.942549    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbf3dfb7-c560-4f58-a46f-65e5e3eb38aa-isvc-lightgbm-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-lightgbm-kube-rbac-proxy-sar-config") pod "cbf3dfb7-c560-4f58-a46f-65e5e3eb38aa" (UID: "cbf3dfb7-c560-4f58-a46f-65e5e3eb38aa"). InnerVolumeSpecName "isvc-lightgbm-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 11:32:19.942654 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:32:19.942625    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbf3dfb7-c560-4f58-a46f-65e5e3eb38aa-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "cbf3dfb7-c560-4f58-a46f-65e5e3eb38aa" (UID: "cbf3dfb7-c560-4f58-a46f-65e5e3eb38aa"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:32:19.944111 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:32:19.944088    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbf3dfb7-c560-4f58-a46f-65e5e3eb38aa-kube-api-access-4vld7" (OuterVolumeSpecName: "kube-api-access-4vld7") pod "cbf3dfb7-c560-4f58-a46f-65e5e3eb38aa" (UID: "cbf3dfb7-c560-4f58-a46f-65e5e3eb38aa"). InnerVolumeSpecName "kube-api-access-4vld7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 11:32:19.944205 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:32:19.944188    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cbf3dfb7-c560-4f58-a46f-65e5e3eb38aa-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "cbf3dfb7-c560-4f58-a46f-65e5e3eb38aa" (UID: "cbf3dfb7-c560-4f58-a46f-65e5e3eb38aa"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 11:32:20.043879 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:32:20.043858    2561 reconciler_common.go:299] "Volume detached for volume \"isvc-lightgbm-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cbf3dfb7-c560-4f58-a46f-65e5e3eb38aa-isvc-lightgbm-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-227.ec2.internal\" DevicePath \"\""
May 25 11:32:20.043879 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:32:20.043879    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cbf3dfb7-c560-4f58-a46f-65e5e3eb38aa-kserve-provision-location\") on node \"ip-10-0-136-227.ec2.internal\" DevicePath \"\""
May 25 11:32:20.044004 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:32:20.043890    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cbf3dfb7-c560-4f58-a46f-65e5e3eb38aa-proxy-tls\") on node \"ip-10-0-136-227.ec2.internal\" DevicePath \"\""
May 25 11:32:20.044004 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:32:20.043900    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4vld7\" (UniqueName: \"kubernetes.io/projected/cbf3dfb7-c560-4f58-a46f-65e5e3eb38aa-kube-api-access-4vld7\") on node \"ip-10-0-136-227.ec2.internal\" DevicePath \"\""
May 25 11:32:20.375643 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:32:20.375608    2561 generic.go:358] "Generic (PLEG): container finished" podID="cbf3dfb7-c560-4f58-a46f-65e5e3eb38aa" containerID="b5ddb399ad304a3f3bd484c6b93e31c7d8571540c930c6ab8687942f994517da" exitCode=0
May 25 11:32:20.375772 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:32:20.375691    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-w9l9h" event={"ID":"cbf3dfb7-c560-4f58-a46f-65e5e3eb38aa","Type":"ContainerDied","Data":"b5ddb399ad304a3f3bd484c6b93e31c7d8571540c930c6ab8687942f994517da"}
May 25 11:32:20.375772 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:32:20.375726    2561 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-w9l9h"
May 25 11:32:20.375772 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:32:20.375743    2561 scope.go:117] "RemoveContainer" containerID="b6fb06873cd8ee6f841568831a02cc8fe5850075e651196839e1f8804b8f05b6"
May 25 11:32:20.375903 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:32:20.375733    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-w9l9h" event={"ID":"cbf3dfb7-c560-4f58-a46f-65e5e3eb38aa","Type":"ContainerDied","Data":"79d90de54b6727715b3b477642b8111c89000bbaca4247444e9c708fa6973e31"}
May 25 11:32:20.377098 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:32:20.377070    2561 generic.go:358] "Generic (PLEG): container finished" podID="791fa6a6-1afd-4d66-bf37-846c657cc7ee" containerID="2e6264478ac3fb0fba7f236a6e6db24410a30f2c9412cf450712b3db57d13166" exitCode=0
May 25 11:32:20.377215 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:32:20.377100    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-r9xxl" event={"ID":"791fa6a6-1afd-4d66-bf37-846c657cc7ee","Type":"ContainerDied","Data":"2e6264478ac3fb0fba7f236a6e6db24410a30f2c9412cf450712b3db57d13166"}
May 25 11:32:20.384129 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:32:20.384114    2561 scope.go:117] "RemoveContainer" containerID="b5ddb399ad304a3f3bd484c6b93e31c7d8571540c930c6ab8687942f994517da"
May 25 11:32:20.394537 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:32:20.394516    2561 scope.go:117] "RemoveContainer" containerID="457a81bd72cd61bee5383aac067bc1de93087715bceda1dc4515987f353f7e49"
May 25 11:32:20.401882 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:32:20.401866    2561 scope.go:117] "RemoveContainer" containerID="b6fb06873cd8ee6f841568831a02cc8fe5850075e651196839e1f8804b8f05b6"
May 25 11:32:20.402135 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:32:20.402111    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6fb06873cd8ee6f841568831a02cc8fe5850075e651196839e1f8804b8f05b6\": container with ID starting with b6fb06873cd8ee6f841568831a02cc8fe5850075e651196839e1f8804b8f05b6 not found: ID does not exist" containerID="b6fb06873cd8ee6f841568831a02cc8fe5850075e651196839e1f8804b8f05b6"
May 25 11:32:20.402214 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:32:20.402150    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6fb06873cd8ee6f841568831a02cc8fe5850075e651196839e1f8804b8f05b6"} err="failed to get container status \"b6fb06873cd8ee6f841568831a02cc8fe5850075e651196839e1f8804b8f05b6\": rpc error: code = NotFound desc = could not find container \"b6fb06873cd8ee6f841568831a02cc8fe5850075e651196839e1f8804b8f05b6\": container with ID starting with b6fb06873cd8ee6f841568831a02cc8fe5850075e651196839e1f8804b8f05b6 not found: ID does not exist"
May 25 11:32:20.402214 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:32:20.402173    2561 scope.go:117] "RemoveContainer" containerID="b5ddb399ad304a3f3bd484c6b93e31c7d8571540c930c6ab8687942f994517da"
May 25 11:32:20.402468 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:32:20.402450    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5ddb399ad304a3f3bd484c6b93e31c7d8571540c930c6ab8687942f994517da\": container with ID starting with b5ddb399ad304a3f3bd484c6b93e31c7d8571540c930c6ab8687942f994517da not found: ID does not exist" containerID="b5ddb399ad304a3f3bd484c6b93e31c7d8571540c930c6ab8687942f994517da"
May 25 11:32:20.402515 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:32:20.402474    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5ddb399ad304a3f3bd484c6b93e31c7d8571540c930c6ab8687942f994517da"} err="failed to get container status \"b5ddb399ad304a3f3bd484c6b93e31c7d8571540c930c6ab8687942f994517da\": rpc error: code = NotFound desc = could not find container \"b5ddb399ad304a3f3bd484c6b93e31c7d8571540c930c6ab8687942f994517da\": container with ID starting with b5ddb399ad304a3f3bd484c6b93e31c7d8571540c930c6ab8687942f994517da not found: ID does not exist"
May 25 11:32:20.402515 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:32:20.402490    2561 scope.go:117] "RemoveContainer" containerID="457a81bd72cd61bee5383aac067bc1de93087715bceda1dc4515987f353f7e49"
May 25 11:32:20.402723 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:32:20.402707    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"457a81bd72cd61bee5383aac067bc1de93087715bceda1dc4515987f353f7e49\": container with ID starting with 457a81bd72cd61bee5383aac067bc1de93087715bceda1dc4515987f353f7e49 not found: ID does not exist" containerID="457a81bd72cd61bee5383aac067bc1de93087715bceda1dc4515987f353f7e49"
May 25 11:32:20.402805 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:32:20.402725    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"457a81bd72cd61bee5383aac067bc1de93087715bceda1dc4515987f353f7e49"} err="failed to get container status \"457a81bd72cd61bee5383aac067bc1de93087715bceda1dc4515987f353f7e49\": rpc error: code = NotFound desc = could not find container \"457a81bd72cd61bee5383aac067bc1de93087715bceda1dc4515987f353f7e49\": container with ID starting with 457a81bd72cd61bee5383aac067bc1de93087715bceda1dc4515987f353f7e49 not found: ID does not exist"
May 25 11:32:20.426349 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:32:20.426330    2561 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-w9l9h"]
May 25 11:32:20.435119 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:32:20.435099    2561 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-bdf964bd-w9l9h"]
May 25 11:32:21.117465 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:32:21.117437    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cbf3dfb7-c560-4f58-a46f-65e5e3eb38aa" path="/var/lib/kubelet/pods/cbf3dfb7-c560-4f58-a46f-65e5e3eb38aa/volumes"
May 25 11:32:21.382478 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:32:21.382401    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-r9xxl" event={"ID":"791fa6a6-1afd-4d66-bf37-846c657cc7ee","Type":"ContainerStarted","Data":"f17e671c0fc61345177130eaaa0274dbf4f0dab5c5853c253575cd93a56e6481"}
May 25 11:32:21.382478 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:32:21.382436    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-r9xxl" event={"ID":"791fa6a6-1afd-4d66-bf37-846c657cc7ee","Type":"ContainerStarted","Data":"11b9f7ea52dd997bdd4f1f095acb6895cfaf5ca0adb2af17781f0470d5615429"}
May 25 11:32:21.382653 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:32:21.382637    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-r9xxl"
May 25 11:32:21.406220 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:32:21.406170    2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-r9xxl" podStartSLOduration=6.406159328 podStartE2EDuration="6.406159328s" podCreationTimestamp="2026-05-25 11:32:15 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 11:32:21.405637682 +0000 UTC m=+914.950815286" watchObservedRunningTime="2026-05-25 11:32:21.406159328 +0000 UTC m=+914.951336931"
May 25 11:32:22.386184 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:32:22.386153    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-r9xxl"
May 25 11:32:22.387243 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:32:22.387211    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-r9xxl" podUID="791fa6a6-1afd-4d66-bf37-846c657cc7ee" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
May 25 11:32:23.389205 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:32:23.389174    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-r9xxl" podUID="791fa6a6-1afd-4d66-bf37-846c657cc7ee" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
May 25 11:32:28.393680 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:32:28.393652    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-r9xxl"
May 25 11:32:28.394234 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:32:28.394208    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-r9xxl" podUID="791fa6a6-1afd-4d66-bf37-846c657cc7ee" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
May 25 11:32:38.394745 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:32:38.394701    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-r9xxl" podUID="791fa6a6-1afd-4d66-bf37-846c657cc7ee" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
May 25 11:32:48.394792 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:32:48.394752    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-r9xxl" podUID="791fa6a6-1afd-4d66-bf37-846c657cc7ee" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
May 25 11:32:58.395163 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:32:58.395124    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-r9xxl" podUID="791fa6a6-1afd-4d66-bf37-846c657cc7ee" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
May 25 11:33:08.394154 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:33:08.394119    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-r9xxl" podUID="791fa6a6-1afd-4d66-bf37-846c657cc7ee" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
May 25 11:33:18.395123 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:33:18.395083    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-r9xxl" podUID="791fa6a6-1afd-4d66-bf37-846c657cc7ee" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
May 25 11:33:28.394707 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:33:28.394623    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-r9xxl" podUID="791fa6a6-1afd-4d66-bf37-846c657cc7ee" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
May 25 11:33:32.114152 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:33:32.114119    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-r9xxl"
May 25 11:33:36.200112 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:33:36.200079    2561 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-r9xxl"]
May 25 11:33:36.200607 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:33:36.200400    2561 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-r9xxl" podUID="791fa6a6-1afd-4d66-bf37-846c657cc7ee" containerName="kserve-container" containerID="cri-o://11b9f7ea52dd997bdd4f1f095acb6895cfaf5ca0adb2af17781f0470d5615429" gracePeriod=30
May 25 11:33:36.200607 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:33:36.200427    2561 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-r9xxl" podUID="791fa6a6-1afd-4d66-bf37-846c657cc7ee" containerName="kube-rbac-proxy" containerID="cri-o://f17e671c0fc61345177130eaaa0274dbf4f0dab5c5853c253575cd93a56e6481" gracePeriod=30
May 25 11:33:36.301986 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:33:36.301956    2561 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-bhq2s"]
May 25 11:33:36.302392 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:33:36.302375    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cbf3dfb7-c560-4f58-a46f-65e5e3eb38aa" containerName="storage-initializer"
May 25 11:33:36.302457 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:33:36.302394    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="cbf3dfb7-c560-4f58-a46f-65e5e3eb38aa" containerName="storage-initializer"
May 25 11:33:36.302457 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:33:36.302418    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cbf3dfb7-c560-4f58-a46f-65e5e3eb38aa" containerName="kserve-container"
May 25 11:33:36.302457 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:33:36.302423    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="cbf3dfb7-c560-4f58-a46f-65e5e3eb38aa" containerName="kserve-container"
May 25 11:33:36.302457 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:33:36.302432    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cbf3dfb7-c560-4f58-a46f-65e5e3eb38aa" containerName="kube-rbac-proxy"
May 25 11:33:36.302457 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:33:36.302437    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="cbf3dfb7-c560-4f58-a46f-65e5e3eb38aa" containerName="kube-rbac-proxy"
May 25 11:33:36.302664 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:33:36.302491    2561 memory_manager.go:356] "RemoveStaleState removing state" podUID="cbf3dfb7-c560-4f58-a46f-65e5e3eb38aa" containerName="kserve-container"
May 25 11:33:36.302664 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:33:36.302502    2561 memory_manager.go:356] "RemoveStaleState removing state" podUID="cbf3dfb7-c560-4f58-a46f-65e5e3eb38aa" containerName="kube-rbac-proxy"
May 25 11:33:36.305960 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:33:36.305940    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-bhq2s"
May 25 11:33:36.308373 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:33:36.308356    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-runtime-predictor-serving-cert\""
May 25 11:33:36.308373 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:33:36.308363    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config\""
May 25 11:33:36.317051 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:33:36.317029    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-bhq2s"]
May 25 11:33:36.323280 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:33:36.323245    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e9065062-773e-40bd-959b-c0c91b7de1f1-kserve-provision-location\") pod \"isvc-lightgbm-v2-runtime-predictor-8765c9667-bhq2s\" (UID: \"e9065062-773e-40bd-959b-c0c91b7de1f1\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-bhq2s"
May 25 11:33:36.323372 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:33:36.323352    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e9065062-773e-40bd-959b-c0c91b7de1f1-isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-runtime-predictor-8765c9667-bhq2s\" (UID: \"e9065062-773e-40bd-959b-c0c91b7de1f1\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-bhq2s"
May 25 11:33:36.323430 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:33:36.323414    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xjld\" (UniqueName: \"kubernetes.io/projected/e9065062-773e-40bd-959b-c0c91b7de1f1-kube-api-access-7xjld\") pod \"isvc-lightgbm-v2-runtime-predictor-8765c9667-bhq2s\" (UID: \"e9065062-773e-40bd-959b-c0c91b7de1f1\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-bhq2s"
May 25 11:33:36.323503 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:33:36.323485    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e9065062-773e-40bd-959b-c0c91b7de1f1-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-8765c9667-bhq2s\" (UID: \"e9065062-773e-40bd-959b-c0c91b7de1f1\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-bhq2s"
May 25 11:33:36.424218 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:33:36.424191    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e9065062-773e-40bd-959b-c0c91b7de1f1-kserve-provision-location\") pod \"isvc-lightgbm-v2-runtime-predictor-8765c9667-bhq2s\" (UID: \"e9065062-773e-40bd-959b-c0c91b7de1f1\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-bhq2s"
May 25 11:33:36.424339 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:33:36.424225    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e9065062-773e-40bd-959b-c0c91b7de1f1-isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-runtime-predictor-8765c9667-bhq2s\" (UID: \"e9065062-773e-40bd-959b-c0c91b7de1f1\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-bhq2s"
May 25 11:33:36.424339 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:33:36.424262    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-7xjld\" (UniqueName: \"kubernetes.io/projected/e9065062-773e-40bd-959b-c0c91b7de1f1-kube-api-access-7xjld\") pod \"isvc-lightgbm-v2-runtime-predictor-8765c9667-bhq2s\" (UID: \"e9065062-773e-40bd-959b-c0c91b7de1f1\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-bhq2s"
May 25 11:33:36.424339 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:33:36.424320    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e9065062-773e-40bd-959b-c0c91b7de1f1-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-8765c9667-bhq2s\" (UID: \"e9065062-773e-40bd-959b-c0c91b7de1f1\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-bhq2s"
May 25 11:33:36.424483 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:33:36.424468    2561 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-serving-cert: secret "isvc-lightgbm-v2-runtime-predictor-serving-cert" not found
May 25 11:33:36.424553 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:33:36.424544    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e9065062-773e-40bd-959b-c0c91b7de1f1-proxy-tls podName:e9065062-773e-40bd-959b-c0c91b7de1f1 nodeName:}" failed. No retries permitted until 2026-05-25 11:33:36.924530144 +0000 UTC m=+990.469707726 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/e9065062-773e-40bd-959b-c0c91b7de1f1-proxy-tls") pod "isvc-lightgbm-v2-runtime-predictor-8765c9667-bhq2s" (UID: "e9065062-773e-40bd-959b-c0c91b7de1f1") : secret "isvc-lightgbm-v2-runtime-predictor-serving-cert" not found
May 25 11:33:36.424606 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:33:36.424572    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e9065062-773e-40bd-959b-c0c91b7de1f1-kserve-provision-location\") pod \"isvc-lightgbm-v2-runtime-predictor-8765c9667-bhq2s\" (UID: \"e9065062-773e-40bd-959b-c0c91b7de1f1\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-bhq2s"
May 25 11:33:36.424875 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:33:36.424856    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e9065062-773e-40bd-959b-c0c91b7de1f1-isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-runtime-predictor-8765c9667-bhq2s\" (UID: \"e9065062-773e-40bd-959b-c0c91b7de1f1\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-bhq2s"
May 25 11:33:36.433590 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:33:36.433559    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xjld\" (UniqueName: \"kubernetes.io/projected/e9065062-773e-40bd-959b-c0c91b7de1f1-kube-api-access-7xjld\") pod \"isvc-lightgbm-v2-runtime-predictor-8765c9667-bhq2s\" (UID: \"e9065062-773e-40bd-959b-c0c91b7de1f1\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-bhq2s"
May 25 11:33:36.632612 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:33:36.632586    2561 generic.go:358] "Generic (PLEG): container finished" podID="791fa6a6-1afd-4d66-bf37-846c657cc7ee" containerID="f17e671c0fc61345177130eaaa0274dbf4f0dab5c5853c253575cd93a56e6481" exitCode=2
May 25 11:33:36.632715 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:33:36.632656    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-r9xxl" event={"ID":"791fa6a6-1afd-4d66-bf37-846c657cc7ee","Type":"ContainerDied","Data":"f17e671c0fc61345177130eaaa0274dbf4f0dab5c5853c253575cd93a56e6481"}
May 25 11:33:36.928313 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:33:36.928222    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e9065062-773e-40bd-959b-c0c91b7de1f1-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-8765c9667-bhq2s\" (UID: \"e9065062-773e-40bd-959b-c0c91b7de1f1\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-bhq2s"
May 25 11:33:36.930570 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:33:36.930545    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e9065062-773e-40bd-959b-c0c91b7de1f1-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-8765c9667-bhq2s\" (UID: \"e9065062-773e-40bd-959b-c0c91b7de1f1\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-bhq2s"
May 25 11:33:37.218165 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:33:37.218091    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-bhq2s"
May 25 11:33:37.356935 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:33:37.356913    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-bhq2s"]
May 25 11:33:37.358964 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:33:37.358938    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9065062_773e_40bd_959b_c0c91b7de1f1.slice/crio-be1a4c36950832ab2636a3b5b47e9ddd57fad95a9010dda6cff3dff96b9ff02a WatchSource:0}: Error finding container be1a4c36950832ab2636a3b5b47e9ddd57fad95a9010dda6cff3dff96b9ff02a: Status 404 returned error can't find the container with id be1a4c36950832ab2636a3b5b47e9ddd57fad95a9010dda6cff3dff96b9ff02a
May 25 11:33:37.361210 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:33:37.361187    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 25 11:33:37.636984 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:33:37.636948    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-bhq2s" event={"ID":"e9065062-773e-40bd-959b-c0c91b7de1f1","Type":"ContainerStarted","Data":"be69fe8941d8c4bfe3430a9bbc05713d82d394c1d6b9d6644efd43ab9160ef7f"}
May 25 11:33:37.636984 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:33:37.636985    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-bhq2s" event={"ID":"e9065062-773e-40bd-959b-c0c91b7de1f1","Type":"ContainerStarted","Data":"be1a4c36950832ab2636a3b5b47e9ddd57fad95a9010dda6cff3dff96b9ff02a"}
May 25 11:33:38.390478 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:33:38.390438    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-r9xxl" podUID="791fa6a6-1afd-4d66-bf37-846c657cc7ee" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.40:8643/healthz\": dial tcp 10.132.0.40:8643: connect: connection refused"
May 25 11:33:40.348098 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:33:40.348077    2561 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-r9xxl"
May 25 11:33:40.354540 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:33:40.354523    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/791fa6a6-1afd-4d66-bf37-846c657cc7ee-proxy-tls\") pod \"791fa6a6-1afd-4d66-bf37-846c657cc7ee\" (UID: \"791fa6a6-1afd-4d66-bf37-846c657cc7ee\") "
May 25 11:33:40.354624 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:33:40.354611    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-lightgbm-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/791fa6a6-1afd-4d66-bf37-846c657cc7ee-isvc-lightgbm-runtime-kube-rbac-proxy-sar-config\") pod \"791fa6a6-1afd-4d66-bf37-846c657cc7ee\" (UID: \"791fa6a6-1afd-4d66-bf37-846c657cc7ee\") "
May 25 11:33:40.354678 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:33:40.354641    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/791fa6a6-1afd-4d66-bf37-846c657cc7ee-kserve-provision-location\") pod \"791fa6a6-1afd-4d66-bf37-846c657cc7ee\" (UID: \"791fa6a6-1afd-4d66-bf37-846c657cc7ee\") "
May 25 11:33:40.354713 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:33:40.354683    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pnpcf\" (UniqueName: \"kubernetes.io/projected/791fa6a6-1afd-4d66-bf37-846c657cc7ee-kube-api-access-pnpcf\") pod \"791fa6a6-1afd-4d66-bf37-846c657cc7ee\" (UID: \"791fa6a6-1afd-4d66-bf37-846c657cc7ee\") "
May 25 11:33:40.354926 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:33:40.354901    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/791fa6a6-1afd-4d66-bf37-846c657cc7ee-isvc-lightgbm-runtime-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-lightgbm-runtime-kube-rbac-proxy-sar-config") pod "791fa6a6-1afd-4d66-bf37-846c657cc7ee" (UID: "791fa6a6-1afd-4d66-bf37-846c657cc7ee"). InnerVolumeSpecName "isvc-lightgbm-runtime-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 11:33:40.355003 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:33:40.354942    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/791fa6a6-1afd-4d66-bf37-846c657cc7ee-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "791fa6a6-1afd-4d66-bf37-846c657cc7ee" (UID: "791fa6a6-1afd-4d66-bf37-846c657cc7ee"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:33:40.356415 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:33:40.356392    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/791fa6a6-1afd-4d66-bf37-846c657cc7ee-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "791fa6a6-1afd-4d66-bf37-846c657cc7ee" (UID: "791fa6a6-1afd-4d66-bf37-846c657cc7ee"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 11:33:40.356695 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:33:40.356673    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/791fa6a6-1afd-4d66-bf37-846c657cc7ee-kube-api-access-pnpcf" (OuterVolumeSpecName: "kube-api-access-pnpcf") pod "791fa6a6-1afd-4d66-bf37-846c657cc7ee" (UID: "791fa6a6-1afd-4d66-bf37-846c657cc7ee"). InnerVolumeSpecName "kube-api-access-pnpcf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 11:33:40.456157 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:33:40.456089    2561 reconciler_common.go:299] "Volume detached for volume \"isvc-lightgbm-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/791fa6a6-1afd-4d66-bf37-846c657cc7ee-isvc-lightgbm-runtime-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-227.ec2.internal\" DevicePath \"\""
May 25 11:33:40.456157 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:33:40.456114    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/791fa6a6-1afd-4d66-bf37-846c657cc7ee-kserve-provision-location\") on node \"ip-10-0-136-227.ec2.internal\" DevicePath \"\""
May 25 11:33:40.456157 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:33:40.456126    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pnpcf\" (UniqueName: \"kubernetes.io/projected/791fa6a6-1afd-4d66-bf37-846c657cc7ee-kube-api-access-pnpcf\") on node \"ip-10-0-136-227.ec2.internal\" DevicePath \"\""
May 25 11:33:40.456157 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:33:40.456135    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/791fa6a6-1afd-4d66-bf37-846c657cc7ee-proxy-tls\") on node \"ip-10-0-136-227.ec2.internal\" DevicePath \"\""
May 25 11:33:40.649161 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:33:40.649125    2561 generic.go:358] "Generic (PLEG): container finished" podID="791fa6a6-1afd-4d66-bf37-846c657cc7ee" containerID="11b9f7ea52dd997bdd4f1f095acb6895cfaf5ca0adb2af17781f0470d5615429" exitCode=0
May 25 11:33:40.649329 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:33:40.649199    2561 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-r9xxl"
May 25 11:33:40.649329 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:33:40.649214    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-r9xxl" event={"ID":"791fa6a6-1afd-4d66-bf37-846c657cc7ee","Type":"ContainerDied","Data":"11b9f7ea52dd997bdd4f1f095acb6895cfaf5ca0adb2af17781f0470d5615429"}
May 25 11:33:40.649329 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:33:40.649250    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-r9xxl" event={"ID":"791fa6a6-1afd-4d66-bf37-846c657cc7ee","Type":"ContainerDied","Data":"9c318b7eee89c2a0403b179a8f4f81d198b8a7eb9a57f5e598ab1c0b9c2bbf6a"}
May 25 11:33:40.649329 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:33:40.649290    2561 scope.go:117] "RemoveContainer" containerID="f17e671c0fc61345177130eaaa0274dbf4f0dab5c5853c253575cd93a56e6481"
May 25 11:33:40.657602 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:33:40.657587    2561 scope.go:117] "RemoveContainer" containerID="11b9f7ea52dd997bdd4f1f095acb6895cfaf5ca0adb2af17781f0470d5615429"
May 25 11:33:40.664819 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:33:40.664801    2561 scope.go:117] "RemoveContainer" containerID="2e6264478ac3fb0fba7f236a6e6db24410a30f2c9412cf450712b3db57d13166"
May 25 11:33:40.671995 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:33:40.671971    2561 scope.go:117] "RemoveContainer" containerID="f17e671c0fc61345177130eaaa0274dbf4f0dab5c5853c253575cd93a56e6481"
May 25 11:33:40.672454 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:33:40.672422    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f17e671c0fc61345177130eaaa0274dbf4f0dab5c5853c253575cd93a56e6481\": container with ID starting with f17e671c0fc61345177130eaaa0274dbf4f0dab5c5853c253575cd93a56e6481 not found: ID does not exist" containerID="f17e671c0fc61345177130eaaa0274dbf4f0dab5c5853c253575cd93a56e6481"
May 25 11:33:40.672555 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:33:40.672463    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f17e671c0fc61345177130eaaa0274dbf4f0dab5c5853c253575cd93a56e6481"} err="failed to get container status \"f17e671c0fc61345177130eaaa0274dbf4f0dab5c5853c253575cd93a56e6481\": rpc error: code = NotFound desc = could not find container \"f17e671c0fc61345177130eaaa0274dbf4f0dab5c5853c253575cd93a56e6481\": container with ID starting with f17e671c0fc61345177130eaaa0274dbf4f0dab5c5853c253575cd93a56e6481 not found: ID does not exist"
May 25 11:33:40.672555 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:33:40.672495    2561 scope.go:117] "RemoveContainer" containerID="11b9f7ea52dd997bdd4f1f095acb6895cfaf5ca0adb2af17781f0470d5615429"
May 25 11:33:40.672773 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:33:40.672738    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11b9f7ea52dd997bdd4f1f095acb6895cfaf5ca0adb2af17781f0470d5615429\": container with ID starting with 11b9f7ea52dd997bdd4f1f095acb6895cfaf5ca0adb2af17781f0470d5615429 not found: ID does not exist" containerID="11b9f7ea52dd997bdd4f1f095acb6895cfaf5ca0adb2af17781f0470d5615429"
May 25 11:33:40.672833 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:33:40.672768    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11b9f7ea52dd997bdd4f1f095acb6895cfaf5ca0adb2af17781f0470d5615429"} err="failed to get container status \"11b9f7ea52dd997bdd4f1f095acb6895cfaf5ca0adb2af17781f0470d5615429\": rpc error: code = NotFound desc = could not find container \"11b9f7ea52dd997bdd4f1f095acb6895cfaf5ca0adb2af17781f0470d5615429\": container with ID starting with 11b9f7ea52dd997bdd4f1f095acb6895cfaf5ca0adb2af17781f0470d5615429 not found: ID does not exist"
May 25 11:33:40.672833 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:33:40.672786    2561 scope.go:117] "RemoveContainer" containerID="2e6264478ac3fb0fba7f236a6e6db24410a30f2c9412cf450712b3db57d13166"
May 25 11:33:40.673028 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:33:40.673010    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e6264478ac3fb0fba7f236a6e6db24410a30f2c9412cf450712b3db57d13166\": container with ID starting with 2e6264478ac3fb0fba7f236a6e6db24410a30f2c9412cf450712b3db57d13166 not found: ID does not exist" containerID="2e6264478ac3fb0fba7f236a6e6db24410a30f2c9412cf450712b3db57d13166"
May 25 11:33:40.673094 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:33:40.673031    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e6264478ac3fb0fba7f236a6e6db24410a30f2c9412cf450712b3db57d13166"} err="failed to get container status \"2e6264478ac3fb0fba7f236a6e6db24410a30f2c9412cf450712b3db57d13166\": rpc error: code = NotFound desc = could not find container \"2e6264478ac3fb0fba7f236a6e6db24410a30f2c9412cf450712b3db57d13166\": container with ID starting with 2e6264478ac3fb0fba7f236a6e6db24410a30f2c9412cf450712b3db57d13166 not found: ID does not exist"
May 25 11:33:40.674929 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:33:40.674901    2561 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-r9xxl"]
May 25 11:33:40.680342 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:33:40.680323    2561 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-749c4f6d58-r9xxl"]
May 25 11:33:41.116908 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:33:41.116874    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="791fa6a6-1afd-4d66-bf37-846c657cc7ee" path="/var/lib/kubelet/pods/791fa6a6-1afd-4d66-bf37-846c657cc7ee/volumes"
May 25 11:33:41.654413 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:33:41.654379    2561 generic.go:358] "Generic (PLEG): container finished" podID="e9065062-773e-40bd-959b-c0c91b7de1f1" containerID="be69fe8941d8c4bfe3430a9bbc05713d82d394c1d6b9d6644efd43ab9160ef7f" exitCode=0
May 25 11:33:41.654814 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:33:41.654450    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-bhq2s" event={"ID":"e9065062-773e-40bd-959b-c0c91b7de1f1","Type":"ContainerDied","Data":"be69fe8941d8c4bfe3430a9bbc05713d82d394c1d6b9d6644efd43ab9160ef7f"}
May 25 11:35:39.103033 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:35:39.102989    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-bhq2s" event={"ID":"e9065062-773e-40bd-959b-c0c91b7de1f1","Type":"ContainerStarted","Data":"08e5d21d9899cb67ee0bdb29732328fd3cdeb1ea434f72f3d66cf6bf741e0525"}
May 25 11:35:39.103033 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:35:39.103039    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-bhq2s" event={"ID":"e9065062-773e-40bd-959b-c0c91b7de1f1","Type":"ContainerStarted","Data":"be6df7ceadf6432bbc50bfc92d93a0fd5a140bb1eb95ad0b7b438b93d0df240f"}
May 25 11:35:39.103509 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:35:39.103101    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-bhq2s"
May 25 11:35:39.103509 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:35:39.103210    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-bhq2s"
May 25 11:35:39.149543 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:35:39.149493    2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-bhq2s" podStartSLOduration=6.386495764 podStartE2EDuration="2m3.14948173s" podCreationTimestamp="2026-05-25 11:33:36 +0000 UTC" firstStartedPulling="2026-05-25 11:33:41.655454397 +0000 UTC m=+995.200631982" lastFinishedPulling="2026-05-25 11:35:38.418440362 +0000 UTC m=+1111.963617948" observedRunningTime="2026-05-25 11:35:39.14921307 +0000 UTC m=+1112.694390674" watchObservedRunningTime="2026-05-25 11:35:39.14948173 +0000 UTC m=+1112.694659334"
May 25 11:35:45.117296 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:35:45.117249    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-bhq2s"
May 25 11:36:15.121226 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:36:15.121197    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-bhq2s"
May 25 11:36:16.540503 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:36:16.540464    2561 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-bhq2s"]
May 25 11:36:16.540947 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:36:16.540848    2561 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-bhq2s" podUID="e9065062-773e-40bd-959b-c0c91b7de1f1" containerName="kserve-container" containerID="cri-o://be6df7ceadf6432bbc50bfc92d93a0fd5a140bb1eb95ad0b7b438b93d0df240f" gracePeriod=30
May 25 11:36:16.541018 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:36:16.540974    2561 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-bhq2s" podUID="e9065062-773e-40bd-959b-c0c91b7de1f1" containerName="kube-rbac-proxy" containerID="cri-o://08e5d21d9899cb67ee0bdb29732328fd3cdeb1ea434f72f3d66cf6bf741e0525" gracePeriod=30
May 25 11:36:17.233629 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:36:17.233595    2561 generic.go:358] "Generic (PLEG): container finished" podID="e9065062-773e-40bd-959b-c0c91b7de1f1" containerID="08e5d21d9899cb67ee0bdb29732328fd3cdeb1ea434f72f3d66cf6bf741e0525" exitCode=2
May 25 11:36:17.233792 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:36:17.233631    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-bhq2s" event={"ID":"e9065062-773e-40bd-959b-c0c91b7de1f1","Type":"ContainerDied","Data":"08e5d21d9899cb67ee0bdb29732328fd3cdeb1ea434f72f3d66cf6bf741e0525"}
May 25 11:36:17.495527 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:36:17.495474    2561 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-bhq2s"
May 25 11:36:17.568240 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:36:17.568212    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e9065062-773e-40bd-959b-c0c91b7de1f1-isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config\") pod \"e9065062-773e-40bd-959b-c0c91b7de1f1\" (UID: \"e9065062-773e-40bd-959b-c0c91b7de1f1\") "
May 25 11:36:17.568646 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:36:17.568297    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7xjld\" (UniqueName: \"kubernetes.io/projected/e9065062-773e-40bd-959b-c0c91b7de1f1-kube-api-access-7xjld\") pod \"e9065062-773e-40bd-959b-c0c91b7de1f1\" (UID: \"e9065062-773e-40bd-959b-c0c91b7de1f1\") "
May 25 11:36:17.568646 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:36:17.568336    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e9065062-773e-40bd-959b-c0c91b7de1f1-proxy-tls\") pod \"e9065062-773e-40bd-959b-c0c91b7de1f1\" (UID: \"e9065062-773e-40bd-959b-c0c91b7de1f1\") "
May 25 11:36:17.568646 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:36:17.568396    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e9065062-773e-40bd-959b-c0c91b7de1f1-kserve-provision-location\") pod \"e9065062-773e-40bd-959b-c0c91b7de1f1\" (UID: \"e9065062-773e-40bd-959b-c0c91b7de1f1\") "
May 25 11:36:17.568646 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:36:17.568574    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9065062-773e-40bd-959b-c0c91b7de1f1-isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config") pod "e9065062-773e-40bd-959b-c0c91b7de1f1" (UID: "e9065062-773e-40bd-959b-c0c91b7de1f1"). InnerVolumeSpecName "isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 11:36:17.568855 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:36:17.568735    2561 reconciler_common.go:299] "Volume detached for volume \"isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e9065062-773e-40bd-959b-c0c91b7de1f1-isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-227.ec2.internal\" DevicePath \"\""
May 25 11:36:17.568855 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:36:17.568764    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9065062-773e-40bd-959b-c0c91b7de1f1-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e9065062-773e-40bd-959b-c0c91b7de1f1" (UID: "e9065062-773e-40bd-959b-c0c91b7de1f1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:36:17.570397 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:36:17.570369    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9065062-773e-40bd-959b-c0c91b7de1f1-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e9065062-773e-40bd-959b-c0c91b7de1f1" (UID: "e9065062-773e-40bd-959b-c0c91b7de1f1"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 11:36:17.570397 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:36:17.570383    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9065062-773e-40bd-959b-c0c91b7de1f1-kube-api-access-7xjld" (OuterVolumeSpecName: "kube-api-access-7xjld") pod "e9065062-773e-40bd-959b-c0c91b7de1f1" (UID: "e9065062-773e-40bd-959b-c0c91b7de1f1"). InnerVolumeSpecName "kube-api-access-7xjld". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 11:36:17.669610 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:36:17.669584    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7xjld\" (UniqueName: \"kubernetes.io/projected/e9065062-773e-40bd-959b-c0c91b7de1f1-kube-api-access-7xjld\") on node \"ip-10-0-136-227.ec2.internal\" DevicePath \"\""
May 25 11:36:17.669610 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:36:17.669607    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e9065062-773e-40bd-959b-c0c91b7de1f1-proxy-tls\") on node \"ip-10-0-136-227.ec2.internal\" DevicePath \"\""
May 25 11:36:17.669763 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:36:17.669618    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e9065062-773e-40bd-959b-c0c91b7de1f1-kserve-provision-location\") on node \"ip-10-0-136-227.ec2.internal\" DevicePath \"\""
May 25 11:36:18.238781 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:36:18.238742    2561 generic.go:358] "Generic (PLEG): container finished" podID="e9065062-773e-40bd-959b-c0c91b7de1f1" containerID="be6df7ceadf6432bbc50bfc92d93a0fd5a140bb1eb95ad0b7b438b93d0df240f" exitCode=0
May 25 11:36:18.238977 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:36:18.238817    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-bhq2s" event={"ID":"e9065062-773e-40bd-959b-c0c91b7de1f1","Type":"ContainerDied","Data":"be6df7ceadf6432bbc50bfc92d93a0fd5a140bb1eb95ad0b7b438b93d0df240f"}
May 25 11:36:18.238977 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:36:18.238845    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-bhq2s" event={"ID":"e9065062-773e-40bd-959b-c0c91b7de1f1","Type":"ContainerDied","Data":"be1a4c36950832ab2636a3b5b47e9ddd57fad95a9010dda6cff3dff96b9ff02a"}
May 25 11:36:18.238977 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:36:18.238854    2561 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-bhq2s"
May 25 11:36:18.238977 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:36:18.238860    2561 scope.go:117] "RemoveContainer" containerID="08e5d21d9899cb67ee0bdb29732328fd3cdeb1ea434f72f3d66cf6bf741e0525"
May 25 11:36:18.247913 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:36:18.247896    2561 scope.go:117] "RemoveContainer" containerID="be6df7ceadf6432bbc50bfc92d93a0fd5a140bb1eb95ad0b7b438b93d0df240f"
May 25 11:36:18.255017 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:36:18.255002    2561 scope.go:117] "RemoveContainer" containerID="be69fe8941d8c4bfe3430a9bbc05713d82d394c1d6b9d6644efd43ab9160ef7f"
May 25 11:36:18.262204 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:36:18.262180    2561 scope.go:117] "RemoveContainer" containerID="08e5d21d9899cb67ee0bdb29732328fd3cdeb1ea434f72f3d66cf6bf741e0525"
May 25 11:36:18.262480 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:36:18.262459    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08e5d21d9899cb67ee0bdb29732328fd3cdeb1ea434f72f3d66cf6bf741e0525\": container with ID starting with 08e5d21d9899cb67ee0bdb29732328fd3cdeb1ea434f72f3d66cf6bf741e0525 not found: ID does not exist" containerID="08e5d21d9899cb67ee0bdb29732328fd3cdeb1ea434f72f3d66cf6bf741e0525"
May 25 11:36:18.262556 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:36:18.262488    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08e5d21d9899cb67ee0bdb29732328fd3cdeb1ea434f72f3d66cf6bf741e0525"} err="failed to get container status \"08e5d21d9899cb67ee0bdb29732328fd3cdeb1ea434f72f3d66cf6bf741e0525\": rpc error: code = NotFound desc = could not find container \"08e5d21d9899cb67ee0bdb29732328fd3cdeb1ea434f72f3d66cf6bf741e0525\": container with ID starting with 08e5d21d9899cb67ee0bdb29732328fd3cdeb1ea434f72f3d66cf6bf741e0525 not found: ID does not exist"
May 25 11:36:18.262556 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:36:18.262506    2561 scope.go:117] "RemoveContainer" containerID="be6df7ceadf6432bbc50bfc92d93a0fd5a140bb1eb95ad0b7b438b93d0df240f"
May 25 11:36:18.262691 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:36:18.262613    2561 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-bhq2s"]
May 25 11:36:18.262741 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:36:18.262717    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be6df7ceadf6432bbc50bfc92d93a0fd5a140bb1eb95ad0b7b438b93d0df240f\": container with ID starting with be6df7ceadf6432bbc50bfc92d93a0fd5a140bb1eb95ad0b7b438b93d0df240f not found: ID does not exist" containerID="be6df7ceadf6432bbc50bfc92d93a0fd5a140bb1eb95ad0b7b438b93d0df240f"
May 25 11:36:18.262803 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:36:18.262738    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be6df7ceadf6432bbc50bfc92d93a0fd5a140bb1eb95ad0b7b438b93d0df240f"} err="failed to get container status \"be6df7ceadf6432bbc50bfc92d93a0fd5a140bb1eb95ad0b7b438b93d0df240f\": rpc error: code = NotFound desc = could not find container \"be6df7ceadf6432bbc50bfc92d93a0fd5a140bb1eb95ad0b7b438b93d0df240f\": container with ID starting with be6df7ceadf6432bbc50bfc92d93a0fd5a140bb1eb95ad0b7b438b93d0df240f not found: ID does not exist"
May 25 11:36:18.262803 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:36:18.262751    2561 scope.go:117] "RemoveContainer" containerID="be69fe8941d8c4bfe3430a9bbc05713d82d394c1d6b9d6644efd43ab9160ef7f"
May 25 11:36:18.262980 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:36:18.262962    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be69fe8941d8c4bfe3430a9bbc05713d82d394c1d6b9d6644efd43ab9160ef7f\": container with ID starting with be69fe8941d8c4bfe3430a9bbc05713d82d394c1d6b9d6644efd43ab9160ef7f not found: ID does not exist" containerID="be69fe8941d8c4bfe3430a9bbc05713d82d394c1d6b9d6644efd43ab9160ef7f"
May 25 11:36:18.263038 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:36:18.262997    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be69fe8941d8c4bfe3430a9bbc05713d82d394c1d6b9d6644efd43ab9160ef7f"} err="failed to get container status \"be69fe8941d8c4bfe3430a9bbc05713d82d394c1d6b9d6644efd43ab9160ef7f\": rpc error: code = NotFound desc = could not find container \"be69fe8941d8c4bfe3430a9bbc05713d82d394c1d6b9d6644efd43ab9160ef7f\": container with ID starting with be69fe8941d8c4bfe3430a9bbc05713d82d394c1d6b9d6644efd43ab9160ef7f not found: ID does not exist"
May 25 11:36:18.267148 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:36:18.267127    2561 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-8765c9667-bhq2s"]
May 25 11:36:19.117544 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:36:19.117512    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e9065062-773e-40bd-959b-c0c91b7de1f1" path="/var/lib/kubelet/pods/e9065062-773e-40bd-959b-c0c91b7de1f1/volumes"
May 25 11:36:46.824559 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:36:46.824525    2561 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-wmndf"]
May 25 11:36:46.825170 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:36:46.825098    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="791fa6a6-1afd-4d66-bf37-846c657cc7ee" containerName="storage-initializer"
May 25 11:36:46.825170 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:36:46.825132    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="791fa6a6-1afd-4d66-bf37-846c657cc7ee" containerName="storage-initializer"
May 25 11:36:46.825170 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:36:46.825145    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9065062-773e-40bd-959b-c0c91b7de1f1" containerName="kube-rbac-proxy"
May 25 11:36:46.825170 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:36:46.825154    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9065062-773e-40bd-959b-c0c91b7de1f1" containerName="kube-rbac-proxy"
May 25 11:36:46.825170 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:36:46.825169    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9065062-773e-40bd-959b-c0c91b7de1f1" containerName="storage-initializer"
May 25 11:36:46.825485 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:36:46.825178    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9065062-773e-40bd-959b-c0c91b7de1f1" containerName="storage-initializer"
May 25 11:36:46.825485 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:36:46.825191    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="791fa6a6-1afd-4d66-bf37-846c657cc7ee" containerName="kserve-container"
May 25 11:36:46.825485 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:36:46.825201    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="791fa6a6-1afd-4d66-bf37-846c657cc7ee" containerName="kserve-container"
May 25 11:36:46.825485 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:36:46.825211    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9065062-773e-40bd-959b-c0c91b7de1f1" containerName="kserve-container"
May 25 11:36:46.825485 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:36:46.825219    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9065062-773e-40bd-959b-c0c91b7de1f1" containerName="kserve-container"
May 25 11:36:46.825485 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:36:46.825259    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="791fa6a6-1afd-4d66-bf37-846c657cc7ee" containerName="kube-rbac-proxy"
May 25 11:36:46.825485 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:36:46.825283    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="791fa6a6-1afd-4d66-bf37-846c657cc7ee" containerName="kube-rbac-proxy"
May 25 11:36:46.825485 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:36:46.825376    2561 memory_manager.go:356] "RemoveStaleState removing state" podUID="791fa6a6-1afd-4d66-bf37-846c657cc7ee" containerName="kserve-container"
May 25 11:36:46.825485 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:36:46.825391    2561 memory_manager.go:356] "RemoveStaleState removing state" podUID="e9065062-773e-40bd-959b-c0c91b7de1f1" containerName="kube-rbac-proxy"
May 25 11:36:46.825485 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:36:46.825403    2561 memory_manager.go:356] "RemoveStaleState removing state" podUID="791fa6a6-1afd-4d66-bf37-846c657cc7ee" containerName="kube-rbac-proxy"
May 25 11:36:46.825485 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:36:46.825418    2561 memory_manager.go:356] "RemoveStaleState removing state" podUID="e9065062-773e-40bd-959b-c0c91b7de1f1" containerName="kserve-container"
May 25 11:36:46.827745 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:36:46.827724    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-wmndf"
May 25 11:36:46.833186 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:36:46.833163    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config\""
May 25 11:36:46.833387 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:36:46.833251    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
May 25 11:36:46.833387 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:36:46.833205    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-mlflow-v2-runtime-predictor-serving-cert\""
May 25 11:36:46.833636 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:36:46.833621    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-svs62\""
May 25 11:36:46.833884 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:36:46.833867    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
May 25 11:36:46.840396 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:36:46.840375    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-wmndf"]
May 25 11:36:46.987406 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:36:46.987375    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f7349a3e-b649-477b-bd08-8ecc4970d244-kserve-provision-location\") pod \"isvc-mlflow-v2-runtime-predictor-5fdb47d546-wmndf\" (UID: \"f7349a3e-b649-477b-bd08-8ecc4970d244\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-wmndf"
May 25 11:36:46.987552 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:36:46.987408    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f7349a3e-b649-477b-bd08-8ecc4970d244-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-5fdb47d546-wmndf\" (UID: \"f7349a3e-b649-477b-bd08-8ecc4970d244\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-wmndf"
May 25 11:36:46.987552 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:36:46.987449    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x26j8\" (UniqueName: \"kubernetes.io/projected/f7349a3e-b649-477b-bd08-8ecc4970d244-kube-api-access-x26j8\") pod \"isvc-mlflow-v2-runtime-predictor-5fdb47d546-wmndf\" (UID: \"f7349a3e-b649-477b-bd08-8ecc4970d244\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-wmndf"
May 25 11:36:46.987552 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:36:46.987501    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f7349a3e-b649-477b-bd08-8ecc4970d244-isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-mlflow-v2-runtime-predictor-5fdb47d546-wmndf\" (UID: \"f7349a3e-b649-477b-bd08-8ecc4970d244\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-wmndf"
May 25 11:36:47.088522 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:36:47.088463    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f7349a3e-b649-477b-bd08-8ecc4970d244-isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-mlflow-v2-runtime-predictor-5fdb47d546-wmndf\" (UID: \"f7349a3e-b649-477b-bd08-8ecc4970d244\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-wmndf"
May 25 11:36:47.088634 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:36:47.088522    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f7349a3e-b649-477b-bd08-8ecc4970d244-kserve-provision-location\") pod \"isvc-mlflow-v2-runtime-predictor-5fdb47d546-wmndf\" (UID: \"f7349a3e-b649-477b-bd08-8ecc4970d244\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-wmndf"
May 25 11:36:47.088634 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:36:47.088545    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f7349a3e-b649-477b-bd08-8ecc4970d244-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-5fdb47d546-wmndf\" (UID: \"f7349a3e-b649-477b-bd08-8ecc4970d244\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-wmndf"
May 25 11:36:47.088634 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:36:47.088567    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-x26j8\" (UniqueName: \"kubernetes.io/projected/f7349a3e-b649-477b-bd08-8ecc4970d244-kube-api-access-x26j8\") pod \"isvc-mlflow-v2-runtime-predictor-5fdb47d546-wmndf\" (UID: \"f7349a3e-b649-477b-bd08-8ecc4970d244\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-wmndf"
May 25 11:36:47.088955 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:36:47.088937    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f7349a3e-b649-477b-bd08-8ecc4970d244-kserve-provision-location\") pod \"isvc-mlflow-v2-runtime-predictor-5fdb47d546-wmndf\" (UID: \"f7349a3e-b649-477b-bd08-8ecc4970d244\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-wmndf"
May 25 11:36:47.089080 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:36:47.089060    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f7349a3e-b649-477b-bd08-8ecc4970d244-isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-mlflow-v2-runtime-predictor-5fdb47d546-wmndf\" (UID: \"f7349a3e-b649-477b-bd08-8ecc4970d244\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-wmndf"
May 25 11:36:47.090948 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:36:47.090931    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f7349a3e-b649-477b-bd08-8ecc4970d244-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-5fdb47d546-wmndf\" (UID: \"f7349a3e-b649-477b-bd08-8ecc4970d244\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-wmndf"
May 25 11:36:47.099619 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:36:47.099598    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x26j8\" (UniqueName: \"kubernetes.io/projected/f7349a3e-b649-477b-bd08-8ecc4970d244-kube-api-access-x26j8\") pod \"isvc-mlflow-v2-runtime-predictor-5fdb47d546-wmndf\" (UID: \"f7349a3e-b649-477b-bd08-8ecc4970d244\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-wmndf"
May 25 11:36:47.137430 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:36:47.137402    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-wmndf"
May 25 11:36:47.272067 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:36:47.272040    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-wmndf"]
May 25 11:36:47.279477 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:36:47.279451    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7349a3e_b649_477b_bd08_8ecc4970d244.slice/crio-b222080bf49fd94ddf865561494f79ff3a25e3f2409b3cdd132b2bbc160edbab WatchSource:0}: Error finding container b222080bf49fd94ddf865561494f79ff3a25e3f2409b3cdd132b2bbc160edbab: Status 404 returned error can't find the container with id b222080bf49fd94ddf865561494f79ff3a25e3f2409b3cdd132b2bbc160edbab
May 25 11:36:47.337060 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:36:47.337034    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-wmndf" event={"ID":"f7349a3e-b649-477b-bd08-8ecc4970d244","Type":"ContainerStarted","Data":"b222080bf49fd94ddf865561494f79ff3a25e3f2409b3cdd132b2bbc160edbab"}
May 25 11:36:48.341554 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:36:48.341518    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-wmndf" event={"ID":"f7349a3e-b649-477b-bd08-8ecc4970d244","Type":"ContainerStarted","Data":"2e4a0e95be4742d9aae288261d144ac19d8e7ab4f1152d287df09830fd79454b"}
May 25 11:36:51.353221 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:36:51.353195    2561 generic.go:358] "Generic (PLEG): container finished" podID="f7349a3e-b649-477b-bd08-8ecc4970d244" containerID="2e4a0e95be4742d9aae288261d144ac19d8e7ab4f1152d287df09830fd79454b" exitCode=0
May 25 11:36:51.353548 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:36:51.353285    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-wmndf" event={"ID":"f7349a3e-b649-477b-bd08-8ecc4970d244","Type":"ContainerDied","Data":"2e4a0e95be4742d9aae288261d144ac19d8e7ab4f1152d287df09830fd79454b"}
May 25 11:36:52.358646 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:36:52.358604    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-wmndf" event={"ID":"f7349a3e-b649-477b-bd08-8ecc4970d244","Type":"ContainerStarted","Data":"3e4aef64dd12117d2c2f6044bd1ed01a62a850dac1b383b96b64bb4473af9900"}
May 25 11:36:52.359113 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:36:52.358659    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-wmndf" event={"ID":"f7349a3e-b649-477b-bd08-8ecc4970d244","Type":"ContainerStarted","Data":"2d4ac8f183427e63b5f8b22b6b26c908bc30b08b706109d4ce390b09ada2f58f"}
May 25 11:36:52.359113 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:36:52.358908    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-wmndf"
May 25 11:36:52.383938 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:36:52.383883    2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-wmndf" podStartSLOduration=6.383864734 podStartE2EDuration="6.383864734s" podCreationTimestamp="2026-05-25 11:36:46 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 11:36:52.380446067 +0000 UTC m=+1185.925623670" watchObservedRunningTime="2026-05-25 11:36:52.383864734 +0000 UTC m=+1185.929042338"
May 25 11:36:53.364610 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:36:53.364578    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-wmndf"
May 25 11:36:59.374061 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:36:59.374029    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-wmndf"
May 25 11:37:07.062055 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:37:07.062030    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-9lj95_10a66344-3304-49a1-bbc1-b885800f5acf/console-operator/1.log"
May 25 11:37:07.065596 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:37:07.065563    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-9lj95_10a66344-3304-49a1-bbc1-b885800f5acf/console-operator/1.log"
May 25 11:37:29.378005 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:37:29.377971    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-wmndf"
May 25 11:37:36.868930 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:37:36.868891    2561 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-wmndf"]
May 25 11:37:36.869463 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:37:36.869226    2561 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-wmndf" podUID="f7349a3e-b649-477b-bd08-8ecc4970d244" containerName="kserve-container" containerID="cri-o://2d4ac8f183427e63b5f8b22b6b26c908bc30b08b706109d4ce390b09ada2f58f" gracePeriod=30
May 25 11:37:36.869463 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:37:36.869298    2561 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-wmndf" podUID="f7349a3e-b649-477b-bd08-8ecc4970d244" containerName="kube-rbac-proxy" containerID="cri-o://3e4aef64dd12117d2c2f6044bd1ed01a62a850dac1b383b96b64bb4473af9900" gracePeriod=30
May 25 11:37:36.985003 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:37:36.984969    2561 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7785464fbb-w4m22"]
May 25 11:37:36.988689 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:37:36.988672    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7785464fbb-w4m22"
May 25 11:37:36.991249 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:37:36.991199    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-mcp-predictor-serving-cert\""
May 25 11:37:36.991458 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:37:36.991441    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-mcp-kube-rbac-proxy-sar-config\""
May 25 11:37:37.001118 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:37:37.001093    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7785464fbb-w4m22"]
May 25 11:37:37.101236 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:37:37.101196    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/df528f46-e9b9-4050-bd49-87f158d5be9f-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-7785464fbb-w4m22\" (UID: \"df528f46-e9b9-4050-bd49-87f158d5be9f\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7785464fbb-w4m22"
May 25 11:37:37.101419 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:37:37.101260    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ck8ps\" (UniqueName: \"kubernetes.io/projected/df528f46-e9b9-4050-bd49-87f158d5be9f-kube-api-access-ck8ps\") pod \"isvc-sklearn-mcp-predictor-7785464fbb-w4m22\" (UID: \"df528f46-e9b9-4050-bd49-87f158d5be9f\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7785464fbb-w4m22"
May 25 11:37:37.101419 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:37:37.101375    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/df528f46-e9b9-4050-bd49-87f158d5be9f-kserve-provision-location\") pod \"isvc-sklearn-mcp-predictor-7785464fbb-w4m22\" (UID: \"df528f46-e9b9-4050-bd49-87f158d5be9f\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7785464fbb-w4m22"
May 25 11:37:37.101501 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:37:37.101417    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-sklearn-mcp-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/df528f46-e9b9-4050-bd49-87f158d5be9f-isvc-sklearn-mcp-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-mcp-predictor-7785464fbb-w4m22\" (UID: \"df528f46-e9b9-4050-bd49-87f158d5be9f\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7785464fbb-w4m22"
May 25 11:37:37.202570 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:37:37.202479    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/df528f46-e9b9-4050-bd49-87f158d5be9f-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-7785464fbb-w4m22\" (UID: \"df528f46-e9b9-4050-bd49-87f158d5be9f\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7785464fbb-w4m22"
May 25 11:37:37.202570 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:37:37.202543    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-ck8ps\" (UniqueName: \"kubernetes.io/projected/df528f46-e9b9-4050-bd49-87f158d5be9f-kube-api-access-ck8ps\") pod \"isvc-sklearn-mcp-predictor-7785464fbb-w4m22\" (UID: \"df528f46-e9b9-4050-bd49-87f158d5be9f\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7785464fbb-w4m22"
May 25 11:37:37.202786 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:37:37.202580    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/df528f46-e9b9-4050-bd49-87f158d5be9f-kserve-provision-location\") pod \"isvc-sklearn-mcp-predictor-7785464fbb-w4m22\" (UID: \"df528f46-e9b9-4050-bd49-87f158d5be9f\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7785464fbb-w4m22"
May 25 11:37:37.202786 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:37:37.202605    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-sklearn-mcp-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/df528f46-e9b9-4050-bd49-87f158d5be9f-isvc-sklearn-mcp-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-mcp-predictor-7785464fbb-w4m22\" (UID: \"df528f46-e9b9-4050-bd49-87f158d5be9f\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7785464fbb-w4m22"
May 25 11:37:37.203049 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:37:37.203030    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/df528f46-e9b9-4050-bd49-87f158d5be9f-kserve-provision-location\") pod \"isvc-sklearn-mcp-predictor-7785464fbb-w4m22\" (UID: \"df528f46-e9b9-4050-bd49-87f158d5be9f\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7785464fbb-w4m22"
May 25 11:37:37.203226 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:37:37.203207    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-sklearn-mcp-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/df528f46-e9b9-4050-bd49-87f158d5be9f-isvc-sklearn-mcp-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-mcp-predictor-7785464fbb-w4m22\" (UID: \"df528f46-e9b9-4050-bd49-87f158d5be9f\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7785464fbb-w4m22"
May 25 11:37:37.204940 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:37:37.204917    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/df528f46-e9b9-4050-bd49-87f158d5be9f-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-7785464fbb-w4m22\" (UID: \"df528f46-e9b9-4050-bd49-87f158d5be9f\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7785464fbb-w4m22"
May 25 11:37:37.211518 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:37:37.211491    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ck8ps\" (UniqueName: \"kubernetes.io/projected/df528f46-e9b9-4050-bd49-87f158d5be9f-kube-api-access-ck8ps\") pod \"isvc-sklearn-mcp-predictor-7785464fbb-w4m22\" (UID: \"df528f46-e9b9-4050-bd49-87f158d5be9f\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7785464fbb-w4m22"
May 25 11:37:37.300146 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:37:37.300112    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7785464fbb-w4m22"
May 25 11:37:37.432825 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:37:37.432799    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7785464fbb-w4m22"]
May 25 11:37:37.435517 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:37:37.435488    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf528f46_e9b9_4050_bd49_87f158d5be9f.slice/crio-9dfa15ac59ee6f9fd71b9a28c7b950ef7a9927609eb4af8fc15502876d505189 WatchSource:0}: Error finding container 9dfa15ac59ee6f9fd71b9a28c7b950ef7a9927609eb4af8fc15502876d505189: Status 404 returned error can't find the container with id 9dfa15ac59ee6f9fd71b9a28c7b950ef7a9927609eb4af8fc15502876d505189
May 25 11:37:37.515811 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:37:37.515776    2561 generic.go:358] "Generic (PLEG): container finished" podID="f7349a3e-b649-477b-bd08-8ecc4970d244" containerID="3e4aef64dd12117d2c2f6044bd1ed01a62a850dac1b383b96b64bb4473af9900" exitCode=2
May 25 11:37:37.515972 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:37:37.515846    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-wmndf" event={"ID":"f7349a3e-b649-477b-bd08-8ecc4970d244","Type":"ContainerDied","Data":"3e4aef64dd12117d2c2f6044bd1ed01a62a850dac1b383b96b64bb4473af9900"}
May 25 11:37:37.517436 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:37:37.517409    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7785464fbb-w4m22" event={"ID":"df528f46-e9b9-4050-bd49-87f158d5be9f","Type":"ContainerStarted","Data":"8c07652fb9163669bb47b1e2b4ce69bb3d7eeb8d0f55ef1f1da3c377cded3488"}
May 25 11:37:37.517557 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:37:37.517443    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7785464fbb-w4m22" event={"ID":"df528f46-e9b9-4050-bd49-87f158d5be9f","Type":"ContainerStarted","Data":"9dfa15ac59ee6f9fd71b9a28c7b950ef7a9927609eb4af8fc15502876d505189"}
May 25 11:37:38.060483 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:37:38.060463    2561 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-wmndf"
May 25 11:37:38.211676 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:37:38.211593    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f7349a3e-b649-477b-bd08-8ecc4970d244-kserve-provision-location\") pod \"f7349a3e-b649-477b-bd08-8ecc4970d244\" (UID: \"f7349a3e-b649-477b-bd08-8ecc4970d244\") "
May 25 11:37:38.211676 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:37:38.211669    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f7349a3e-b649-477b-bd08-8ecc4970d244-proxy-tls\") pod \"f7349a3e-b649-477b-bd08-8ecc4970d244\" (UID: \"f7349a3e-b649-477b-bd08-8ecc4970d244\") "
May 25 11:37:38.211853 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:37:38.211705    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x26j8\" (UniqueName: \"kubernetes.io/projected/f7349a3e-b649-477b-bd08-8ecc4970d244-kube-api-access-x26j8\") pod \"f7349a3e-b649-477b-bd08-8ecc4970d244\" (UID: \"f7349a3e-b649-477b-bd08-8ecc4970d244\") "
May 25 11:37:38.211853 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:37:38.211730    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f7349a3e-b649-477b-bd08-8ecc4970d244-isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config\") pod \"f7349a3e-b649-477b-bd08-8ecc4970d244\" (UID: \"f7349a3e-b649-477b-bd08-8ecc4970d244\") "
May 25 11:37:38.212097 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:37:38.212070    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7349a3e-b649-477b-bd08-8ecc4970d244-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f7349a3e-b649-477b-bd08-8ecc4970d244" (UID: "f7349a3e-b649-477b-bd08-8ecc4970d244"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:37:38.212160 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:37:38.212103    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7349a3e-b649-477b-bd08-8ecc4970d244-isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config") pod "f7349a3e-b649-477b-bd08-8ecc4970d244" (UID: "f7349a3e-b649-477b-bd08-8ecc4970d244"). InnerVolumeSpecName "isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 11:37:38.213740 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:37:38.213715    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7349a3e-b649-477b-bd08-8ecc4970d244-kube-api-access-x26j8" (OuterVolumeSpecName: "kube-api-access-x26j8") pod "f7349a3e-b649-477b-bd08-8ecc4970d244" (UID: "f7349a3e-b649-477b-bd08-8ecc4970d244"). InnerVolumeSpecName "kube-api-access-x26j8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 11:37:38.213817 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:37:38.213771    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7349a3e-b649-477b-bd08-8ecc4970d244-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f7349a3e-b649-477b-bd08-8ecc4970d244" (UID: "f7349a3e-b649-477b-bd08-8ecc4970d244"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 11:37:38.312989 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:37:38.312957    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f7349a3e-b649-477b-bd08-8ecc4970d244-kserve-provision-location\") on node \"ip-10-0-136-227.ec2.internal\" DevicePath \"\""
May 25 11:37:38.312989 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:37:38.312984    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f7349a3e-b649-477b-bd08-8ecc4970d244-proxy-tls\") on node \"ip-10-0-136-227.ec2.internal\" DevicePath \"\""
May 25 11:37:38.312989 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:37:38.312995    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x26j8\" (UniqueName: \"kubernetes.io/projected/f7349a3e-b649-477b-bd08-8ecc4970d244-kube-api-access-x26j8\") on node \"ip-10-0-136-227.ec2.internal\" DevicePath \"\""
May 25 11:37:38.313219 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:37:38.313004    2561 reconciler_common.go:299] "Volume detached for volume \"isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f7349a3e-b649-477b-bd08-8ecc4970d244-isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-227.ec2.internal\" DevicePath \"\""
May 25 11:37:38.522435 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:37:38.522254    2561 generic.go:358] "Generic (PLEG): container finished" podID="f7349a3e-b649-477b-bd08-8ecc4970d244" containerID="2d4ac8f183427e63b5f8b22b6b26c908bc30b08b706109d4ce390b09ada2f58f" exitCode=0
May 25 11:37:38.522435 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:37:38.522299    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-wmndf" event={"ID":"f7349a3e-b649-477b-bd08-8ecc4970d244","Type":"ContainerDied","Data":"2d4ac8f183427e63b5f8b22b6b26c908bc30b08b706109d4ce390b09ada2f58f"}
May 25 11:37:38.522435 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:37:38.522343    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-wmndf" event={"ID":"f7349a3e-b649-477b-bd08-8ecc4970d244","Type":"ContainerDied","Data":"b222080bf49fd94ddf865561494f79ff3a25e3f2409b3cdd132b2bbc160edbab"}
May 25 11:37:38.522435 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:37:38.522358    2561 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-wmndf"
May 25 11:37:38.557076 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:37:38.522361    2561 scope.go:117] "RemoveContainer" containerID="3e4aef64dd12117d2c2f6044bd1ed01a62a850dac1b383b96b64bb4473af9900"
May 25 11:37:38.557076 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:37:38.546739    2561 scope.go:117] "RemoveContainer" containerID="2d4ac8f183427e63b5f8b22b6b26c908bc30b08b706109d4ce390b09ada2f58f"
May 25 11:37:38.557076 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:37:38.554449    2561 scope.go:117] "RemoveContainer" containerID="2e4a0e95be4742d9aae288261d144ac19d8e7ab4f1152d287df09830fd79454b"
May 25 11:37:38.561829 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:37:38.561802    2561 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-wmndf"]
May 25 11:37:38.562326 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:37:38.562288    2561 scope.go:117] "RemoveContainer" containerID="3e4aef64dd12117d2c2f6044bd1ed01a62a850dac1b383b96b64bb4473af9900"
May 25 11:37:38.562594 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:37:38.562577    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e4aef64dd12117d2c2f6044bd1ed01a62a850dac1b383b96b64bb4473af9900\": container with ID starting with 3e4aef64dd12117d2c2f6044bd1ed01a62a850dac1b383b96b64bb4473af9900 not found: ID does not exist" containerID="3e4aef64dd12117d2c2f6044bd1ed01a62a850dac1b383b96b64bb4473af9900"
May 25 11:37:38.562671 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:37:38.562606    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e4aef64dd12117d2c2f6044bd1ed01a62a850dac1b383b96b64bb4473af9900"} err="failed to get container status \"3e4aef64dd12117d2c2f6044bd1ed01a62a850dac1b383b96b64bb4473af9900\": rpc error: code = NotFound desc = could not find container \"3e4aef64dd12117d2c2f6044bd1ed01a62a850dac1b383b96b64bb4473af9900\": container with ID starting with 3e4aef64dd12117d2c2f6044bd1ed01a62a850dac1b383b96b64bb4473af9900 not found: ID does not exist"
May 25 11:37:38.562671 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:37:38.562633    2561 scope.go:117] "RemoveContainer" containerID="2d4ac8f183427e63b5f8b22b6b26c908bc30b08b706109d4ce390b09ada2f58f"
May 25 11:37:38.562865 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:37:38.562845    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d4ac8f183427e63b5f8b22b6b26c908bc30b08b706109d4ce390b09ada2f58f\": container with ID starting with 2d4ac8f183427e63b5f8b22b6b26c908bc30b08b706109d4ce390b09ada2f58f not found: ID does not exist" containerID="2d4ac8f183427e63b5f8b22b6b26c908bc30b08b706109d4ce390b09ada2f58f"
May 25 11:37:38.562905 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:37:38.562872    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d4ac8f183427e63b5f8b22b6b26c908bc30b08b706109d4ce390b09ada2f58f"} err="failed to get container status \"2d4ac8f183427e63b5f8b22b6b26c908bc30b08b706109d4ce390b09ada2f58f\": rpc error: code = NotFound desc = could not find container \"2d4ac8f183427e63b5f8b22b6b26c908bc30b08b706109d4ce390b09ada2f58f\": container with ID starting with 2d4ac8f183427e63b5f8b22b6b26c908bc30b08b706109d4ce390b09ada2f58f not found: ID does not exist"
May 25 11:37:38.562905 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:37:38.562889    2561 scope.go:117] "RemoveContainer" containerID="2e4a0e95be4742d9aae288261d144ac19d8e7ab4f1152d287df09830fd79454b"
May 25 11:37:38.563223 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:37:38.563160    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e4a0e95be4742d9aae288261d144ac19d8e7ab4f1152d287df09830fd79454b\": container with ID starting with 2e4a0e95be4742d9aae288261d144ac19d8e7ab4f1152d287df09830fd79454b not found: ID does not exist" containerID="2e4a0e95be4742d9aae288261d144ac19d8e7ab4f1152d287df09830fd79454b"
May 25 11:37:38.563223 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:37:38.563199    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e4a0e95be4742d9aae288261d144ac19d8e7ab4f1152d287df09830fd79454b"} err="failed to get container status \"2e4a0e95be4742d9aae288261d144ac19d8e7ab4f1152d287df09830fd79454b\": rpc error: code = NotFound desc = could not find container \"2e4a0e95be4742d9aae288261d144ac19d8e7ab4f1152d287df09830fd79454b\": container with ID starting with 2e4a0e95be4742d9aae288261d144ac19d8e7ab4f1152d287df09830fd79454b not found: ID does not exist"
May 25 11:37:38.565551 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:37:38.565529    2561 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-5fdb47d546-wmndf"]
May 25 11:37:39.116995 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:37:39.116962    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f7349a3e-b649-477b-bd08-8ecc4970d244" path="/var/lib/kubelet/pods/f7349a3e-b649-477b-bd08-8ecc4970d244/volumes"
May 25 11:37:41.539450 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:37:41.539416    2561 generic.go:358] "Generic (PLEG): container finished" podID="df528f46-e9b9-4050-bd49-87f158d5be9f" containerID="8c07652fb9163669bb47b1e2b4ce69bb3d7eeb8d0f55ef1f1da3c377cded3488" exitCode=0
May 25 11:37:41.539781 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:37:41.539492    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7785464fbb-w4m22" event={"ID":"df528f46-e9b9-4050-bd49-87f158d5be9f","Type":"ContainerDied","Data":"8c07652fb9163669bb47b1e2b4ce69bb3d7eeb8d0f55ef1f1da3c377cded3488"}
May 25 11:37:42.545793 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:37:42.545752    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7785464fbb-w4m22" event={"ID":"df528f46-e9b9-4050-bd49-87f158d5be9f","Type":"ContainerStarted","Data":"39254b25438ecb4349495a974507545aabf183b3a64e2b00270f6556ba294015"}
May 25 11:37:44.557321 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:37:44.557283    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7785464fbb-w4m22" event={"ID":"df528f46-e9b9-4050-bd49-87f158d5be9f","Type":"ContainerStarted","Data":"0a75398c0825e7e83eb3a67d21af4504de2a5e0a593771f4992d21f962b4d84b"}
May 25 11:37:44.557321 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:37:44.557324    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7785464fbb-w4m22" event={"ID":"df528f46-e9b9-4050-bd49-87f158d5be9f","Type":"ContainerStarted","Data":"5974fb4ae925af4036efddf4bf7f8a3757f8fcc5e0526c37091f2282314e5887"}
May 25 11:37:44.557729 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:37:44.557443    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7785464fbb-w4m22"
May 25 11:37:44.557729 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:37:44.557455    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7785464fbb-w4m22"
May 25 11:37:44.584206 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:37:44.584157    2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7785464fbb-w4m22" podStartSLOduration=6.4506880540000004 podStartE2EDuration="8.584139787s" podCreationTimestamp="2026-05-25 11:37:36 +0000 UTC" firstStartedPulling="2026-05-25 11:37:41.61649183 +0000 UTC m=+1235.161669410" lastFinishedPulling="2026-05-25 11:37:43.74994356 +0000 UTC m=+1237.295121143" observedRunningTime="2026-05-25 11:37:44.582211955 +0000 UTC m=+1238.127389559" watchObservedRunningTime="2026-05-25 11:37:44.584139787 +0000 UTC m=+1238.129317390"
May 25 11:37:45.560907 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:37:45.560878    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7785464fbb-w4m22"
May 25 11:37:51.569786 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:37:51.569751    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7785464fbb-w4m22"
May 25 11:38:11.570650 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:38:11.570608    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7785464fbb-w4m22" podUID="df528f46-e9b9-4050-bd49-87f158d5be9f" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.43:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.132.0.43:8080: connect: connection refused"
May 25 11:38:21.572067 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:38:21.572032    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7785464fbb-w4m22"
May 25 11:38:51.573025 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:38:51.572991    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7785464fbb-w4m22"
May 25 11:38:57.036374 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:38:57.036339    2561 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7785464fbb-w4m22"]
May 25 11:38:57.036842 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:38:57.036677    2561 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7785464fbb-w4m22" podUID="df528f46-e9b9-4050-bd49-87f158d5be9f" containerName="kserve-container" containerID="cri-o://39254b25438ecb4349495a974507545aabf183b3a64e2b00270f6556ba294015" gracePeriod=30
May 25 11:38:57.036842 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:38:57.036705    2561 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7785464fbb-w4m22" podUID="df528f46-e9b9-4050-bd49-87f158d5be9f" containerName="kube-rbac-proxy" containerID="cri-o://0a75398c0825e7e83eb3a67d21af4504de2a5e0a593771f4992d21f962b4d84b" gracePeriod=30
May 25 11:38:57.036842 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:38:57.036755    2561 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7785464fbb-w4m22" podUID="df528f46-e9b9-4050-bd49-87f158d5be9f" containerName="kserve-agent" containerID="cri-o://5974fb4ae925af4036efddf4bf7f8a3757f8fcc5e0526c37091f2282314e5887" gracePeriod=30
May 25 11:38:57.813385 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:38:57.813347    2561 generic.go:358] "Generic (PLEG): container finished" podID="df528f46-e9b9-4050-bd49-87f158d5be9f" containerID="0a75398c0825e7e83eb3a67d21af4504de2a5e0a593771f4992d21f962b4d84b" exitCode=2
May 25 11:38:57.813555 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:38:57.813422    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7785464fbb-w4m22" event={"ID":"df528f46-e9b9-4050-bd49-87f158d5be9f","Type":"ContainerDied","Data":"0a75398c0825e7e83eb3a67d21af4504de2a5e0a593771f4992d21f962b4d84b"}
May 25 11:38:59.822135 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:38:59.822102    2561 generic.go:358] "Generic (PLEG): container finished" podID="df528f46-e9b9-4050-bd49-87f158d5be9f" containerID="39254b25438ecb4349495a974507545aabf183b3a64e2b00270f6556ba294015" exitCode=0
May 25 11:38:59.822510 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:38:59.822151    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7785464fbb-w4m22" event={"ID":"df528f46-e9b9-4050-bd49-87f158d5be9f","Type":"ContainerDied","Data":"39254b25438ecb4349495a974507545aabf183b3a64e2b00270f6556ba294015"}
May 25 11:39:01.565205 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:39:01.565163    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7785464fbb-w4m22" podUID="df528f46-e9b9-4050-bd49-87f158d5be9f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.43:8643/healthz\": dial tcp 10.132.0.43:8643: connect: connection refused"
May 25 11:39:01.571108 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:39:01.571081    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7785464fbb-w4m22" podUID="df528f46-e9b9-4050-bd49-87f158d5be9f" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.43:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.132.0.43:8080: connect: connection refused"
May 25 11:39:06.565357 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:39:06.565312    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7785464fbb-w4m22" podUID="df528f46-e9b9-4050-bd49-87f158d5be9f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.43:8643/healthz\": dial tcp 10.132.0.43:8643: connect: connection refused"
May 25 11:39:11.565618 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:39:11.565573    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7785464fbb-w4m22" podUID="df528f46-e9b9-4050-bd49-87f158d5be9f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.43:8643/healthz\": dial tcp 10.132.0.43:8643: connect: connection refused"
May 25 11:39:11.566073 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:39:11.565721    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7785464fbb-w4m22"
May 25 11:39:11.570693 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:39:11.570670    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7785464fbb-w4m22" podUID="df528f46-e9b9-4050-bd49-87f158d5be9f" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.43:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.132.0.43:8080: connect: connection refused"
May 25 11:39:16.565489 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:39:16.565446    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7785464fbb-w4m22" podUID="df528f46-e9b9-4050-bd49-87f158d5be9f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.43:8643/healthz\": dial tcp 10.132.0.43:8643: connect: connection refused"
May 25 11:39:21.565553 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:39:21.565503    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7785464fbb-w4m22" podUID="df528f46-e9b9-4050-bd49-87f158d5be9f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.43:8643/healthz\": dial tcp 10.132.0.43:8643: connect: connection refused"
May 25 11:39:21.571380 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:39:21.571346    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7785464fbb-w4m22" podUID="df528f46-e9b9-4050-bd49-87f158d5be9f" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.43:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.132.0.43:8080: connect: connection refused"
May 25 11:39:21.571521 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:39:21.571489    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7785464fbb-w4m22"
May 25 11:39:26.565571 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:39:26.565458    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7785464fbb-w4m22" podUID="df528f46-e9b9-4050-bd49-87f158d5be9f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.43:8643/healthz\": dial tcp 10.132.0.43:8643: connect: connection refused"
May 25 11:39:27.183773 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:39:27.183749    2561 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7785464fbb-w4m22"
May 25 11:39:27.282087 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:39:27.282055    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-sklearn-mcp-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/df528f46-e9b9-4050-bd49-87f158d5be9f-isvc-sklearn-mcp-kube-rbac-proxy-sar-config\") pod \"df528f46-e9b9-4050-bd49-87f158d5be9f\" (UID: \"df528f46-e9b9-4050-bd49-87f158d5be9f\") "
May 25 11:39:27.282297 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:39:27.282155    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/df528f46-e9b9-4050-bd49-87f158d5be9f-proxy-tls\") pod \"df528f46-e9b9-4050-bd49-87f158d5be9f\" (UID: \"df528f46-e9b9-4050-bd49-87f158d5be9f\") "
May 25 11:39:27.282297 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:39:27.282181    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/df528f46-e9b9-4050-bd49-87f158d5be9f-kserve-provision-location\") pod \"df528f46-e9b9-4050-bd49-87f158d5be9f\" (UID: \"df528f46-e9b9-4050-bd49-87f158d5be9f\") "
May 25 11:39:27.282297 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:39:27.282220    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ck8ps\" (UniqueName: \"kubernetes.io/projected/df528f46-e9b9-4050-bd49-87f158d5be9f-kube-api-access-ck8ps\") pod \"df528f46-e9b9-4050-bd49-87f158d5be9f\" (UID: \"df528f46-e9b9-4050-bd49-87f158d5be9f\") "
May 25 11:39:27.282564 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:39:27.282540    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df528f46-e9b9-4050-bd49-87f158d5be9f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "df528f46-e9b9-4050-bd49-87f158d5be9f" (UID: "df528f46-e9b9-4050-bd49-87f158d5be9f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:39:27.282629 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:39:27.282543    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df528f46-e9b9-4050-bd49-87f158d5be9f-isvc-sklearn-mcp-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-sklearn-mcp-kube-rbac-proxy-sar-config") pod "df528f46-e9b9-4050-bd49-87f158d5be9f" (UID: "df528f46-e9b9-4050-bd49-87f158d5be9f"). InnerVolumeSpecName "isvc-sklearn-mcp-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 11:39:27.284311 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:39:27.284281    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df528f46-e9b9-4050-bd49-87f158d5be9f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "df528f46-e9b9-4050-bd49-87f158d5be9f" (UID: "df528f46-e9b9-4050-bd49-87f158d5be9f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 11:39:27.284420 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:39:27.284282    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df528f46-e9b9-4050-bd49-87f158d5be9f-kube-api-access-ck8ps" (OuterVolumeSpecName: "kube-api-access-ck8ps") pod "df528f46-e9b9-4050-bd49-87f158d5be9f" (UID: "df528f46-e9b9-4050-bd49-87f158d5be9f"). InnerVolumeSpecName "kube-api-access-ck8ps". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 11:39:27.383247 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:39:27.383221    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ck8ps\" (UniqueName: \"kubernetes.io/projected/df528f46-e9b9-4050-bd49-87f158d5be9f-kube-api-access-ck8ps\") on node \"ip-10-0-136-227.ec2.internal\" DevicePath \"\""
May 25 11:39:27.383247 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:39:27.383245    2561 reconciler_common.go:299] "Volume detached for volume \"isvc-sklearn-mcp-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/df528f46-e9b9-4050-bd49-87f158d5be9f-isvc-sklearn-mcp-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-227.ec2.internal\" DevicePath \"\""
May 25 11:39:27.383438 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:39:27.383255    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/df528f46-e9b9-4050-bd49-87f158d5be9f-proxy-tls\") on node \"ip-10-0-136-227.ec2.internal\" DevicePath \"\""
May 25 11:39:27.383438 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:39:27.383284    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/df528f46-e9b9-4050-bd49-87f158d5be9f-kserve-provision-location\") on node \"ip-10-0-136-227.ec2.internal\" DevicePath \"\""
May 25 11:39:27.918790 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:39:27.918758    2561 generic.go:358] "Generic (PLEG): container finished" podID="df528f46-e9b9-4050-bd49-87f158d5be9f" containerID="5974fb4ae925af4036efddf4bf7f8a3757f8fcc5e0526c37091f2282314e5887" exitCode=0
May 25 11:39:27.919190 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:39:27.918848    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7785464fbb-w4m22" event={"ID":"df528f46-e9b9-4050-bd49-87f158d5be9f","Type":"ContainerDied","Data":"5974fb4ae925af4036efddf4bf7f8a3757f8fcc5e0526c37091f2282314e5887"}
May 25 11:39:27.919190 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:39:27.918874    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7785464fbb-w4m22" event={"ID":"df528f46-e9b9-4050-bd49-87f158d5be9f","Type":"ContainerDied","Data":"9dfa15ac59ee6f9fd71b9a28c7b950ef7a9927609eb4af8fc15502876d505189"}
May 25 11:39:27.919190 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:39:27.918888    2561 scope.go:117] "RemoveContainer" containerID="0a75398c0825e7e83eb3a67d21af4504de2a5e0a593771f4992d21f962b4d84b"
May 25 11:39:27.919190 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:39:27.918903    2561 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7785464fbb-w4m22"
May 25 11:39:27.927994 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:39:27.927968    2561 scope.go:117] "RemoveContainer" containerID="5974fb4ae925af4036efddf4bf7f8a3757f8fcc5e0526c37091f2282314e5887"
May 25 11:39:27.935946 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:39:27.935930    2561 scope.go:117] "RemoveContainer" containerID="39254b25438ecb4349495a974507545aabf183b3a64e2b00270f6556ba294015"
May 25 11:39:27.943776 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:39:27.943756    2561 scope.go:117] "RemoveContainer" containerID="8c07652fb9163669bb47b1e2b4ce69bb3d7eeb8d0f55ef1f1da3c377cded3488"
May 25 11:39:27.944572 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:39:27.944549    2561 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7785464fbb-w4m22"]
May 25 11:39:27.945124 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:39:27.945093    2561 cadvisor_stats_provider.go:525] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf528f46_e9b9_4050_bd49_87f158d5be9f.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf528f46_e9b9_4050_bd49_87f158d5be9f.slice/crio-9dfa15ac59ee6f9fd71b9a28c7b950ef7a9927609eb4af8fc15502876d505189\": RecentStats: unable to find data in memory cache]"
May 25 11:39:27.949654 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:39:27.949634    2561 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-7785464fbb-w4m22"]
May 25 11:39:27.952013 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:39:27.951986    2561 scope.go:117] "RemoveContainer" containerID="0a75398c0825e7e83eb3a67d21af4504de2a5e0a593771f4992d21f962b4d84b"
May 25 11:39:27.952285 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:39:27.952239    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a75398c0825e7e83eb3a67d21af4504de2a5e0a593771f4992d21f962b4d84b\": container with ID starting with 0a75398c0825e7e83eb3a67d21af4504de2a5e0a593771f4992d21f962b4d84b not found: ID does not exist" containerID="0a75398c0825e7e83eb3a67d21af4504de2a5e0a593771f4992d21f962b4d84b"
May 25 11:39:27.952356 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:39:27.952285    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a75398c0825e7e83eb3a67d21af4504de2a5e0a593771f4992d21f962b4d84b"} err="failed to get container status \"0a75398c0825e7e83eb3a67d21af4504de2a5e0a593771f4992d21f962b4d84b\": rpc error: code = NotFound desc = could not find container \"0a75398c0825e7e83eb3a67d21af4504de2a5e0a593771f4992d21f962b4d84b\": container with ID starting with 0a75398c0825e7e83eb3a67d21af4504de2a5e0a593771f4992d21f962b4d84b not found: ID does not exist"
May 25 11:39:27.952356 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:39:27.952311    2561 scope.go:117] "RemoveContainer" containerID="5974fb4ae925af4036efddf4bf7f8a3757f8fcc5e0526c37091f2282314e5887"
May 25 11:39:27.952568 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:39:27.952549    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5974fb4ae925af4036efddf4bf7f8a3757f8fcc5e0526c37091f2282314e5887\": container with ID starting with 5974fb4ae925af4036efddf4bf7f8a3757f8fcc5e0526c37091f2282314e5887 not found: ID does not exist" containerID="5974fb4ae925af4036efddf4bf7f8a3757f8fcc5e0526c37091f2282314e5887"
May 25 11:39:27.952633 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:39:27.952575    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5974fb4ae925af4036efddf4bf7f8a3757f8fcc5e0526c37091f2282314e5887"} err="failed to get container status \"5974fb4ae925af4036efddf4bf7f8a3757f8fcc5e0526c37091f2282314e5887\": rpc error: code = NotFound desc = could not find container \"5974fb4ae925af4036efddf4bf7f8a3757f8fcc5e0526c37091f2282314e5887\": container with ID starting with 5974fb4ae925af4036efddf4bf7f8a3757f8fcc5e0526c37091f2282314e5887 not found: ID does not exist"
May 25 11:39:27.952633 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:39:27.952595    2561 scope.go:117] "RemoveContainer" containerID="39254b25438ecb4349495a974507545aabf183b3a64e2b00270f6556ba294015"
May 25 11:39:27.952811 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:39:27.952795    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39254b25438ecb4349495a974507545aabf183b3a64e2b00270f6556ba294015\": container with ID starting with 39254b25438ecb4349495a974507545aabf183b3a64e2b00270f6556ba294015 not found: ID does not exist" containerID="39254b25438ecb4349495a974507545aabf183b3a64e2b00270f6556ba294015"
May 25 11:39:27.952873 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:39:27.952816    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39254b25438ecb4349495a974507545aabf183b3a64e2b00270f6556ba294015"} err="failed to get container status \"39254b25438ecb4349495a974507545aabf183b3a64e2b00270f6556ba294015\": rpc error: code = NotFound desc = could not find container \"39254b25438ecb4349495a974507545aabf183b3a64e2b00270f6556ba294015\": container with ID starting with 39254b25438ecb4349495a974507545aabf183b3a64e2b00270f6556ba294015 not found: ID does not exist"
May 25 11:39:27.952873 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:39:27.952835    2561 scope.go:117] "RemoveContainer" containerID="8c07652fb9163669bb47b1e2b4ce69bb3d7eeb8d0f55ef1f1da3c377cded3488"
May 25 11:39:27.953035 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:39:27.953017    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c07652fb9163669bb47b1e2b4ce69bb3d7eeb8d0f55ef1f1da3c377cded3488\": container with ID starting with 8c07652fb9163669bb47b1e2b4ce69bb3d7eeb8d0f55ef1f1da3c377cded3488 not found: ID does not exist" containerID="8c07652fb9163669bb47b1e2b4ce69bb3d7eeb8d0f55ef1f1da3c377cded3488"
May 25 11:39:27.953076 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:39:27.953041    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c07652fb9163669bb47b1e2b4ce69bb3d7eeb8d0f55ef1f1da3c377cded3488"} err="failed to get container status \"8c07652fb9163669bb47b1e2b4ce69bb3d7eeb8d0f55ef1f1da3c377cded3488\": rpc error: code = NotFound desc = could not find container \"8c07652fb9163669bb47b1e2b4ce69bb3d7eeb8d0f55ef1f1da3c377cded3488\": container with ID starting with 8c07652fb9163669bb47b1e2b4ce69bb3d7eeb8d0f55ef1f1da3c377cded3488 not found: ID does not exist"
May 25 11:39:29.118095 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:39:29.118060    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="df528f46-e9b9-4050-bd49-87f158d5be9f" path="/var/lib/kubelet/pods/df528f46-e9b9-4050-bd49-87f158d5be9f/volumes"
May 25 11:42:07.088192 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:42:07.088166    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-9lj95_10a66344-3304-49a1-bbc1-b885800f5acf/console-operator/1.log"
May 25 11:42:07.093213 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:42:07.093188    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-9lj95_10a66344-3304-49a1-bbc1-b885800f5acf/console-operator/1.log"
May 25 11:46:56.780178 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:46:56.780100    2561 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-primary-495374-predictor-cf965cf74-sdz6d"]
May 25 11:46:56.780586 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:46:56.780491    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7349a3e-b649-477b-bd08-8ecc4970d244" containerName="storage-initializer"
May 25 11:46:56.780586 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:46:56.780504    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7349a3e-b649-477b-bd08-8ecc4970d244" containerName="storage-initializer"
May 25 11:46:56.780586 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:46:56.780513    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7349a3e-b649-477b-bd08-8ecc4970d244" containerName="kserve-container"
May 25 11:46:56.780586 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:46:56.780518    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7349a3e-b649-477b-bd08-8ecc4970d244" containerName="kserve-container"
May 25 11:46:56.780586 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:46:56.780526    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df528f46-e9b9-4050-bd49-87f158d5be9f" containerName="kube-rbac-proxy"
May 25 11:46:56.780586 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:46:56.780531    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="df528f46-e9b9-4050-bd49-87f158d5be9f" containerName="kube-rbac-proxy"
May 25 11:46:56.780586 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:46:56.780547    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7349a3e-b649-477b-bd08-8ecc4970d244" containerName="kube-rbac-proxy"
May 25 11:46:56.780586 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:46:56.780555    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7349a3e-b649-477b-bd08-8ecc4970d244" containerName="kube-rbac-proxy"
May 25 11:46:56.780586 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:46:56.780561    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df528f46-e9b9-4050-bd49-87f158d5be9f" containerName="kserve-container"
May 25 11:46:56.780586 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:46:56.780566    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="df528f46-e9b9-4050-bd49-87f158d5be9f" containerName="kserve-container"
May 25 11:46:56.780889 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:46:56.780603    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df528f46-e9b9-4050-bd49-87f158d5be9f" containerName="storage-initializer"
May 25 11:46:56.780889 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:46:56.780609    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="df528f46-e9b9-4050-bd49-87f158d5be9f" containerName="storage-initializer"
May 25 11:46:56.780889 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:46:56.780620    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df528f46-e9b9-4050-bd49-87f158d5be9f" containerName="kserve-agent"
May 25 11:46:56.780889 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:46:56.780628    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="df528f46-e9b9-4050-bd49-87f158d5be9f" containerName="kserve-agent"
May 25 11:46:56.780889 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:46:56.780681    2561 memory_manager.go:356] "RemoveStaleState removing state" podUID="df528f46-e9b9-4050-bd49-87f158d5be9f" containerName="kube-rbac-proxy"
May 25 11:46:56.780889 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:46:56.780691    2561 memory_manager.go:356] "RemoveStaleState removing state" podUID="f7349a3e-b649-477b-bd08-8ecc4970d244" containerName="kserve-container"
May 25 11:46:56.780889 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:46:56.780703    2561 memory_manager.go:356] "RemoveStaleState removing state" podUID="df528f46-e9b9-4050-bd49-87f158d5be9f" containerName="kserve-agent"
May 25 11:46:56.780889 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:46:56.780713    2561 memory_manager.go:356] "RemoveStaleState removing state" podUID="df528f46-e9b9-4050-bd49-87f158d5be9f" containerName="kserve-container"
May 25 11:46:56.780889 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:46:56.780719    2561 memory_manager.go:356] "RemoveStaleState removing state" podUID="f7349a3e-b649-477b-bd08-8ecc4970d244" containerName="kube-rbac-proxy"
May 25 11:46:56.784011 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:46:56.783992    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-495374-predictor-cf965cf74-sdz6d"
May 25 11:46:56.787807 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:46:56.787776    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
May 25 11:46:56.787982 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:46:56.787964    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
May 25 11:46:56.788204 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:46:56.788180    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-495374-predictor-serving-cert\""
May 25 11:46:56.788405 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:46:56.788386    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-495374-kube-rbac-proxy-sar-config\""
May 25 11:46:56.788595 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:46:56.788453    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-svs62\""
May 25 11:46:56.795259 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:46:56.795228    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-495374-predictor-cf965cf74-sdz6d"]
May 25 11:46:56.809917 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:46:56.809889    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-primary-495374-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/88e76793-e05b-4318-8526-aea31883cca0-isvc-primary-495374-kube-rbac-proxy-sar-config\") pod \"isvc-primary-495374-predictor-cf965cf74-sdz6d\" (UID: \"88e76793-e05b-4318-8526-aea31883cca0\") " pod="kserve-ci-e2e-test/isvc-primary-495374-predictor-cf965cf74-sdz6d"
May 25 11:46:56.810090 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:46:56.809989    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5kx4\" (UniqueName: \"kubernetes.io/projected/88e76793-e05b-4318-8526-aea31883cca0-kube-api-access-t5kx4\") pod \"isvc-primary-495374-predictor-cf965cf74-sdz6d\" (UID: \"88e76793-e05b-4318-8526-aea31883cca0\") " pod="kserve-ci-e2e-test/isvc-primary-495374-predictor-cf965cf74-sdz6d"
May 25 11:46:56.810090 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:46:56.810038    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/88e76793-e05b-4318-8526-aea31883cca0-proxy-tls\") pod \"isvc-primary-495374-predictor-cf965cf74-sdz6d\" (UID: \"88e76793-e05b-4318-8526-aea31883cca0\") " pod="kserve-ci-e2e-test/isvc-primary-495374-predictor-cf965cf74-sdz6d"
May 25 11:46:56.810090 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:46:56.810063    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/88e76793-e05b-4318-8526-aea31883cca0-kserve-provision-location\") pod \"isvc-primary-495374-predictor-cf965cf74-sdz6d\" (UID: \"88e76793-e05b-4318-8526-aea31883cca0\") " pod="kserve-ci-e2e-test/isvc-primary-495374-predictor-cf965cf74-sdz6d"
May 25 11:46:56.911489 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:46:56.911447    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-t5kx4\" (UniqueName: \"kubernetes.io/projected/88e76793-e05b-4318-8526-aea31883cca0-kube-api-access-t5kx4\") pod \"isvc-primary-495374-predictor-cf965cf74-sdz6d\" (UID: \"88e76793-e05b-4318-8526-aea31883cca0\") " pod="kserve-ci-e2e-test/isvc-primary-495374-predictor-cf965cf74-sdz6d"
May 25 11:46:56.911646 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:46:56.911499    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/88e76793-e05b-4318-8526-aea31883cca0-proxy-tls\") pod \"isvc-primary-495374-predictor-cf965cf74-sdz6d\" (UID: \"88e76793-e05b-4318-8526-aea31883cca0\") " pod="kserve-ci-e2e-test/isvc-primary-495374-predictor-cf965cf74-sdz6d"
May 25 11:46:56.911646 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:46:56.911519    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/88e76793-e05b-4318-8526-aea31883cca0-kserve-provision-location\") pod \"isvc-primary-495374-predictor-cf965cf74-sdz6d\" (UID: \"88e76793-e05b-4318-8526-aea31883cca0\") " pod="kserve-ci-e2e-test/isvc-primary-495374-predictor-cf965cf74-sdz6d"
May 25 11:46:56.911646 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:46:56.911553    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-primary-495374-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/88e76793-e05b-4318-8526-aea31883cca0-isvc-primary-495374-kube-rbac-proxy-sar-config\") pod \"isvc-primary-495374-predictor-cf965cf74-sdz6d\" (UID: \"88e76793-e05b-4318-8526-aea31883cca0\") " pod="kserve-ci-e2e-test/isvc-primary-495374-predictor-cf965cf74-sdz6d"
May 25 11:46:56.912012 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:46:56.911987    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/88e76793-e05b-4318-8526-aea31883cca0-kserve-provision-location\") pod \"isvc-primary-495374-predictor-cf965cf74-sdz6d\" (UID: \"88e76793-e05b-4318-8526-aea31883cca0\") " pod="kserve-ci-e2e-test/isvc-primary-495374-predictor-cf965cf74-sdz6d"
May 25 11:46:56.912236 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:46:56.912216    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-primary-495374-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/88e76793-e05b-4318-8526-aea31883cca0-isvc-primary-495374-kube-rbac-proxy-sar-config\") pod \"isvc-primary-495374-predictor-cf965cf74-sdz6d\" (UID: \"88e76793-e05b-4318-8526-aea31883cca0\") " pod="kserve-ci-e2e-test/isvc-primary-495374-predictor-cf965cf74-sdz6d"
May 25 11:46:56.913836 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:46:56.913819    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/88e76793-e05b-4318-8526-aea31883cca0-proxy-tls\") pod \"isvc-primary-495374-predictor-cf965cf74-sdz6d\" (UID: \"88e76793-e05b-4318-8526-aea31883cca0\") " pod="kserve-ci-e2e-test/isvc-primary-495374-predictor-cf965cf74-sdz6d"
May 25 11:46:56.922356 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:46:56.922329    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5kx4\" (UniqueName: \"kubernetes.io/projected/88e76793-e05b-4318-8526-aea31883cca0-kube-api-access-t5kx4\") pod \"isvc-primary-495374-predictor-cf965cf74-sdz6d\" (UID: \"88e76793-e05b-4318-8526-aea31883cca0\") " pod="kserve-ci-e2e-test/isvc-primary-495374-predictor-cf965cf74-sdz6d"
May 25 11:46:57.097575 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:46:57.097550    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-495374-predictor-cf965cf74-sdz6d"
May 25 11:46:57.226433 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:46:57.226404    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-495374-predictor-cf965cf74-sdz6d"]
May 25 11:46:57.229198 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:46:57.229171    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88e76793_e05b_4318_8526_aea31883cca0.slice/crio-91c430789e3833b225d186b500ece3ea92c07a6708609f722726f2bbbeccbb4e WatchSource:0}: Error finding container 91c430789e3833b225d186b500ece3ea92c07a6708609f722726f2bbbeccbb4e: Status 404 returned error can't find the container with id 91c430789e3833b225d186b500ece3ea92c07a6708609f722726f2bbbeccbb4e
May 25 11:46:57.230899 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:46:57.230884    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 25 11:46:57.375884 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:46:57.375785    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-495374-predictor-cf965cf74-sdz6d" event={"ID":"88e76793-e05b-4318-8526-aea31883cca0","Type":"ContainerStarted","Data":"31420c9e2ca61086d0735708c36b90a3e52420f986b46c8adf7720371badef4c"}
May 25 11:46:57.375884 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:46:57.375828    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-495374-predictor-cf965cf74-sdz6d" event={"ID":"88e76793-e05b-4318-8526-aea31883cca0","Type":"ContainerStarted","Data":"91c430789e3833b225d186b500ece3ea92c07a6708609f722726f2bbbeccbb4e"}
May 25 11:47:01.389989 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:47:01.389951    2561 generic.go:358] "Generic (PLEG): container finished" podID="88e76793-e05b-4318-8526-aea31883cca0" containerID="31420c9e2ca61086d0735708c36b90a3e52420f986b46c8adf7720371badef4c" exitCode=0
May 25 11:47:01.390384 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:47:01.390029    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-495374-predictor-cf965cf74-sdz6d" event={"ID":"88e76793-e05b-4318-8526-aea31883cca0","Type":"ContainerDied","Data":"31420c9e2ca61086d0735708c36b90a3e52420f986b46c8adf7720371badef4c"}
May 25 11:47:02.394606 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:47:02.394574    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-495374-predictor-cf965cf74-sdz6d" event={"ID":"88e76793-e05b-4318-8526-aea31883cca0","Type":"ContainerStarted","Data":"28fcfe894d2b853726b2535d4612e16690724d2b474488a58064605a9a52673c"}
May 25 11:47:02.394606 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:47:02.394608    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-495374-predictor-cf965cf74-sdz6d" event={"ID":"88e76793-e05b-4318-8526-aea31883cca0","Type":"ContainerStarted","Data":"e97b558da2a6266a883639ddd931de06dbe55a07745afabb5a76e6ba44ccfb16"}
May 25 11:47:02.395109 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:47:02.394881    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-495374-predictor-cf965cf74-sdz6d"
May 25 11:47:02.395109 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:47:02.395003    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-495374-predictor-cf965cf74-sdz6d"
May 25 11:47:02.396378 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:47:02.396352    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-495374-predictor-cf965cf74-sdz6d" podUID="88e76793-e05b-4318-8526-aea31883cca0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
May 25 11:47:02.418213 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:47:02.418167    2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-primary-495374-predictor-cf965cf74-sdz6d" podStartSLOduration=6.418155103 podStartE2EDuration="6.418155103s" podCreationTimestamp="2026-05-25 11:46:56 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 11:47:02.416169131 +0000 UTC m=+1795.961346735" watchObservedRunningTime="2026-05-25 11:47:02.418155103 +0000 UTC m=+1795.963332707"
May 25 11:47:03.398611 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:47:03.398574    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-495374-predictor-cf965cf74-sdz6d" podUID="88e76793-e05b-4318-8526-aea31883cca0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
May 25 11:47:07.112082 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:47:07.112054    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-9lj95_10a66344-3304-49a1-bbc1-b885800f5acf/console-operator/1.log"
May 25 11:47:07.118599 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:47:07.118576    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-9lj95_10a66344-3304-49a1-bbc1-b885800f5acf/console-operator/1.log"
May 25 11:47:08.402175 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:47:08.402149    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-495374-predictor-cf965cf74-sdz6d"
May 25 11:47:08.402735 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:47:08.402708    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-495374-predictor-cf965cf74-sdz6d" podUID="88e76793-e05b-4318-8526-aea31883cca0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
May 25 11:47:18.403507 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:47:18.403466    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-495374-predictor-cf965cf74-sdz6d" podUID="88e76793-e05b-4318-8526-aea31883cca0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
May 25 11:47:28.402697 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:47:28.402661    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-495374-predictor-cf965cf74-sdz6d" podUID="88e76793-e05b-4318-8526-aea31883cca0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
May 25 11:47:38.403081 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:47:38.403039    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-495374-predictor-cf965cf74-sdz6d" podUID="88e76793-e05b-4318-8526-aea31883cca0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
May 25 11:47:48.403249 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:47:48.403211    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-495374-predictor-cf965cf74-sdz6d" podUID="88e76793-e05b-4318-8526-aea31883cca0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
May 25 11:47:58.402797 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:47:58.402761    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-495374-predictor-cf965cf74-sdz6d" podUID="88e76793-e05b-4318-8526-aea31883cca0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
May 25 11:48:08.403461 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:48:08.403423    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-495374-predictor-cf965cf74-sdz6d"
May 25 11:48:16.257351 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:48:16.257315    2561 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-495374-predictor-5f4494779c-d9k7n"]
May 25 11:48:16.261181 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:48:16.261161    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-495374-predictor-5f4494779c-d9k7n"
May 25 11:48:16.268903 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:48:16.268883    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-495374-predictor-serving-cert\""
May 25 11:48:16.269022 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:48:16.268996    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-secret-495374\""
May 25 11:48:16.269395 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:48:16.269378    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-495374-kube-rbac-proxy-sar-config\""
May 25 11:48:16.269895 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:48:16.269882    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
May 25 11:48:16.269937 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:48:16.269892    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-sa-495374-dockercfg-dpnvz\""
May 25 11:48:16.279514 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:48:16.279482    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-495374-predictor-5f4494779c-d9k7n"]
May 25 11:48:16.332362 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:48:16.332332    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0e27991b-a8a5-4da7-bde0-0887bd98781c-proxy-tls\") pod \"isvc-secondary-495374-predictor-5f4494779c-d9k7n\" (UID: \"0e27991b-a8a5-4da7-bde0-0887bd98781c\") " pod="kserve-ci-e2e-test/isvc-secondary-495374-predictor-5f4494779c-d9k7n"
May 25 11:48:16.332508 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:48:16.332382    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-secondary-495374-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0e27991b-a8a5-4da7-bde0-0887bd98781c-isvc-secondary-495374-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-495374-predictor-5f4494779c-d9k7n\" (UID: \"0e27991b-a8a5-4da7-bde0-0887bd98781c\") " pod="kserve-ci-e2e-test/isvc-secondary-495374-predictor-5f4494779c-d9k7n"
May 25 11:48:16.332508 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:48:16.332469    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9crpb\" (UniqueName: \"kubernetes.io/projected/0e27991b-a8a5-4da7-bde0-0887bd98781c-kube-api-access-9crpb\") pod \"isvc-secondary-495374-predictor-5f4494779c-d9k7n\" (UID: \"0e27991b-a8a5-4da7-bde0-0887bd98781c\") " pod="kserve-ci-e2e-test/isvc-secondary-495374-predictor-5f4494779c-d9k7n"
May 25 11:48:16.332508 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:48:16.332502    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0e27991b-a8a5-4da7-bde0-0887bd98781c-kserve-provision-location\") pod \"isvc-secondary-495374-predictor-5f4494779c-d9k7n\" (UID: \"0e27991b-a8a5-4da7-bde0-0887bd98781c\") " pod="kserve-ci-e2e-test/isvc-secondary-495374-predictor-5f4494779c-d9k7n"
May 25 11:48:16.332636 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:48:16.332549    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0e27991b-a8a5-4da7-bde0-0887bd98781c-cabundle-cert\") pod \"isvc-secondary-495374-predictor-5f4494779c-d9k7n\" (UID: \"0e27991b-a8a5-4da7-bde0-0887bd98781c\") " pod="kserve-ci-e2e-test/isvc-secondary-495374-predictor-5f4494779c-d9k7n"
May 25 11:48:16.433515 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:48:16.433471    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0e27991b-a8a5-4da7-bde0-0887bd98781c-proxy-tls\") pod \"isvc-secondary-495374-predictor-5f4494779c-d9k7n\" (UID: \"0e27991b-a8a5-4da7-bde0-0887bd98781c\") " pod="kserve-ci-e2e-test/isvc-secondary-495374-predictor-5f4494779c-d9k7n"
May 25 11:48:16.433712 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:48:16.433527    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-secondary-495374-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0e27991b-a8a5-4da7-bde0-0887bd98781c-isvc-secondary-495374-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-495374-predictor-5f4494779c-d9k7n\" (UID: \"0e27991b-a8a5-4da7-bde0-0887bd98781c\") " pod="kserve-ci-e2e-test/isvc-secondary-495374-predictor-5f4494779c-d9k7n"
May 25 11:48:16.433712 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:48:16.433584    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-9crpb\" (UniqueName: \"kubernetes.io/projected/0e27991b-a8a5-4da7-bde0-0887bd98781c-kube-api-access-9crpb\") pod \"isvc-secondary-495374-predictor-5f4494779c-d9k7n\" (UID: \"0e27991b-a8a5-4da7-bde0-0887bd98781c\") " pod="kserve-ci-e2e-test/isvc-secondary-495374-predictor-5f4494779c-d9k7n"
May 25 11:48:16.433712 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:48:16.433613    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0e27991b-a8a5-4da7-bde0-0887bd98781c-kserve-provision-location\") pod \"isvc-secondary-495374-predictor-5f4494779c-d9k7n\" (UID: \"0e27991b-a8a5-4da7-bde0-0887bd98781c\") " pod="kserve-ci-e2e-test/isvc-secondary-495374-predictor-5f4494779c-d9k7n"
May 25 11:48:16.433712 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:48:16.433640    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0e27991b-a8a5-4da7-bde0-0887bd98781c-cabundle-cert\") pod \"isvc-secondary-495374-predictor-5f4494779c-d9k7n\" (UID: \"0e27991b-a8a5-4da7-bde0-0887bd98781c\") " pod="kserve-ci-e2e-test/isvc-secondary-495374-predictor-5f4494779c-d9k7n"
May 25 11:48:16.434183 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:48:16.434155    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0e27991b-a8a5-4da7-bde0-0887bd98781c-kserve-provision-location\") pod \"isvc-secondary-495374-predictor-5f4494779c-d9k7n\" (UID: \"0e27991b-a8a5-4da7-bde0-0887bd98781c\") " pod="kserve-ci-e2e-test/isvc-secondary-495374-predictor-5f4494779c-d9k7n"
May 25 11:48:16.434353 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:48:16.434334    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0e27991b-a8a5-4da7-bde0-0887bd98781c-cabundle-cert\") pod \"isvc-secondary-495374-predictor-5f4494779c-d9k7n\" (UID: \"0e27991b-a8a5-4da7-bde0-0887bd98781c\") " pod="kserve-ci-e2e-test/isvc-secondary-495374-predictor-5f4494779c-d9k7n"
May 25 11:48:16.434353 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:48:16.434345    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-secondary-495374-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0e27991b-a8a5-4da7-bde0-0887bd98781c-isvc-secondary-495374-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-495374-predictor-5f4494779c-d9k7n\" (UID: \"0e27991b-a8a5-4da7-bde0-0887bd98781c\") " pod="kserve-ci-e2e-test/isvc-secondary-495374-predictor-5f4494779c-d9k7n"
May 25 11:48:16.435997 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:48:16.435981    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0e27991b-a8a5-4da7-bde0-0887bd98781c-proxy-tls\") pod \"isvc-secondary-495374-predictor-5f4494779c-d9k7n\" (UID: \"0e27991b-a8a5-4da7-bde0-0887bd98781c\") " pod="kserve-ci-e2e-test/isvc-secondary-495374-predictor-5f4494779c-d9k7n"
May 25 11:48:16.441682 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:48:16.441664    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9crpb\" (UniqueName: \"kubernetes.io/projected/0e27991b-a8a5-4da7-bde0-0887bd98781c-kube-api-access-9crpb\") pod \"isvc-secondary-495374-predictor-5f4494779c-d9k7n\" (UID: \"0e27991b-a8a5-4da7-bde0-0887bd98781c\") " pod="kserve-ci-e2e-test/isvc-secondary-495374-predictor-5f4494779c-d9k7n"
May 25 11:48:16.572428 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:48:16.572397    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-495374-predictor-5f4494779c-d9k7n"
May 25 11:48:16.706688 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:48:16.706661    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-495374-predictor-5f4494779c-d9k7n"]
May 25 11:48:16.709753 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:48:16.709720    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e27991b_a8a5_4da7_bde0_0887bd98781c.slice/crio-a898f5efc4dbd94959d808444c003eee2e156bfaa6185c381afb624f61ae37f0 WatchSource:0}: Error finding container a898f5efc4dbd94959d808444c003eee2e156bfaa6185c381afb624f61ae37f0: Status 404 returned error can't find the container with id a898f5efc4dbd94959d808444c003eee2e156bfaa6185c381afb624f61ae37f0
May 25 11:48:17.646604 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:48:17.646567    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-495374-predictor-5f4494779c-d9k7n" event={"ID":"0e27991b-a8a5-4da7-bde0-0887bd98781c","Type":"ContainerStarted","Data":"46b5c3966639fe38797c8b4680633cf0796a3a0dfcf526342592dac21e2e34c8"}
May 25 11:48:17.646604 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:48:17.646607    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-495374-predictor-5f4494779c-d9k7n" event={"ID":"0e27991b-a8a5-4da7-bde0-0887bd98781c","Type":"ContainerStarted","Data":"a898f5efc4dbd94959d808444c003eee2e156bfaa6185c381afb624f61ae37f0"}
May 25 11:48:21.661017 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:48:21.660984    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-495374-predictor-5f4494779c-d9k7n_0e27991b-a8a5-4da7-bde0-0887bd98781c/storage-initializer/0.log"
May 25 11:48:21.661433 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:48:21.661029    2561 generic.go:358] "Generic (PLEG): container finished" podID="0e27991b-a8a5-4da7-bde0-0887bd98781c" containerID="46b5c3966639fe38797c8b4680633cf0796a3a0dfcf526342592dac21e2e34c8" exitCode=1
May 25 11:48:21.661433 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:48:21.661104    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-495374-predictor-5f4494779c-d9k7n" event={"ID":"0e27991b-a8a5-4da7-bde0-0887bd98781c","Type":"ContainerDied","Data":"46b5c3966639fe38797c8b4680633cf0796a3a0dfcf526342592dac21e2e34c8"}
May 25 11:48:22.665754 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:48:22.665724    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-495374-predictor-5f4494779c-d9k7n_0e27991b-a8a5-4da7-bde0-0887bd98781c/storage-initializer/0.log"
May 25 11:48:22.666243 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:48:22.665853    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-495374-predictor-5f4494779c-d9k7n" event={"ID":"0e27991b-a8a5-4da7-bde0-0887bd98781c","Type":"ContainerStarted","Data":"1a76b164e7f73370be921d43b0e771f7853efb0273177f8f9a24a8a262092db0"}
May 25 11:48:25.677556 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:48:25.677483    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-495374-predictor-5f4494779c-d9k7n_0e27991b-a8a5-4da7-bde0-0887bd98781c/storage-initializer/1.log"
May 25 11:48:25.677896 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:48:25.677817    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-495374-predictor-5f4494779c-d9k7n_0e27991b-a8a5-4da7-bde0-0887bd98781c/storage-initializer/0.log"
May 25 11:48:25.677896 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:48:25.677849    2561 generic.go:358] "Generic (PLEG): container finished" podID="0e27991b-a8a5-4da7-bde0-0887bd98781c" containerID="1a76b164e7f73370be921d43b0e771f7853efb0273177f8f9a24a8a262092db0" exitCode=1
May 25 11:48:25.677966 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:48:25.677907    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-495374-predictor-5f4494779c-d9k7n" event={"ID":"0e27991b-a8a5-4da7-bde0-0887bd98781c","Type":"ContainerDied","Data":"1a76b164e7f73370be921d43b0e771f7853efb0273177f8f9a24a8a262092db0"}
May 25 11:48:25.677966 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:48:25.677939    2561 scope.go:117] "RemoveContainer" containerID="46b5c3966639fe38797c8b4680633cf0796a3a0dfcf526342592dac21e2e34c8"
May 25 11:48:25.678394 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:48:25.678373    2561 scope.go:117] "RemoveContainer" containerID="46b5c3966639fe38797c8b4680633cf0796a3a0dfcf526342592dac21e2e34c8"
May 25 11:48:25.688955 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:48:25.688926    2561 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-495374-predictor-5f4494779c-d9k7n_kserve-ci-e2e-test_0e27991b-a8a5-4da7-bde0-0887bd98781c_0 in pod sandbox a898f5efc4dbd94959d808444c003eee2e156bfaa6185c381afb624f61ae37f0 from index: no such id: '46b5c3966639fe38797c8b4680633cf0796a3a0dfcf526342592dac21e2e34c8'" containerID="46b5c3966639fe38797c8b4680633cf0796a3a0dfcf526342592dac21e2e34c8"
May 25 11:48:25.689020 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:48:25.688977    2561 kuberuntime_container.go:951] "Unhandled Error" err="failed to remove pod init container \"storage-initializer\": rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-495374-predictor-5f4494779c-d9k7n_kserve-ci-e2e-test_0e27991b-a8a5-4da7-bde0-0887bd98781c_0 in pod sandbox a898f5efc4dbd94959d808444c003eee2e156bfaa6185c381afb624f61ae37f0 from index: no such id: '46b5c3966639fe38797c8b4680633cf0796a3a0dfcf526342592dac21e2e34c8'; Skipping pod \"isvc-secondary-495374-predictor-5f4494779c-d9k7n_kserve-ci-e2e-test(0e27991b-a8a5-4da7-bde0-0887bd98781c)\"" logger="UnhandledError"
May 25 11:48:25.690349 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:48:25.690320    2561 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-secondary-495374-predictor-5f4494779c-d9k7n_kserve-ci-e2e-test(0e27991b-a8a5-4da7-bde0-0887bd98781c)\"" pod="kserve-ci-e2e-test/isvc-secondary-495374-predictor-5f4494779c-d9k7n" podUID="0e27991b-a8a5-4da7-bde0-0887bd98781c"
May 25 11:48:26.683086 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:48:26.683013    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-495374-predictor-5f4494779c-d9k7n_0e27991b-a8a5-4da7-bde0-0887bd98781c/storage-initializer/1.log"
May 25 11:48:32.313249 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:48:32.313217    2561 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-495374-predictor-5f4494779c-d9k7n"]
May 25 11:48:32.372556 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:48:32.372519    2561 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-495374-predictor-cf965cf74-sdz6d"]
May 25 11:48:32.372963 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:48:32.372932    2561 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-495374-predictor-cf965cf74-sdz6d" podUID="88e76793-e05b-4318-8526-aea31883cca0" containerName="kserve-container" containerID="cri-o://e97b558da2a6266a883639ddd931de06dbe55a07745afabb5a76e6ba44ccfb16" gracePeriod=30
May 25 11:48:32.373243 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:48:32.372971    2561 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-495374-predictor-cf965cf74-sdz6d" podUID="88e76793-e05b-4318-8526-aea31883cca0" containerName="kube-rbac-proxy" containerID="cri-o://28fcfe894d2b853726b2535d4612e16690724d2b474488a58064605a9a52673c" gracePeriod=30
May 25 11:48:32.452210 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:48:32.452174    2561 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-845d47-predictor-8f6b96d89-s6p5d"]
May 25 11:48:32.456076 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:48:32.456055    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-845d47-predictor-8f6b96d89-s6p5d"
May 25 11:48:32.458741 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:48:32.458717    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-sa-845d47-dockercfg-zqtcm\""
May 25 11:48:32.458965 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:48:32.458942    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-secret-845d47\""
May 25 11:48:32.459200 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:48:32.459183    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-845d47-kube-rbac-proxy-sar-config\""
May 25 11:48:32.459306 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:48:32.459191    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-845d47-predictor-serving-cert\""
May 25 11:48:32.467239 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:48:32.467217    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-845d47-predictor-8f6b96d89-s6p5d"]
May 25 11:48:32.497061 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:48:32.497042    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-495374-predictor-5f4494779c-d9k7n_0e27991b-a8a5-4da7-bde0-0887bd98781c/storage-initializer/1.log"
May 25 11:48:32.497192 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:48:32.497106    2561 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-495374-predictor-5f4494779c-d9k7n"
May 25 11:48:32.575569 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:48:32.575536    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9crpb\" (UniqueName: \"kubernetes.io/projected/0e27991b-a8a5-4da7-bde0-0887bd98781c-kube-api-access-9crpb\") pod \"0e27991b-a8a5-4da7-bde0-0887bd98781c\" (UID: \"0e27991b-a8a5-4da7-bde0-0887bd98781c\") "
May 25 11:48:32.575716 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:48:32.575628    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0e27991b-a8a5-4da7-bde0-0887bd98781c-cabundle-cert\") pod \"0e27991b-a8a5-4da7-bde0-0887bd98781c\" (UID: \"0e27991b-a8a5-4da7-bde0-0887bd98781c\") "
May 25 11:48:32.575716 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:48:32.575665    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0e27991b-a8a5-4da7-bde0-0887bd98781c-proxy-tls\") pod \"0e27991b-a8a5-4da7-bde0-0887bd98781c\" (UID: \"0e27991b-a8a5-4da7-bde0-0887bd98781c\") "
May 25 11:48:32.575716 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:48:32.575689    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0e27991b-a8a5-4da7-bde0-0887bd98781c-kserve-provision-location\") pod \"0e27991b-a8a5-4da7-bde0-0887bd98781c\" (UID: \"0e27991b-a8a5-4da7-bde0-0887bd98781c\") "
May 25 11:48:32.575875 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:48:32.575840    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-secondary-495374-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0e27991b-a8a5-4da7-bde0-0887bd98781c-isvc-secondary-495374-kube-rbac-proxy-sar-config\") pod \"0e27991b-a8a5-4da7-bde0-0887bd98781c\" (UID: \"0e27991b-a8a5-4da7-bde0-0887bd98781c\") "
May 25 11:48:32.576018 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:48:32.575991    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e27991b-a8a5-4da7-bde0-0887bd98781c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0e27991b-a8a5-4da7-bde0-0887bd98781c" (UID: "0e27991b-a8a5-4da7-bde0-0887bd98781c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:48:32.576104 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:48:32.576015    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e10b8dd2-1576-4d3c-ba19-48117e87578b-kserve-provision-location\") pod \"isvc-init-fail-845d47-predictor-8f6b96d89-s6p5d\" (UID: \"e10b8dd2-1576-4d3c-ba19-48117e87578b\") " pod="kserve-ci-e2e-test/isvc-init-fail-845d47-predictor-8f6b96d89-s6p5d"
May 25 11:48:32.576104 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:48:32.576060    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2bf8\" (UniqueName: \"kubernetes.io/projected/e10b8dd2-1576-4d3c-ba19-48117e87578b-kube-api-access-r2bf8\") pod \"isvc-init-fail-845d47-predictor-8f6b96d89-s6p5d\" (UID: \"e10b8dd2-1576-4d3c-ba19-48117e87578b\") " pod="kserve-ci-e2e-test/isvc-init-fail-845d47-predictor-8f6b96d89-s6p5d"
May 25 11:48:32.576104 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:48:32.576093    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e10b8dd2-1576-4d3c-ba19-48117e87578b-proxy-tls\") pod \"isvc-init-fail-845d47-predictor-8f6b96d89-s6p5d\" (UID: \"e10b8dd2-1576-4d3c-ba19-48117e87578b\") " pod="kserve-ci-e2e-test/isvc-init-fail-845d47-predictor-8f6b96d89-s6p5d"
May 25 11:48:32.576236 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:48:32.576018    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e27991b-a8a5-4da7-bde0-0887bd98781c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "0e27991b-a8a5-4da7-bde0-0887bd98781c" (UID: "0e27991b-a8a5-4da7-bde0-0887bd98781c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 11:48:32.576236 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:48:32.576160    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e27991b-a8a5-4da7-bde0-0887bd98781c-isvc-secondary-495374-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-secondary-495374-kube-rbac-proxy-sar-config") pod "0e27991b-a8a5-4da7-bde0-0887bd98781c" (UID: "0e27991b-a8a5-4da7-bde0-0887bd98781c"). InnerVolumeSpecName "isvc-secondary-495374-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 11:48:32.576345 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:48:32.576231    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e10b8dd2-1576-4d3c-ba19-48117e87578b-cabundle-cert\") pod \"isvc-init-fail-845d47-predictor-8f6b96d89-s6p5d\" (UID: \"e10b8dd2-1576-4d3c-ba19-48117e87578b\") " pod="kserve-ci-e2e-test/isvc-init-fail-845d47-predictor-8f6b96d89-s6p5d"
May 25 11:48:32.576345 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:48:32.576261    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-init-fail-845d47-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e10b8dd2-1576-4d3c-ba19-48117e87578b-isvc-init-fail-845d47-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-845d47-predictor-8f6b96d89-s6p5d\" (UID: \"e10b8dd2-1576-4d3c-ba19-48117e87578b\") " pod="kserve-ci-e2e-test/isvc-init-fail-845d47-predictor-8f6b96d89-s6p5d"
May 25 11:48:32.576423 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:48:32.576388    2561 reconciler_common.go:299] "Volume detached for volume \"isvc-secondary-495374-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0e27991b-a8a5-4da7-bde0-0887bd98781c-isvc-secondary-495374-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-227.ec2.internal\" DevicePath \"\""
May 25 11:48:32.576423 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:48:32.576413    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0e27991b-a8a5-4da7-bde0-0887bd98781c-cabundle-cert\") on node \"ip-10-0-136-227.ec2.internal\" DevicePath \"\""
May 25 11:48:32.576495 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:48:32.576430    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0e27991b-a8a5-4da7-bde0-0887bd98781c-kserve-provision-location\") on node \"ip-10-0-136-227.ec2.internal\" DevicePath \"\""
May 25 11:48:32.577730 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:48:32.577710    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e27991b-a8a5-4da7-bde0-0887bd98781c-kube-api-access-9crpb" (OuterVolumeSpecName: "kube-api-access-9crpb") pod "0e27991b-a8a5-4da7-bde0-0887bd98781c" (UID: "0e27991b-a8a5-4da7-bde0-0887bd98781c"). InnerVolumeSpecName "kube-api-access-9crpb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 11:48:32.577730 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:48:32.577715    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e27991b-a8a5-4da7-bde0-0887bd98781c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0e27991b-a8a5-4da7-bde0-0887bd98781c" (UID: "0e27991b-a8a5-4da7-bde0-0887bd98781c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 11:48:32.677248 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:48:32.677215    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e10b8dd2-1576-4d3c-ba19-48117e87578b-cabundle-cert\") pod \"isvc-init-fail-845d47-predictor-8f6b96d89-s6p5d\" (UID: \"e10b8dd2-1576-4d3c-ba19-48117e87578b\") " pod="kserve-ci-e2e-test/isvc-init-fail-845d47-predictor-8f6b96d89-s6p5d"
May 25 11:48:32.677470 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:48:32.677257    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-init-fail-845d47-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e10b8dd2-1576-4d3c-ba19-48117e87578b-isvc-init-fail-845d47-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-845d47-predictor-8f6b96d89-s6p5d\" (UID: \"e10b8dd2-1576-4d3c-ba19-48117e87578b\") " pod="kserve-ci-e2e-test/isvc-init-fail-845d47-predictor-8f6b96d89-s6p5d"
May 25 11:48:32.677470 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:48:32.677358    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e10b8dd2-1576-4d3c-ba19-48117e87578b-kserve-provision-location\") pod \"isvc-init-fail-845d47-predictor-8f6b96d89-s6p5d\" (UID: \"e10b8dd2-1576-4d3c-ba19-48117e87578b\") " pod="kserve-ci-e2e-test/isvc-init-fail-845d47-predictor-8f6b96d89-s6p5d"
May 25 11:48:32.677470 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:48:32.677384    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-r2bf8\" (UniqueName: \"kubernetes.io/projected/e10b8dd2-1576-4d3c-ba19-48117e87578b-kube-api-access-r2bf8\") pod \"isvc-init-fail-845d47-predictor-8f6b96d89-s6p5d\" (UID: \"e10b8dd2-1576-4d3c-ba19-48117e87578b\") " pod="kserve-ci-e2e-test/isvc-init-fail-845d47-predictor-8f6b96d89-s6p5d"
May 25 11:48:32.677470 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:48:32.677414    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e10b8dd2-1576-4d3c-ba19-48117e87578b-proxy-tls\") pod \"isvc-init-fail-845d47-predictor-8f6b96d89-s6p5d\" (UID: \"e10b8dd2-1576-4d3c-ba19-48117e87578b\") " pod="kserve-ci-e2e-test/isvc-init-fail-845d47-predictor-8f6b96d89-s6p5d"
May 25 11:48:32.677470 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:48:32.677458    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9crpb\" (UniqueName: \"kubernetes.io/projected/0e27991b-a8a5-4da7-bde0-0887bd98781c-kube-api-access-9crpb\") on node \"ip-10-0-136-227.ec2.internal\" DevicePath \"\""
May 25 11:48:32.677736 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:48:32.677475    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0e27991b-a8a5-4da7-bde0-0887bd98781c-proxy-tls\") on node \"ip-10-0-136-227.ec2.internal\" DevicePath \"\""
May 25 11:48:32.677830 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:48:32.677803    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e10b8dd2-1576-4d3c-ba19-48117e87578b-kserve-provision-location\") pod \"isvc-init-fail-845d47-predictor-8f6b96d89-s6p5d\" (UID: \"e10b8dd2-1576-4d3c-ba19-48117e87578b\") " pod="kserve-ci-e2e-test/isvc-init-fail-845d47-predictor-8f6b96d89-s6p5d"
May 25 11:48:32.677924 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:48:32.677907    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e10b8dd2-1576-4d3c-ba19-48117e87578b-cabundle-cert\") pod \"isvc-init-fail-845d47-predictor-8f6b96d89-s6p5d\" (UID: \"e10b8dd2-1576-4d3c-ba19-48117e87578b\") " pod="kserve-ci-e2e-test/isvc-init-fail-845d47-predictor-8f6b96d89-s6p5d"
May 25 11:48:32.677966 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:48:32.677947    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-init-fail-845d47-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e10b8dd2-1576-4d3c-ba19-48117e87578b-isvc-init-fail-845d47-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-845d47-predictor-8f6b96d89-s6p5d\" (UID: \"e10b8dd2-1576-4d3c-ba19-48117e87578b\") " pod="kserve-ci-e2e-test/isvc-init-fail-845d47-predictor-8f6b96d89-s6p5d"
May 25 11:48:32.679943 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:48:32.679922    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e10b8dd2-1576-4d3c-ba19-48117e87578b-proxy-tls\") pod \"isvc-init-fail-845d47-predictor-8f6b96d89-s6p5d\" (UID: \"e10b8dd2-1576-4d3c-ba19-48117e87578b\") " pod="kserve-ci-e2e-test/isvc-init-fail-845d47-predictor-8f6b96d89-s6p5d"
May 25 11:48:32.685844 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:48:32.685826    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2bf8\" (UniqueName: \"kubernetes.io/projected/e10b8dd2-1576-4d3c-ba19-48117e87578b-kube-api-access-r2bf8\") pod \"isvc-init-fail-845d47-predictor-8f6b96d89-s6p5d\" (UID: \"e10b8dd2-1576-4d3c-ba19-48117e87578b\") " pod="kserve-ci-e2e-test/isvc-init-fail-845d47-predictor-8f6b96d89-s6p5d"
May 25 11:48:32.704506 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:48:32.704478    2561 generic.go:358] "Generic (PLEG): container finished" podID="88e76793-e05b-4318-8526-aea31883cca0" containerID="28fcfe894d2b853726b2535d4612e16690724d2b474488a58064605a9a52673c" exitCode=2
May 25 11:48:32.704622 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:48:32.704551    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-495374-predictor-cf965cf74-sdz6d" event={"ID":"88e76793-e05b-4318-8526-aea31883cca0","Type":"ContainerDied","Data":"28fcfe894d2b853726b2535d4612e16690724d2b474488a58064605a9a52673c"}
May 25 11:48:32.705679 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:48:32.705660    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-495374-predictor-5f4494779c-d9k7n_0e27991b-a8a5-4da7-bde0-0887bd98781c/storage-initializer/1.log"
May 25 11:48:32.705776 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:48:32.705709    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-495374-predictor-5f4494779c-d9k7n" event={"ID":"0e27991b-a8a5-4da7-bde0-0887bd98781c","Type":"ContainerDied","Data":"a898f5efc4dbd94959d808444c003eee2e156bfaa6185c381afb624f61ae37f0"}
May 25 11:48:32.705776 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:48:32.705745    2561 scope.go:117] "RemoveContainer" containerID="1a76b164e7f73370be921d43b0e771f7853efb0273177f8f9a24a8a262092db0"
May 25 11:48:32.705865 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:48:32.705801    2561 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-495374-predictor-5f4494779c-d9k7n"
May 25 11:48:32.743673 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:48:32.743639    2561 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-495374-predictor-5f4494779c-d9k7n"]
May 25 11:48:32.747431 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:48:32.747405    2561 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-495374-predictor-5f4494779c-d9k7n"]
May 25 11:48:32.768679 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:48:32.768653    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-845d47-predictor-8f6b96d89-s6p5d"
May 25 11:48:32.896915 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:48:32.896890    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-845d47-predictor-8f6b96d89-s6p5d"]
May 25 11:48:32.900672 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:48:32.900636    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode10b8dd2_1576_4d3c_ba19_48117e87578b.slice/crio-ad0b2aa707af9ca4a71955d8dad86852449647faf62cbb1ddf325dd79420c52f WatchSource:0}: Error finding container ad0b2aa707af9ca4a71955d8dad86852449647faf62cbb1ddf325dd79420c52f: Status 404 returned error can't find the container with id ad0b2aa707af9ca4a71955d8dad86852449647faf62cbb1ddf325dd79420c52f
May 25 11:48:33.118373 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:48:33.118295    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0e27991b-a8a5-4da7-bde0-0887bd98781c" path="/var/lib/kubelet/pods/0e27991b-a8a5-4da7-bde0-0887bd98781c/volumes"
May 25 11:48:33.398992 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:48:33.398896    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-495374-predictor-cf965cf74-sdz6d" podUID="88e76793-e05b-4318-8526-aea31883cca0" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.44:8643/healthz\": dial tcp 10.132.0.44:8643: connect: connection refused"
May 25 11:48:33.712746 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:48:33.712658    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-845d47-predictor-8f6b96d89-s6p5d" event={"ID":"e10b8dd2-1576-4d3c-ba19-48117e87578b","Type":"ContainerStarted","Data":"6bbe68323e19bbe179b2dbf278e5a80e7b98def964f843d81d0b409b9867f0e6"}
May 25 11:48:33.712746 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:48:33.712705    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-845d47-predictor-8f6b96d89-s6p5d" event={"ID":"e10b8dd2-1576-4d3c-ba19-48117e87578b","Type":"ContainerStarted","Data":"ad0b2aa707af9ca4a71955d8dad86852449647faf62cbb1ddf325dd79420c52f"}
May 25 11:48:36.627705 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:48:36.627683    2561 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-495374-predictor-cf965cf74-sdz6d"
May 25 11:48:36.714827 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:48:36.714753    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t5kx4\" (UniqueName: \"kubernetes.io/projected/88e76793-e05b-4318-8526-aea31883cca0-kube-api-access-t5kx4\") pod \"88e76793-e05b-4318-8526-aea31883cca0\" (UID: \"88e76793-e05b-4318-8526-aea31883cca0\") "
May 25 11:48:36.714827 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:48:36.714811    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/88e76793-e05b-4318-8526-aea31883cca0-proxy-tls\") pod \"88e76793-e05b-4318-8526-aea31883cca0\" (UID: \"88e76793-e05b-4318-8526-aea31883cca0\") "
May 25 11:48:36.715048 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:48:36.714882    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-primary-495374-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/88e76793-e05b-4318-8526-aea31883cca0-isvc-primary-495374-kube-rbac-proxy-sar-config\") pod \"88e76793-e05b-4318-8526-aea31883cca0\" (UID: \"88e76793-e05b-4318-8526-aea31883cca0\") "
May 25 11:48:36.715048 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:48:36.714930    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/88e76793-e05b-4318-8526-aea31883cca0-kserve-provision-location\") pod \"88e76793-e05b-4318-8526-aea31883cca0\" (UID: \"88e76793-e05b-4318-8526-aea31883cca0\") "
May 25 11:48:36.715282 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:48:36.715240    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88e76793-e05b-4318-8526-aea31883cca0-isvc-primary-495374-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-primary-495374-kube-rbac-proxy-sar-config") pod "88e76793-e05b-4318-8526-aea31883cca0" (UID: "88e76793-e05b-4318-8526-aea31883cca0"). InnerVolumeSpecName "isvc-primary-495374-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 11:48:36.715365 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:48:36.715245    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88e76793-e05b-4318-8526-aea31883cca0-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "88e76793-e05b-4318-8526-aea31883cca0" (UID: "88e76793-e05b-4318-8526-aea31883cca0"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:48:36.716824 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:48:36.716803    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88e76793-e05b-4318-8526-aea31883cca0-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "88e76793-e05b-4318-8526-aea31883cca0" (UID: "88e76793-e05b-4318-8526-aea31883cca0"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 11:48:36.716875 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:48:36.716807    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88e76793-e05b-4318-8526-aea31883cca0-kube-api-access-t5kx4" (OuterVolumeSpecName: "kube-api-access-t5kx4") pod "88e76793-e05b-4318-8526-aea31883cca0" (UID: "88e76793-e05b-4318-8526-aea31883cca0"). InnerVolumeSpecName "kube-api-access-t5kx4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 11:48:36.726086 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:48:36.726057    2561 generic.go:358] "Generic (PLEG): container finished" podID="88e76793-e05b-4318-8526-aea31883cca0" containerID="e97b558da2a6266a883639ddd931de06dbe55a07745afabb5a76e6ba44ccfb16" exitCode=0
May 25 11:48:36.726203 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:48:36.726141    2561 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-495374-predictor-cf965cf74-sdz6d"
May 25 11:48:36.726203 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:48:36.726188    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-495374-predictor-cf965cf74-sdz6d" event={"ID":"88e76793-e05b-4318-8526-aea31883cca0","Type":"ContainerDied","Data":"e97b558da2a6266a883639ddd931de06dbe55a07745afabb5a76e6ba44ccfb16"}
May 25 11:48:36.726351 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:48:36.726241    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-495374-predictor-cf965cf74-sdz6d" event={"ID":"88e76793-e05b-4318-8526-aea31883cca0","Type":"ContainerDied","Data":"91c430789e3833b225d186b500ece3ea92c07a6708609f722726f2bbbeccbb4e"}
May 25 11:48:36.726351 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:48:36.726286    2561 scope.go:117] "RemoveContainer" containerID="28fcfe894d2b853726b2535d4612e16690724d2b474488a58064605a9a52673c"
May 25 11:48:36.735028 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:48:36.734991    2561 scope.go:117] "RemoveContainer" containerID="e97b558da2a6266a883639ddd931de06dbe55a07745afabb5a76e6ba44ccfb16"
May 25 11:48:36.742358 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:48:36.742342    2561 scope.go:117] "RemoveContainer" containerID="31420c9e2ca61086d0735708c36b90a3e52420f986b46c8adf7720371badef4c"
May 25 11:48:36.748879 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:48:36.748864    2561 scope.go:117] "RemoveContainer" containerID="28fcfe894d2b853726b2535d4612e16690724d2b474488a58064605a9a52673c"
May 25 11:48:36.749114 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:48:36.749095    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28fcfe894d2b853726b2535d4612e16690724d2b474488a58064605a9a52673c\": container with ID starting with 28fcfe894d2b853726b2535d4612e16690724d2b474488a58064605a9a52673c not found: ID does not exist" containerID="28fcfe894d2b853726b2535d4612e16690724d2b474488a58064605a9a52673c"
May 25 11:48:36.749168 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:48:36.749123    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28fcfe894d2b853726b2535d4612e16690724d2b474488a58064605a9a52673c"} err="failed to get container status \"28fcfe894d2b853726b2535d4612e16690724d2b474488a58064605a9a52673c\": rpc error: code = NotFound desc = could not find container \"28fcfe894d2b853726b2535d4612e16690724d2b474488a58064605a9a52673c\": container with ID starting with 28fcfe894d2b853726b2535d4612e16690724d2b474488a58064605a9a52673c not found: ID does not exist"
May 25 11:48:36.749168 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:48:36.749140    2561 scope.go:117] "RemoveContainer" containerID="e97b558da2a6266a883639ddd931de06dbe55a07745afabb5a76e6ba44ccfb16"
May 25 11:48:36.749364 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:48:36.749348    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e97b558da2a6266a883639ddd931de06dbe55a07745afabb5a76e6ba44ccfb16\": container with ID starting with e97b558da2a6266a883639ddd931de06dbe55a07745afabb5a76e6ba44ccfb16 not found: ID does not exist" containerID="e97b558da2a6266a883639ddd931de06dbe55a07745afabb5a76e6ba44ccfb16"
May 25 11:48:36.749416 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:48:36.749369    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e97b558da2a6266a883639ddd931de06dbe55a07745afabb5a76e6ba44ccfb16"} err="failed to get container status \"e97b558da2a6266a883639ddd931de06dbe55a07745afabb5a76e6ba44ccfb16\": rpc error: code = NotFound desc = could not find container \"e97b558da2a6266a883639ddd931de06dbe55a07745afabb5a76e6ba44ccfb16\": container with ID starting with e97b558da2a6266a883639ddd931de06dbe55a07745afabb5a76e6ba44ccfb16 not found: ID does not exist"
May 25 11:48:36.749416 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:48:36.749383    2561 scope.go:117] "RemoveContainer" containerID="31420c9e2ca61086d0735708c36b90a3e52420f986b46c8adf7720371badef4c"
May 25 11:48:36.749605 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:48:36.749587    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31420c9e2ca61086d0735708c36b90a3e52420f986b46c8adf7720371badef4c\": container with ID starting with 31420c9e2ca61086d0735708c36b90a3e52420f986b46c8adf7720371badef4c not found: ID does not exist" containerID="31420c9e2ca61086d0735708c36b90a3e52420f986b46c8adf7720371badef4c"
May 25 11:48:36.749660 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:48:36.749613    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31420c9e2ca61086d0735708c36b90a3e52420f986b46c8adf7720371badef4c"} err="failed to get container status \"31420c9e2ca61086d0735708c36b90a3e52420f986b46c8adf7720371badef4c\": rpc error: code = NotFound desc = could not find container \"31420c9e2ca61086d0735708c36b90a3e52420f986b46c8adf7720371badef4c\": container with ID starting with 31420c9e2ca61086d0735708c36b90a3e52420f986b46c8adf7720371badef4c not found: ID does not exist"
May 25 11:48:36.755608 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:48:36.755589    2561 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-495374-predictor-cf965cf74-sdz6d"]
May 25 11:48:36.766046 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:48:36.766025    2561 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-495374-predictor-cf965cf74-sdz6d"]
May 25 11:48:36.816157 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:48:36.816135    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/88e76793-e05b-4318-8526-aea31883cca0-proxy-tls\") on node \"ip-10-0-136-227.ec2.internal\" DevicePath \"\""
May 25 11:48:36.816157 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:48:36.816156    2561 reconciler_common.go:299] "Volume detached for volume \"isvc-primary-495374-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/88e76793-e05b-4318-8526-aea31883cca0-isvc-primary-495374-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-227.ec2.internal\" DevicePath \"\""
May 25 11:48:36.816309 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:48:36.816167    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/88e76793-e05b-4318-8526-aea31883cca0-kserve-provision-location\") on node \"ip-10-0-136-227.ec2.internal\" DevicePath \"\""
May 25 11:48:36.816309 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:48:36.816180    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t5kx4\" (UniqueName: \"kubernetes.io/projected/88e76793-e05b-4318-8526-aea31883cca0-kube-api-access-t5kx4\") on node \"ip-10-0-136-227.ec2.internal\" DevicePath \"\""
May 25 11:48:37.117834 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:48:37.117803    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="88e76793-e05b-4318-8526-aea31883cca0" path="/var/lib/kubelet/pods/88e76793-e05b-4318-8526-aea31883cca0/volumes"
May 25 11:48:38.734471 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:48:38.734398    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-845d47-predictor-8f6b96d89-s6p5d_e10b8dd2-1576-4d3c-ba19-48117e87578b/storage-initializer/0.log"
May 25 11:48:38.734471 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:48:38.734435    2561 generic.go:358] "Generic (PLEG): container finished" podID="e10b8dd2-1576-4d3c-ba19-48117e87578b" containerID="6bbe68323e19bbe179b2dbf278e5a80e7b98def964f843d81d0b409b9867f0e6" exitCode=1
May 25 11:48:38.734881 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:48:38.734481    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-845d47-predictor-8f6b96d89-s6p5d" event={"ID":"e10b8dd2-1576-4d3c-ba19-48117e87578b","Type":"ContainerDied","Data":"6bbe68323e19bbe179b2dbf278e5a80e7b98def964f843d81d0b409b9867f0e6"}
May 25 11:48:39.742091 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:48:39.742062    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-845d47-predictor-8f6b96d89-s6p5d_e10b8dd2-1576-4d3c-ba19-48117e87578b/storage-initializer/0.log"
May 25 11:48:39.742457 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:48:39.742141    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-845d47-predictor-8f6b96d89-s6p5d" event={"ID":"e10b8dd2-1576-4d3c-ba19-48117e87578b","Type":"ContainerStarted","Data":"7aa2e4a3918f2523166e15bb25acc777dd52c1f3d77e15a5879cde4443068b20"}
May 25 11:48:42.443163 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:48:42.443130    2561 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-845d47-predictor-8f6b96d89-s6p5d"]
May 25 11:48:42.443600 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:48:42.443478    2561 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-init-fail-845d47-predictor-8f6b96d89-s6p5d" podUID="e10b8dd2-1576-4d3c-ba19-48117e87578b" containerName="storage-initializer" containerID="cri-o://7aa2e4a3918f2523166e15bb25acc777dd52c1f3d77e15a5879cde4443068b20" gracePeriod=30
May 25 11:48:43.598429 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:48:43.598408    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-845d47-predictor-8f6b96d89-s6p5d_e10b8dd2-1576-4d3c-ba19-48117e87578b/storage-initializer/1.log"
May 25 11:48:43.598781 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:48:43.598767    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-845d47-predictor-8f6b96d89-s6p5d_e10b8dd2-1576-4d3c-ba19-48117e87578b/storage-initializer/0.log"
May 25 11:48:43.598845 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:48:43.598834    2561 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-845d47-predictor-8f6b96d89-s6p5d"
May 25 11:48:43.680685 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:48:43.680652    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r2bf8\" (UniqueName: \"kubernetes.io/projected/e10b8dd2-1576-4d3c-ba19-48117e87578b-kube-api-access-r2bf8\") pod \"e10b8dd2-1576-4d3c-ba19-48117e87578b\" (UID: \"e10b8dd2-1576-4d3c-ba19-48117e87578b\") "
May 25 11:48:43.680685 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:48:43.680685    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e10b8dd2-1576-4d3c-ba19-48117e87578b-cabundle-cert\") pod \"e10b8dd2-1576-4d3c-ba19-48117e87578b\" (UID: \"e10b8dd2-1576-4d3c-ba19-48117e87578b\") "
May 25 11:48:43.680885 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:48:43.680754    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e10b8dd2-1576-4d3c-ba19-48117e87578b-kserve-provision-location\") pod \"e10b8dd2-1576-4d3c-ba19-48117e87578b\" (UID: \"e10b8dd2-1576-4d3c-ba19-48117e87578b\") "
May 25 11:48:43.680885 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:48:43.680775    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e10b8dd2-1576-4d3c-ba19-48117e87578b-proxy-tls\") pod \"e10b8dd2-1576-4d3c-ba19-48117e87578b\" (UID: \"e10b8dd2-1576-4d3c-ba19-48117e87578b\") "
May 25 11:48:43.680885 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:48:43.680794    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-init-fail-845d47-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e10b8dd2-1576-4d3c-ba19-48117e87578b-isvc-init-fail-845d47-kube-rbac-proxy-sar-config\") pod \"e10b8dd2-1576-4d3c-ba19-48117e87578b\" (UID: \"e10b8dd2-1576-4d3c-ba19-48117e87578b\") "
May 25 11:48:43.681071 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:48:43.681044    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e10b8dd2-1576-4d3c-ba19-48117e87578b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e10b8dd2-1576-4d3c-ba19-48117e87578b" (UID: "e10b8dd2-1576-4d3c-ba19-48117e87578b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:48:43.681154 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:48:43.681137    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e10b8dd2-1576-4d3c-ba19-48117e87578b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e10b8dd2-1576-4d3c-ba19-48117e87578b" (UID: "e10b8dd2-1576-4d3c-ba19-48117e87578b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 11:48:43.681192 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:48:43.681171    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e10b8dd2-1576-4d3c-ba19-48117e87578b-isvc-init-fail-845d47-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-init-fail-845d47-kube-rbac-proxy-sar-config") pod "e10b8dd2-1576-4d3c-ba19-48117e87578b" (UID: "e10b8dd2-1576-4d3c-ba19-48117e87578b"). InnerVolumeSpecName "isvc-init-fail-845d47-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 11:48:43.682945 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:48:43.682912    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e10b8dd2-1576-4d3c-ba19-48117e87578b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e10b8dd2-1576-4d3c-ba19-48117e87578b" (UID: "e10b8dd2-1576-4d3c-ba19-48117e87578b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 11:48:43.682945 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:48:43.682940    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e10b8dd2-1576-4d3c-ba19-48117e87578b-kube-api-access-r2bf8" (OuterVolumeSpecName: "kube-api-access-r2bf8") pod "e10b8dd2-1576-4d3c-ba19-48117e87578b" (UID: "e10b8dd2-1576-4d3c-ba19-48117e87578b"). InnerVolumeSpecName "kube-api-access-r2bf8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 11:48:43.757293 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:48:43.757206    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-845d47-predictor-8f6b96d89-s6p5d_e10b8dd2-1576-4d3c-ba19-48117e87578b/storage-initializer/1.log"
May 25 11:48:43.757595 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:48:43.757579    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-845d47-predictor-8f6b96d89-s6p5d_e10b8dd2-1576-4d3c-ba19-48117e87578b/storage-initializer/0.log"
May 25 11:48:43.757655 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:48:43.757614    2561 generic.go:358] "Generic (PLEG): container finished" podID="e10b8dd2-1576-4d3c-ba19-48117e87578b" containerID="7aa2e4a3918f2523166e15bb25acc777dd52c1f3d77e15a5879cde4443068b20" exitCode=1
May 25 11:48:43.757655 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:48:43.757647    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-845d47-predictor-8f6b96d89-s6p5d" event={"ID":"e10b8dd2-1576-4d3c-ba19-48117e87578b","Type":"ContainerDied","Data":"7aa2e4a3918f2523166e15bb25acc777dd52c1f3d77e15a5879cde4443068b20"}
May 25 11:48:43.757727 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:48:43.757671    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-845d47-predictor-8f6b96d89-s6p5d" event={"ID":"e10b8dd2-1576-4d3c-ba19-48117e87578b","Type":"ContainerDied","Data":"ad0b2aa707af9ca4a71955d8dad86852449647faf62cbb1ddf325dd79420c52f"}
May 25 11:48:43.757727 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:48:43.757685    2561 scope.go:117] "RemoveContainer" containerID="7aa2e4a3918f2523166e15bb25acc777dd52c1f3d77e15a5879cde4443068b20"
May 25 11:48:43.757727 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:48:43.757717    2561 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-845d47-predictor-8f6b96d89-s6p5d"
May 25 11:48:43.765729 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:48:43.765711    2561 scope.go:117] "RemoveContainer" containerID="6bbe68323e19bbe179b2dbf278e5a80e7b98def964f843d81d0b409b9867f0e6"
May 25 11:48:43.772960 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:48:43.772941    2561 scope.go:117] "RemoveContainer" containerID="7aa2e4a3918f2523166e15bb25acc777dd52c1f3d77e15a5879cde4443068b20"
May 25 11:48:43.773189 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:48:43.773172    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7aa2e4a3918f2523166e15bb25acc777dd52c1f3d77e15a5879cde4443068b20\": container with ID starting with 7aa2e4a3918f2523166e15bb25acc777dd52c1f3d77e15a5879cde4443068b20 not found: ID does not exist" containerID="7aa2e4a3918f2523166e15bb25acc777dd52c1f3d77e15a5879cde4443068b20"
May 25 11:48:43.773244 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:48:43.773198    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7aa2e4a3918f2523166e15bb25acc777dd52c1f3d77e15a5879cde4443068b20"} err="failed to get container status \"7aa2e4a3918f2523166e15bb25acc777dd52c1f3d77e15a5879cde4443068b20\": rpc error: code = NotFound desc = could not find container \"7aa2e4a3918f2523166e15bb25acc777dd52c1f3d77e15a5879cde4443068b20\": container with ID starting with 7aa2e4a3918f2523166e15bb25acc777dd52c1f3d77e15a5879cde4443068b20 not found: ID does not exist"
May 25 11:48:43.773244 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:48:43.773218    2561 scope.go:117] "RemoveContainer" containerID="6bbe68323e19bbe179b2dbf278e5a80e7b98def964f843d81d0b409b9867f0e6"
May 25 11:48:43.773461 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:48:43.773443    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6bbe68323e19bbe179b2dbf278e5a80e7b98def964f843d81d0b409b9867f0e6\": container with ID starting with 6bbe68323e19bbe179b2dbf278e5a80e7b98def964f843d81d0b409b9867f0e6 not found: ID does not exist" containerID="6bbe68323e19bbe179b2dbf278e5a80e7b98def964f843d81d0b409b9867f0e6"
May 25 11:48:43.773510 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:48:43.773467    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6bbe68323e19bbe179b2dbf278e5a80e7b98def964f843d81d0b409b9867f0e6"} err="failed to get container status \"6bbe68323e19bbe179b2dbf278e5a80e7b98def964f843d81d0b409b9867f0e6\": rpc error: code = NotFound desc = could not find container \"6bbe68323e19bbe179b2dbf278e5a80e7b98def964f843d81d0b409b9867f0e6\": container with ID starting with 6bbe68323e19bbe179b2dbf278e5a80e7b98def964f843d81d0b409b9867f0e6 not found: ID does not exist"
May 25 11:48:43.782136 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:48:43.782120    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e10b8dd2-1576-4d3c-ba19-48117e87578b-kserve-provision-location\") on node \"ip-10-0-136-227.ec2.internal\" DevicePath \"\""
May 25 11:48:43.782200 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:48:43.782140    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e10b8dd2-1576-4d3c-ba19-48117e87578b-proxy-tls\") on node \"ip-10-0-136-227.ec2.internal\" DevicePath \"\""
May 25 11:48:43.782200 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:48:43.782152    2561 reconciler_common.go:299] "Volume detached for volume \"isvc-init-fail-845d47-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e10b8dd2-1576-4d3c-ba19-48117e87578b-isvc-init-fail-845d47-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-227.ec2.internal\" DevicePath \"\""
May 25 11:48:43.782200 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:48:43.782163    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r2bf8\" (UniqueName: \"kubernetes.io/projected/e10b8dd2-1576-4d3c-ba19-48117e87578b-kube-api-access-r2bf8\") on node \"ip-10-0-136-227.ec2.internal\" DevicePath \"\""
May 25 11:48:43.782200 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:48:43.782173    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e10b8dd2-1576-4d3c-ba19-48117e87578b-cabundle-cert\") on node \"ip-10-0-136-227.ec2.internal\" DevicePath \"\""
May 25 11:48:43.795463 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:48:43.795438    2561 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-845d47-predictor-8f6b96d89-s6p5d"]
May 25 11:48:43.801218 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:48:43.801192    2561 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-845d47-predictor-8f6b96d89-s6p5d"]
May 25 11:48:45.118426 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:48:45.118395    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e10b8dd2-1576-4d3c-ba19-48117e87578b" path="/var/lib/kubelet/pods/e10b8dd2-1576-4d3c-ba19-48117e87578b/volumes"
May 25 11:52:07.139362 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:52:07.139330    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-9lj95_10a66344-3304-49a1-bbc1-b885800f5acf/console-operator/1.log"
May 25 11:52:07.143319 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:52:07.143299    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-9lj95_10a66344-3304-49a1-bbc1-b885800f5acf/console-operator/1.log"
May 25 11:57:07.164116 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:57:07.164091    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-9lj95_10a66344-3304-49a1-bbc1-b885800f5acf/console-operator/1.log"
May 25 11:57:07.168969 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:57:07.168948    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-9lj95_10a66344-3304-49a1-bbc1-b885800f5acf/console-operator/1.log"
May 25 11:58:06.165770 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:58:06.165670    2561 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-74b58877c9-tfk9j"]
May 25 11:58:06.166462 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:58:06.166130    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e27991b-a8a5-4da7-bde0-0887bd98781c" containerName="storage-initializer"
May 25 11:58:06.166462 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:58:06.166152    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e27991b-a8a5-4da7-bde0-0887bd98781c" containerName="storage-initializer"
May 25 11:58:06.166462 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:58:06.166165    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88e76793-e05b-4318-8526-aea31883cca0" containerName="storage-initializer"
May 25 11:58:06.166462 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:58:06.166175    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="88e76793-e05b-4318-8526-aea31883cca0" containerName="storage-initializer"
May 25 11:58:06.166462 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:58:06.166185    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e10b8dd2-1576-4d3c-ba19-48117e87578b" containerName="storage-initializer"
May 25 11:58:06.166462 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:58:06.166195    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="e10b8dd2-1576-4d3c-ba19-48117e87578b" containerName="storage-initializer"
May 25 11:58:06.166462 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:58:06.166206    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88e76793-e05b-4318-8526-aea31883cca0" containerName="kube-rbac-proxy"
May 25 11:58:06.166462 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:58:06.166212    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="88e76793-e05b-4318-8526-aea31883cca0" containerName="kube-rbac-proxy"
May 25 11:58:06.166462 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:58:06.166224    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88e76793-e05b-4318-8526-aea31883cca0" containerName="kserve-container"
May 25 11:58:06.166462 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:58:06.166231    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="88e76793-e05b-4318-8526-aea31883cca0" containerName="kserve-container"
May 25 11:58:06.166462 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:58:06.166240    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e27991b-a8a5-4da7-bde0-0887bd98781c" containerName="storage-initializer"
May 25 11:58:06.166462 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:58:06.166256    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e27991b-a8a5-4da7-bde0-0887bd98781c" containerName="storage-initializer"
May 25 11:58:06.166462 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:58:06.166280    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e10b8dd2-1576-4d3c-ba19-48117e87578b" containerName="storage-initializer"
May 25 11:58:06.166462 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:58:06.166289    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="e10b8dd2-1576-4d3c-ba19-48117e87578b" containerName="storage-initializer"
May 25 11:58:06.166462 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:58:06.166381    2561 memory_manager.go:356] "RemoveStaleState removing state" podUID="0e27991b-a8a5-4da7-bde0-0887bd98781c" containerName="storage-initializer"
May 25 11:58:06.166462 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:58:06.166391    2561 memory_manager.go:356] "RemoveStaleState removing state" podUID="88e76793-e05b-4318-8526-aea31883cca0" containerName="kserve-container"
May 25 11:58:06.166462 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:58:06.166397    2561 memory_manager.go:356] "RemoveStaleState removing state" podUID="e10b8dd2-1576-4d3c-ba19-48117e87578b" containerName="storage-initializer"
May 25 11:58:06.166462 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:58:06.166405    2561 memory_manager.go:356] "RemoveStaleState removing state" podUID="0e27991b-a8a5-4da7-bde0-0887bd98781c" containerName="storage-initializer"
May 25 11:58:06.166462 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:58:06.166413    2561 memory_manager.go:356] "RemoveStaleState removing state" podUID="e10b8dd2-1576-4d3c-ba19-48117e87578b" containerName="storage-initializer"
May 25 11:58:06.166462 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:58:06.166420    2561 memory_manager.go:356] "RemoveStaleState removing state" podUID="88e76793-e05b-4318-8526-aea31883cca0" containerName="kube-rbac-proxy"
May 25 11:58:06.174358 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:58:06.174334    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-74b58877c9-tfk9j"
May 25 11:58:06.176764 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:58:06.176742    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
May 25 11:58:06.176890 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:58:06.176818    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
May 25 11:58:06.177692 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:58:06.177673    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-kube-rbac-proxy-sar-config\""
May 25 11:58:06.177692 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:58:06.177687    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-svs62\""
May 25 11:58:06.177860 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:58:06.177687    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-predictor-serving-cert\""
May 25 11:58:06.181999 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:58:06.181691    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-74b58877c9-tfk9j"]
May 25 11:58:06.318125 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:58:06.318097    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-sklearn-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8e59cad6-7470-473c-a688-9f15c147b661-isvc-sklearn-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-predictor-74b58877c9-tfk9j\" (UID: \"8e59cad6-7470-473c-a688-9f15c147b661\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-74b58877c9-tfk9j"
May 25 11:58:06.318259 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:58:06.318176    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ds9kf\" (UniqueName: \"kubernetes.io/projected/8e59cad6-7470-473c-a688-9f15c147b661-kube-api-access-ds9kf\") pod \"isvc-sklearn-predictor-74b58877c9-tfk9j\" (UID: \"8e59cad6-7470-473c-a688-9f15c147b661\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-74b58877c9-tfk9j"
May 25 11:58:06.318259 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:58:06.318196    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8e59cad6-7470-473c-a688-9f15c147b661-proxy-tls\") pod \"isvc-sklearn-predictor-74b58877c9-tfk9j\" (UID: \"8e59cad6-7470-473c-a688-9f15c147b661\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-74b58877c9-tfk9j"
May 25 11:58:06.318259 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:58:06.318218    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8e59cad6-7470-473c-a688-9f15c147b661-kserve-provision-location\") pod \"isvc-sklearn-predictor-74b58877c9-tfk9j\" (UID: \"8e59cad6-7470-473c-a688-9f15c147b661\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-74b58877c9-tfk9j"
May 25 11:58:06.419425 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:58:06.419368    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-sklearn-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8e59cad6-7470-473c-a688-9f15c147b661-isvc-sklearn-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-predictor-74b58877c9-tfk9j\" (UID: \"8e59cad6-7470-473c-a688-9f15c147b661\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-74b58877c9-tfk9j"
May 25 11:58:06.419523 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:58:06.419431    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-ds9kf\" (UniqueName: \"kubernetes.io/projected/8e59cad6-7470-473c-a688-9f15c147b661-kube-api-access-ds9kf\") pod \"isvc-sklearn-predictor-74b58877c9-tfk9j\" (UID: \"8e59cad6-7470-473c-a688-9f15c147b661\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-74b58877c9-tfk9j"
May 25 11:58:06.419523 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:58:06.419454    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8e59cad6-7470-473c-a688-9f15c147b661-proxy-tls\") pod \"isvc-sklearn-predictor-74b58877c9-tfk9j\" (UID: \"8e59cad6-7470-473c-a688-9f15c147b661\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-74b58877c9-tfk9j"
May 25 11:58:06.419523 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:58:06.419476    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8e59cad6-7470-473c-a688-9f15c147b661-kserve-provision-location\") pod \"isvc-sklearn-predictor-74b58877c9-tfk9j\" (UID: \"8e59cad6-7470-473c-a688-9f15c147b661\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-74b58877c9-tfk9j"
May 25 11:58:06.419814 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:58:06.419797    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8e59cad6-7470-473c-a688-9f15c147b661-kserve-provision-location\") pod \"isvc-sklearn-predictor-74b58877c9-tfk9j\" (UID: \"8e59cad6-7470-473c-a688-9f15c147b661\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-74b58877c9-tfk9j"
May 25 11:58:06.419982 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:58:06.419964    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-sklearn-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8e59cad6-7470-473c-a688-9f15c147b661-isvc-sklearn-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-predictor-74b58877c9-tfk9j\" (UID: \"8e59cad6-7470-473c-a688-9f15c147b661\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-74b58877c9-tfk9j"
May 25 11:58:06.421817 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:58:06.421795    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8e59cad6-7470-473c-a688-9f15c147b661-proxy-tls\") pod \"isvc-sklearn-predictor-74b58877c9-tfk9j\" (UID: \"8e59cad6-7470-473c-a688-9f15c147b661\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-74b58877c9-tfk9j"
May 25 11:58:06.427598 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:58:06.427576    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ds9kf\" (UniqueName: \"kubernetes.io/projected/8e59cad6-7470-473c-a688-9f15c147b661-kube-api-access-ds9kf\") pod \"isvc-sklearn-predictor-74b58877c9-tfk9j\" (UID: \"8e59cad6-7470-473c-a688-9f15c147b661\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-74b58877c9-tfk9j"
May 25 11:58:06.485741 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:58:06.485717    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-74b58877c9-tfk9j"
May 25 11:58:06.613680 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:58:06.613643    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-74b58877c9-tfk9j"]
May 25 11:58:06.615907 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:58:06.615880    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e59cad6_7470_473c_a688_9f15c147b661.slice/crio-1551f9fcafb4b80c4af805284fab6083a85f29707a19aebb73c69407018e7cd4 WatchSource:0}: Error finding container 1551f9fcafb4b80c4af805284fab6083a85f29707a19aebb73c69407018e7cd4: Status 404 returned error can't find the container with id 1551f9fcafb4b80c4af805284fab6083a85f29707a19aebb73c69407018e7cd4
May 25 11:58:06.617770 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:58:06.617755    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 25 11:58:07.612766 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:58:07.612730    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-74b58877c9-tfk9j" event={"ID":"8e59cad6-7470-473c-a688-9f15c147b661","Type":"ContainerStarted","Data":"2b0c06d9226f6087e9468076b60c3f9fa730b56b543de9997602ddf179eacfde"}
May 25 11:58:07.612766 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:58:07.612769    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-74b58877c9-tfk9j" event={"ID":"8e59cad6-7470-473c-a688-9f15c147b661","Type":"ContainerStarted","Data":"1551f9fcafb4b80c4af805284fab6083a85f29707a19aebb73c69407018e7cd4"}
May 25 11:58:10.624668 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:58:10.624636    2561 generic.go:358] "Generic (PLEG): container finished" podID="8e59cad6-7470-473c-a688-9f15c147b661" containerID="2b0c06d9226f6087e9468076b60c3f9fa730b56b543de9997602ddf179eacfde" exitCode=0
May 25 11:58:10.625140 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:58:10.624697    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-74b58877c9-tfk9j" event={"ID":"8e59cad6-7470-473c-a688-9f15c147b661","Type":"ContainerDied","Data":"2b0c06d9226f6087e9468076b60c3f9fa730b56b543de9997602ddf179eacfde"}
May 25 11:58:11.629531 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:58:11.629499    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-74b58877c9-tfk9j" event={"ID":"8e59cad6-7470-473c-a688-9f15c147b661","Type":"ContainerStarted","Data":"da1a02e6d0e0d51977bd43192c31629ad96e6a364ed0b684538886c4fe32d168"}
May 25 11:58:11.629910 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:58:11.629542    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-74b58877c9-tfk9j" event={"ID":"8e59cad6-7470-473c-a688-9f15c147b661","Type":"ContainerStarted","Data":"aee981304ea7a68f502a09d45c419103a7b57cfbf3a8ff3ea6dee18096a59d8e"}
May 25 11:58:11.629910 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:58:11.629762    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-74b58877c9-tfk9j"
May 25 11:58:11.650450 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:58:11.650395    2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-74b58877c9-tfk9j" podStartSLOduration=5.65038368 podStartE2EDuration="5.65038368s" podCreationTimestamp="2026-05-25 11:58:06 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 11:58:11.64736473 +0000 UTC m=+2465.192542335" watchObservedRunningTime="2026-05-25 11:58:11.65038368 +0000 UTC m=+2465.195561282"
May 25 11:58:12.633115 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:58:12.633077    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-74b58877c9-tfk9j"
May 25 11:58:12.634151 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:58:12.634128    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-74b58877c9-tfk9j" podUID="8e59cad6-7470-473c-a688-9f15c147b661" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
May 25 11:58:13.636380 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:58:13.636338    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-74b58877c9-tfk9j" podUID="8e59cad6-7470-473c-a688-9f15c147b661" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
May 25 11:58:18.641448 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:58:18.641417    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-74b58877c9-tfk9j"
May 25 11:58:18.642033 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:58:18.642005    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-74b58877c9-tfk9j" podUID="8e59cad6-7470-473c-a688-9f15c147b661" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
May 25 11:58:28.642578 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:58:28.642541    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-74b58877c9-tfk9j" podUID="8e59cad6-7470-473c-a688-9f15c147b661" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
May 25 11:58:38.642005 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:58:38.641962    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-74b58877c9-tfk9j" podUID="8e59cad6-7470-473c-a688-9f15c147b661" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
May 25 11:58:48.642550 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:58:48.642512    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-74b58877c9-tfk9j" podUID="8e59cad6-7470-473c-a688-9f15c147b661" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
May 25 11:58:58.642149 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:58:58.642069    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-74b58877c9-tfk9j" podUID="8e59cad6-7470-473c-a688-9f15c147b661" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
May 25 11:59:08.642298 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:59:08.642232    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-74b58877c9-tfk9j" podUID="8e59cad6-7470-473c-a688-9f15c147b661" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
May 25 11:59:18.643421 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:59:18.643388    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-74b58877c9-tfk9j"
May 25 11:59:26.147408 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:59:26.147373    2561 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-74b58877c9-tfk9j"]
May 25 11:59:26.147797 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:59:26.147770    2561 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-74b58877c9-tfk9j" podUID="8e59cad6-7470-473c-a688-9f15c147b661" containerName="kserve-container" containerID="cri-o://aee981304ea7a68f502a09d45c419103a7b57cfbf3a8ff3ea6dee18096a59d8e" gracePeriod=30
May 25 11:59:26.147937 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:59:26.147885    2561 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-74b58877c9-tfk9j" podUID="8e59cad6-7470-473c-a688-9f15c147b661" containerName="kube-rbac-proxy" containerID="cri-o://da1a02e6d0e0d51977bd43192c31629ad96e6a364ed0b684538886c4fe32d168" gracePeriod=30
May 25 11:59:26.237716 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:59:26.237686    2561 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-qq58x"]
May 25 11:59:26.241517 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:59:26.241497    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-qq58x"
May 25 11:59:26.243845 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:59:26.243827    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sklearn-v2-mlserver-predictor-serving-cert\""
May 25 11:59:26.243937 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:59:26.243861    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sklearn-v2-mlserver-kube-rbac-proxy-sar-config\""
May 25 11:59:26.249981 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:59:26.249958    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-qq58x"]
May 25 11:59:26.319146 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:59:26.319117    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/453630ac-6ff6-40a5-9f71-9c778337c1d0-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-65d8664766-qq58x\" (UID: \"453630ac-6ff6-40a5-9f71-9c778337c1d0\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-qq58x"
May 25 11:59:26.319291 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:59:26.319177    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7ss8\" (UniqueName: \"kubernetes.io/projected/453630ac-6ff6-40a5-9f71-9c778337c1d0-kube-api-access-v7ss8\") pod \"sklearn-v2-mlserver-predictor-65d8664766-qq58x\" (UID: \"453630ac-6ff6-40a5-9f71-9c778337c1d0\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-qq58x"
May 25 11:59:26.319291 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:59:26.319231    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sklearn-v2-mlserver-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/453630ac-6ff6-40a5-9f71-9c778337c1d0-sklearn-v2-mlserver-kube-rbac-proxy-sar-config\") pod \"sklearn-v2-mlserver-predictor-65d8664766-qq58x\" (UID: \"453630ac-6ff6-40a5-9f71-9c778337c1d0\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-qq58x"
May 25 11:59:26.319291 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:59:26.319256    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/453630ac-6ff6-40a5-9f71-9c778337c1d0-kserve-provision-location\") pod \"sklearn-v2-mlserver-predictor-65d8664766-qq58x\" (UID: \"453630ac-6ff6-40a5-9f71-9c778337c1d0\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-qq58x"
May 25 11:59:26.420351 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:59:26.420255    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sklearn-v2-mlserver-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/453630ac-6ff6-40a5-9f71-9c778337c1d0-sklearn-v2-mlserver-kube-rbac-proxy-sar-config\") pod \"sklearn-v2-mlserver-predictor-65d8664766-qq58x\" (UID: \"453630ac-6ff6-40a5-9f71-9c778337c1d0\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-qq58x"
May 25 11:59:26.420351 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:59:26.420310    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/453630ac-6ff6-40a5-9f71-9c778337c1d0-kserve-provision-location\") pod \"sklearn-v2-mlserver-predictor-65d8664766-qq58x\" (UID: \"453630ac-6ff6-40a5-9f71-9c778337c1d0\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-qq58x"
May 25 11:59:26.420351 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:59:26.420345    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/453630ac-6ff6-40a5-9f71-9c778337c1d0-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-65d8664766-qq58x\" (UID: \"453630ac-6ff6-40a5-9f71-9c778337c1d0\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-qq58x"
May 25 11:59:26.420584 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:59:26.420402    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-v7ss8\" (UniqueName: \"kubernetes.io/projected/453630ac-6ff6-40a5-9f71-9c778337c1d0-kube-api-access-v7ss8\") pod \"sklearn-v2-mlserver-predictor-65d8664766-qq58x\" (UID: \"453630ac-6ff6-40a5-9f71-9c778337c1d0\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-qq58x"
May 25 11:59:26.420584 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:59:26.420519    2561 secret.go:189] Couldn't get secret kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-serving-cert: secret "sklearn-v2-mlserver-predictor-serving-cert" not found
May 25 11:59:26.420671 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:59:26.420605    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/453630ac-6ff6-40a5-9f71-9c778337c1d0-proxy-tls podName:453630ac-6ff6-40a5-9f71-9c778337c1d0 nodeName:}" failed. No retries permitted until 2026-05-25 11:59:26.920590178 +0000 UTC m=+2540.465767767 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/453630ac-6ff6-40a5-9f71-9c778337c1d0-proxy-tls") pod "sklearn-v2-mlserver-predictor-65d8664766-qq58x" (UID: "453630ac-6ff6-40a5-9f71-9c778337c1d0") : secret "sklearn-v2-mlserver-predictor-serving-cert" not found
May 25 11:59:26.420746 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:59:26.420725    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/453630ac-6ff6-40a5-9f71-9c778337c1d0-kserve-provision-location\") pod \"sklearn-v2-mlserver-predictor-65d8664766-qq58x\" (UID: \"453630ac-6ff6-40a5-9f71-9c778337c1d0\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-qq58x"
May 25 11:59:26.420957 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:59:26.420940    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sklearn-v2-mlserver-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/453630ac-6ff6-40a5-9f71-9c778337c1d0-sklearn-v2-mlserver-kube-rbac-proxy-sar-config\") pod \"sklearn-v2-mlserver-predictor-65d8664766-qq58x\" (UID: \"453630ac-6ff6-40a5-9f71-9c778337c1d0\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-qq58x"
May 25 11:59:26.428778 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:59:26.428750    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7ss8\" (UniqueName: \"kubernetes.io/projected/453630ac-6ff6-40a5-9f71-9c778337c1d0-kube-api-access-v7ss8\") pod \"sklearn-v2-mlserver-predictor-65d8664766-qq58x\" (UID: \"453630ac-6ff6-40a5-9f71-9c778337c1d0\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-qq58x"
May 25 11:59:26.879184 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:59:26.879149    2561 generic.go:358] "Generic (PLEG): container finished" podID="8e59cad6-7470-473c-a688-9f15c147b661" containerID="da1a02e6d0e0d51977bd43192c31629ad96e6a364ed0b684538886c4fe32d168" exitCode=2
May 25 11:59:26.879373 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:59:26.879223    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-74b58877c9-tfk9j" event={"ID":"8e59cad6-7470-473c-a688-9f15c147b661","Type":"ContainerDied","Data":"da1a02e6d0e0d51977bd43192c31629ad96e6a364ed0b684538886c4fe32d168"}
May 25 11:59:26.924848 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:59:26.924823    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/453630ac-6ff6-40a5-9f71-9c778337c1d0-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-65d8664766-qq58x\" (UID: \"453630ac-6ff6-40a5-9f71-9c778337c1d0\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-qq58x"
May 25 11:59:26.927334 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:59:26.927312    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/453630ac-6ff6-40a5-9f71-9c778337c1d0-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-65d8664766-qq58x\" (UID: \"453630ac-6ff6-40a5-9f71-9c778337c1d0\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-qq58x"
May 25 11:59:27.154041 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:59:27.153964    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-qq58x"
May 25 11:59:27.282429 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:59:27.282342    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-qq58x"]
May 25 11:59:27.286544 ip-10-0-136-227 kubenswrapper[2561]: W0525 11:59:27.286500    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod453630ac_6ff6_40a5_9f71_9c778337c1d0.slice/crio-862c30f1d5c6396399f932dedf0301d57c0bb1eae8c5538d1d7fda01dfd113ae WatchSource:0}: Error finding container 862c30f1d5c6396399f932dedf0301d57c0bb1eae8c5538d1d7fda01dfd113ae: Status 404 returned error can't find the container with id 862c30f1d5c6396399f932dedf0301d57c0bb1eae8c5538d1d7fda01dfd113ae
May 25 11:59:27.884722 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:59:27.884684    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-qq58x" event={"ID":"453630ac-6ff6-40a5-9f71-9c778337c1d0","Type":"ContainerStarted","Data":"f5a00fd120056d50dcc5d5155ff305b49b02320120c0501976a819c42dcd3afd"}
May 25 11:59:27.884722 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:59:27.884725    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-qq58x" event={"ID":"453630ac-6ff6-40a5-9f71-9c778337c1d0","Type":"ContainerStarted","Data":"862c30f1d5c6396399f932dedf0301d57c0bb1eae8c5538d1d7fda01dfd113ae"}
May 25 11:59:28.637260 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:59:28.637213    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-74b58877c9-tfk9j" podUID="8e59cad6-7470-473c-a688-9f15c147b661" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.47:8643/healthz\": dial tcp 10.132.0.47:8643: connect: connection refused"
May 25 11:59:28.642639 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:59:28.642613    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-74b58877c9-tfk9j" podUID="8e59cad6-7470-473c-a688-9f15c147b661" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
May 25 11:59:30.198206 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:59:30.198181    2561 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-74b58877c9-tfk9j"
May 25 11:59:30.253648 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:59:30.253578    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ds9kf\" (UniqueName: \"kubernetes.io/projected/8e59cad6-7470-473c-a688-9f15c147b661-kube-api-access-ds9kf\") pod \"8e59cad6-7470-473c-a688-9f15c147b661\" (UID: \"8e59cad6-7470-473c-a688-9f15c147b661\") "
May 25 11:59:30.253790 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:59:30.253646    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8e59cad6-7470-473c-a688-9f15c147b661-kserve-provision-location\") pod \"8e59cad6-7470-473c-a688-9f15c147b661\" (UID: \"8e59cad6-7470-473c-a688-9f15c147b661\") "
May 25 11:59:30.253790 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:59:30.253707    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-sklearn-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8e59cad6-7470-473c-a688-9f15c147b661-isvc-sklearn-kube-rbac-proxy-sar-config\") pod \"8e59cad6-7470-473c-a688-9f15c147b661\" (UID: \"8e59cad6-7470-473c-a688-9f15c147b661\") "
May 25 11:59:30.253896 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:59:30.253787    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8e59cad6-7470-473c-a688-9f15c147b661-proxy-tls\") pod \"8e59cad6-7470-473c-a688-9f15c147b661\" (UID: \"8e59cad6-7470-473c-a688-9f15c147b661\") "
May 25 11:59:30.254005 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:59:30.253982    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e59cad6-7470-473c-a688-9f15c147b661-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8e59cad6-7470-473c-a688-9f15c147b661" (UID: "8e59cad6-7470-473c-a688-9f15c147b661"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:59:30.254065 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:59:30.254057    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8e59cad6-7470-473c-a688-9f15c147b661-kserve-provision-location\") on node \"ip-10-0-136-227.ec2.internal\" DevicePath \"\""
May 25 11:59:30.254167 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:59:30.254099    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e59cad6-7470-473c-a688-9f15c147b661-isvc-sklearn-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-sklearn-kube-rbac-proxy-sar-config") pod "8e59cad6-7470-473c-a688-9f15c147b661" (UID: "8e59cad6-7470-473c-a688-9f15c147b661"). InnerVolumeSpecName "isvc-sklearn-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 11:59:30.255559 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:59:30.255539    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e59cad6-7470-473c-a688-9f15c147b661-kube-api-access-ds9kf" (OuterVolumeSpecName: "kube-api-access-ds9kf") pod "8e59cad6-7470-473c-a688-9f15c147b661" (UID: "8e59cad6-7470-473c-a688-9f15c147b661"). InnerVolumeSpecName "kube-api-access-ds9kf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 11:59:30.255717 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:59:30.255694    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e59cad6-7470-473c-a688-9f15c147b661-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8e59cad6-7470-473c-a688-9f15c147b661" (UID: "8e59cad6-7470-473c-a688-9f15c147b661"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 11:59:30.354603 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:59:30.354584    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8e59cad6-7470-473c-a688-9f15c147b661-proxy-tls\") on node \"ip-10-0-136-227.ec2.internal\" DevicePath \"\""
May 25 11:59:30.354603 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:59:30.354604    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ds9kf\" (UniqueName: \"kubernetes.io/projected/8e59cad6-7470-473c-a688-9f15c147b661-kube-api-access-ds9kf\") on node \"ip-10-0-136-227.ec2.internal\" DevicePath \"\""
May 25 11:59:30.354743 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:59:30.354615    2561 reconciler_common.go:299] "Volume detached for volume \"isvc-sklearn-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8e59cad6-7470-473c-a688-9f15c147b661-isvc-sklearn-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-227.ec2.internal\" DevicePath \"\""
May 25 11:59:30.897241 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:59:30.897213    2561 generic.go:358] "Generic (PLEG): container finished" podID="8e59cad6-7470-473c-a688-9f15c147b661" containerID="aee981304ea7a68f502a09d45c419103a7b57cfbf3a8ff3ea6dee18096a59d8e" exitCode=0
May 25 11:59:30.897427 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:59:30.897302    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-74b58877c9-tfk9j" event={"ID":"8e59cad6-7470-473c-a688-9f15c147b661","Type":"ContainerDied","Data":"aee981304ea7a68f502a09d45c419103a7b57cfbf3a8ff3ea6dee18096a59d8e"}
May 25 11:59:30.897427 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:59:30.897318    2561 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-74b58877c9-tfk9j"
May 25 11:59:30.897427 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:59:30.897343    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-74b58877c9-tfk9j" event={"ID":"8e59cad6-7470-473c-a688-9f15c147b661","Type":"ContainerDied","Data":"1551f9fcafb4b80c4af805284fab6083a85f29707a19aebb73c69407018e7cd4"}
May 25 11:59:30.897427 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:59:30.897359    2561 scope.go:117] "RemoveContainer" containerID="da1a02e6d0e0d51977bd43192c31629ad96e6a364ed0b684538886c4fe32d168"
May 25 11:59:30.906044 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:59:30.906027    2561 scope.go:117] "RemoveContainer" containerID="aee981304ea7a68f502a09d45c419103a7b57cfbf3a8ff3ea6dee18096a59d8e"
May 25 11:59:30.913146 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:59:30.913131    2561 scope.go:117] "RemoveContainer" containerID="2b0c06d9226f6087e9468076b60c3f9fa730b56b543de9997602ddf179eacfde"
May 25 11:59:30.919772 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:59:30.919755    2561 scope.go:117] "RemoveContainer" containerID="da1a02e6d0e0d51977bd43192c31629ad96e6a364ed0b684538886c4fe32d168"
May 25 11:59:30.919995 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:59:30.919977    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da1a02e6d0e0d51977bd43192c31629ad96e6a364ed0b684538886c4fe32d168\": container with ID starting with da1a02e6d0e0d51977bd43192c31629ad96e6a364ed0b684538886c4fe32d168 not found: ID does not exist" containerID="da1a02e6d0e0d51977bd43192c31629ad96e6a364ed0b684538886c4fe32d168"
May 25 11:59:30.920077 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:59:30.920001    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da1a02e6d0e0d51977bd43192c31629ad96e6a364ed0b684538886c4fe32d168"} err="failed to get container status \"da1a02e6d0e0d51977bd43192c31629ad96e6a364ed0b684538886c4fe32d168\": rpc error: code = NotFound desc = could not find container \"da1a02e6d0e0d51977bd43192c31629ad96e6a364ed0b684538886c4fe32d168\": container with ID starting with da1a02e6d0e0d51977bd43192c31629ad96e6a364ed0b684538886c4fe32d168 not found: ID does not exist"
May 25 11:59:30.920077 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:59:30.920016    2561 scope.go:117] "RemoveContainer" containerID="aee981304ea7a68f502a09d45c419103a7b57cfbf3a8ff3ea6dee18096a59d8e"
May 25 11:59:30.920195 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:59:30.920174    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aee981304ea7a68f502a09d45c419103a7b57cfbf3a8ff3ea6dee18096a59d8e\": container with ID starting with aee981304ea7a68f502a09d45c419103a7b57cfbf3a8ff3ea6dee18096a59d8e not found: ID does not exist" containerID="aee981304ea7a68f502a09d45c419103a7b57cfbf3a8ff3ea6dee18096a59d8e"
May 25 11:59:30.920252 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:59:30.920191    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aee981304ea7a68f502a09d45c419103a7b57cfbf3a8ff3ea6dee18096a59d8e"} err="failed to get container status \"aee981304ea7a68f502a09d45c419103a7b57cfbf3a8ff3ea6dee18096a59d8e\": rpc error: code = NotFound desc = could not find container \"aee981304ea7a68f502a09d45c419103a7b57cfbf3a8ff3ea6dee18096a59d8e\": container with ID starting with aee981304ea7a68f502a09d45c419103a7b57cfbf3a8ff3ea6dee18096a59d8e not found: ID does not exist"
May 25 11:59:30.920252 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:59:30.920204    2561 scope.go:117] "RemoveContainer" containerID="2b0c06d9226f6087e9468076b60c3f9fa730b56b543de9997602ddf179eacfde"
May 25 11:59:30.920427 ip-10-0-136-227 kubenswrapper[2561]: E0525 11:59:30.920410    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b0c06d9226f6087e9468076b60c3f9fa730b56b543de9997602ddf179eacfde\": container with ID starting with 2b0c06d9226f6087e9468076b60c3f9fa730b56b543de9997602ddf179eacfde not found: ID does not exist" containerID="2b0c06d9226f6087e9468076b60c3f9fa730b56b543de9997602ddf179eacfde"
May 25 11:59:30.920483 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:59:30.920430    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b0c06d9226f6087e9468076b60c3f9fa730b56b543de9997602ddf179eacfde"} err="failed to get container status \"2b0c06d9226f6087e9468076b60c3f9fa730b56b543de9997602ddf179eacfde\": rpc error: code = NotFound desc = could not find container \"2b0c06d9226f6087e9468076b60c3f9fa730b56b543de9997602ddf179eacfde\": container with ID starting with 2b0c06d9226f6087e9468076b60c3f9fa730b56b543de9997602ddf179eacfde not found: ID does not exist"
May 25 11:59:30.923807 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:59:30.923786    2561 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-74b58877c9-tfk9j"]
May 25 11:59:30.928165 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:59:30.928141    2561 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-74b58877c9-tfk9j"]
May 25 11:59:31.117308 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:59:31.117255    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8e59cad6-7470-473c-a688-9f15c147b661" path="/var/lib/kubelet/pods/8e59cad6-7470-473c-a688-9f15c147b661/volumes"
May 25 11:59:31.902373 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:59:31.902340    2561 generic.go:358] "Generic (PLEG): container finished" podID="453630ac-6ff6-40a5-9f71-9c778337c1d0" containerID="f5a00fd120056d50dcc5d5155ff305b49b02320120c0501976a819c42dcd3afd" exitCode=0
May 25 11:59:31.902737 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:59:31.902418    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-qq58x" event={"ID":"453630ac-6ff6-40a5-9f71-9c778337c1d0","Type":"ContainerDied","Data":"f5a00fd120056d50dcc5d5155ff305b49b02320120c0501976a819c42dcd3afd"}
May 25 11:59:32.909792 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:59:32.909761    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-qq58x" event={"ID":"453630ac-6ff6-40a5-9f71-9c778337c1d0","Type":"ContainerStarted","Data":"3fd3ece0682fc8414e301ada68a54d3424d8e80689c9c918588b77cd0a516ade"}
May 25 11:59:32.909792 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:59:32.909795    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-qq58x" event={"ID":"453630ac-6ff6-40a5-9f71-9c778337c1d0","Type":"ContainerStarted","Data":"ed681b11b1901f0480efed4a66cd852116d6055f233893c88be21da61e2355ff"}
May 25 11:59:32.910233 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:59:32.909996    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-qq58x"
May 25 11:59:32.930021 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:59:32.929973    2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-qq58x" podStartSLOduration=6.929961372 podStartE2EDuration="6.929961372s" podCreationTimestamp="2026-05-25 11:59:26 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 11:59:32.929318812 +0000 UTC m=+2546.474496416" watchObservedRunningTime="2026-05-25 11:59:32.929961372 +0000 UTC m=+2546.475138974"
May 25 11:59:33.917174 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:59:33.917144    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-qq58x"
May 25 11:59:39.925529 ip-10-0-136-227 kubenswrapper[2561]: I0525 11:59:39.925500    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-qq58x"
May 25 12:00:09.963705 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:00:09.963655    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-qq58x" podUID="453630ac-6ff6-40a5-9f71-9c778337c1d0" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400"
May 25 12:00:19.928394 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:00:19.928367    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-qq58x"
May 25 12:00:26.306878 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:00:26.306815    2561 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-qq58x"]
May 25 12:00:26.307422 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:00:26.307366    2561 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-qq58x" podUID="453630ac-6ff6-40a5-9f71-9c778337c1d0" containerName="kserve-container" containerID="cri-o://ed681b11b1901f0480efed4a66cd852116d6055f233893c88be21da61e2355ff" gracePeriod=30
May 25 12:00:26.307555 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:00:26.307420    2561 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-qq58x" podUID="453630ac-6ff6-40a5-9f71-9c778337c1d0" containerName="kube-rbac-proxy" containerID="cri-o://3fd3ece0682fc8414e301ada68a54d3424d8e80689c9c918588b77cd0a516ade" gracePeriod=30
May 25 12:00:26.402187 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:00:26.402145    2561 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6b9cb59477-pf6d7"]
May 25 12:00:26.402620 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:00:26.402596    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e59cad6-7470-473c-a688-9f15c147b661" containerName="storage-initializer"
May 25 12:00:26.402620 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:00:26.402617    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e59cad6-7470-473c-a688-9f15c147b661" containerName="storage-initializer"
May 25 12:00:26.402775 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:00:26.402641    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e59cad6-7470-473c-a688-9f15c147b661" containerName="kserve-container"
May 25 12:00:26.402775 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:00:26.402651    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e59cad6-7470-473c-a688-9f15c147b661" containerName="kserve-container"
May 25 12:00:26.402775 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:00:26.402677    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e59cad6-7470-473c-a688-9f15c147b661" containerName="kube-rbac-proxy"
May 25 12:00:26.402775 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:00:26.402684    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e59cad6-7470-473c-a688-9f15c147b661" containerName="kube-rbac-proxy"
May 25 12:00:26.402775 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:00:26.402746    2561 memory_manager.go:356] "RemoveStaleState removing state" podUID="8e59cad6-7470-473c-a688-9f15c147b661" containerName="kserve-container"
May 25 12:00:26.402775 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:00:26.402758    2561 memory_manager.go:356] "RemoveStaleState removing state" podUID="8e59cad6-7470-473c-a688-9f15c147b661" containerName="kube-rbac-proxy"
May 25 12:00:26.406233 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:00:26.406189    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6b9cb59477-pf6d7"
May 25 12:00:26.408462 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:00:26.408444    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-runtime-predictor-serving-cert\""
May 25 12:00:26.408536 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:00:26.408496    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-runtime-kube-rbac-proxy-sar-config\""
May 25 12:00:26.418471 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:00:26.418446    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6b9cb59477-pf6d7"]
May 25 12:00:26.517421 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:00:26.517387    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-sklearn-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7b78e3b3-ee79-4279-add7-6c18558f0d67-isvc-sklearn-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-runtime-predictor-6b9cb59477-pf6d7\" (UID: \"7b78e3b3-ee79-4279-add7-6c18558f0d67\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6b9cb59477-pf6d7"
May 25 12:00:26.517421 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:00:26.517423    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7b78e3b3-ee79-4279-add7-6c18558f0d67-kserve-provision-location\") pod \"isvc-sklearn-runtime-predictor-6b9cb59477-pf6d7\" (UID: \"7b78e3b3-ee79-4279-add7-6c18558f0d67\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6b9cb59477-pf6d7"
May 25 12:00:26.517603 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:00:26.517443    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-727nn\" (UniqueName: \"kubernetes.io/projected/7b78e3b3-ee79-4279-add7-6c18558f0d67-kube-api-access-727nn\") pod \"isvc-sklearn-runtime-predictor-6b9cb59477-pf6d7\" (UID: \"7b78e3b3-ee79-4279-add7-6c18558f0d67\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6b9cb59477-pf6d7"
May 25 12:00:26.517603 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:00:26.517590    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7b78e3b3-ee79-4279-add7-6c18558f0d67-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-6b9cb59477-pf6d7\" (UID: \"7b78e3b3-ee79-4279-add7-6c18558f0d67\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6b9cb59477-pf6d7"
May 25 12:00:26.618278 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:00:26.618239    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7b78e3b3-ee79-4279-add7-6c18558f0d67-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-6b9cb59477-pf6d7\" (UID: \"7b78e3b3-ee79-4279-add7-6c18558f0d67\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6b9cb59477-pf6d7"
May 25 12:00:26.618433 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:00:26.618328    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-sklearn-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7b78e3b3-ee79-4279-add7-6c18558f0d67-isvc-sklearn-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-runtime-predictor-6b9cb59477-pf6d7\" (UID: \"7b78e3b3-ee79-4279-add7-6c18558f0d67\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6b9cb59477-pf6d7"
May 25 12:00:26.618433 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:00:26.618352    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7b78e3b3-ee79-4279-add7-6c18558f0d67-kserve-provision-location\") pod \"isvc-sklearn-runtime-predictor-6b9cb59477-pf6d7\" (UID: \"7b78e3b3-ee79-4279-add7-6c18558f0d67\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6b9cb59477-pf6d7"
May 25 12:00:26.618433 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:00:26.618373    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-727nn\" (UniqueName: \"kubernetes.io/projected/7b78e3b3-ee79-4279-add7-6c18558f0d67-kube-api-access-727nn\") pod \"isvc-sklearn-runtime-predictor-6b9cb59477-pf6d7\" (UID: \"7b78e3b3-ee79-4279-add7-6c18558f0d67\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6b9cb59477-pf6d7"
May 25 12:00:26.618821 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:00:26.618798    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7b78e3b3-ee79-4279-add7-6c18558f0d67-kserve-provision-location\") pod \"isvc-sklearn-runtime-predictor-6b9cb59477-pf6d7\" (UID: \"7b78e3b3-ee79-4279-add7-6c18558f0d67\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6b9cb59477-pf6d7"
May 25 12:00:26.618967 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:00:26.618947    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-sklearn-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7b78e3b3-ee79-4279-add7-6c18558f0d67-isvc-sklearn-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-runtime-predictor-6b9cb59477-pf6d7\" (UID: \"7b78e3b3-ee79-4279-add7-6c18558f0d67\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6b9cb59477-pf6d7"
May 25 12:00:26.620746 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:00:26.620723    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7b78e3b3-ee79-4279-add7-6c18558f0d67-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-6b9cb59477-pf6d7\" (UID: \"7b78e3b3-ee79-4279-add7-6c18558f0d67\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6b9cb59477-pf6d7"
May 25 12:00:26.627704 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:00:26.627686    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-727nn\" (UniqueName: \"kubernetes.io/projected/7b78e3b3-ee79-4279-add7-6c18558f0d67-kube-api-access-727nn\") pod \"isvc-sklearn-runtime-predictor-6b9cb59477-pf6d7\" (UID: \"7b78e3b3-ee79-4279-add7-6c18558f0d67\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6b9cb59477-pf6d7"
May 25 12:00:26.721394 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:00:26.721368    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6b9cb59477-pf6d7"
May 25 12:00:26.853217 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:00:26.853190    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6b9cb59477-pf6d7"]
May 25 12:00:26.856070 ip-10-0-136-227 kubenswrapper[2561]: W0525 12:00:26.856039    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b78e3b3_ee79_4279_add7_6c18558f0d67.slice/crio-84ace76928274c36ffc8219bbaa5b7b1d2589d79b2ccaec9e6d9af640ae475a2 WatchSource:0}: Error finding container 84ace76928274c36ffc8219bbaa5b7b1d2589d79b2ccaec9e6d9af640ae475a2: Status 404 returned error can't find the container with id 84ace76928274c36ffc8219bbaa5b7b1d2589d79b2ccaec9e6d9af640ae475a2
May 25 12:00:27.099152 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:00:27.099119    2561 generic.go:358] "Generic (PLEG): container finished" podID="453630ac-6ff6-40a5-9f71-9c778337c1d0" containerID="3fd3ece0682fc8414e301ada68a54d3424d8e80689c9c918588b77cd0a516ade" exitCode=2
May 25 12:00:27.099382 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:00:27.099192    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-qq58x" event={"ID":"453630ac-6ff6-40a5-9f71-9c778337c1d0","Type":"ContainerDied","Data":"3fd3ece0682fc8414e301ada68a54d3424d8e80689c9c918588b77cd0a516ade"}
May 25 12:00:27.100504 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:00:27.100481    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6b9cb59477-pf6d7" event={"ID":"7b78e3b3-ee79-4279-add7-6c18558f0d67","Type":"ContainerStarted","Data":"e8f9d7bfa16dd0fddba8878542496c141d0a69f545f98a2645a6da9197c860cd"}
May 25 12:00:27.100504 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:00:27.100507    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6b9cb59477-pf6d7" event={"ID":"7b78e3b3-ee79-4279-add7-6c18558f0d67","Type":"ContainerStarted","Data":"84ace76928274c36ffc8219bbaa5b7b1d2589d79b2ccaec9e6d9af640ae475a2"}
May 25 12:00:29.921426 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:00:29.921381    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-qq58x" podUID="453630ac-6ff6-40a5-9f71-9c778337c1d0" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.48:8643/healthz\": dial tcp 10.132.0.48:8643: connect: connection refused"
May 25 12:00:29.926874 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:00:29.926840    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-qq58x" podUID="453630ac-6ff6-40a5-9f71-9c778337c1d0" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.48:8080/v2/models/sklearn-v2-mlserver/ready\": dial tcp 10.132.0.48:8080: connect: connection refused"
May 25 12:00:32.961649 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:00:32.961627    2561 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-qq58x"
May 25 12:00:33.069261 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:00:33.069236    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/453630ac-6ff6-40a5-9f71-9c778337c1d0-kserve-provision-location\") pod \"453630ac-6ff6-40a5-9f71-9c778337c1d0\" (UID: \"453630ac-6ff6-40a5-9f71-9c778337c1d0\") "
May 25 12:00:33.069425 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:00:33.069292    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"sklearn-v2-mlserver-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/453630ac-6ff6-40a5-9f71-9c778337c1d0-sklearn-v2-mlserver-kube-rbac-proxy-sar-config\") pod \"453630ac-6ff6-40a5-9f71-9c778337c1d0\" (UID: \"453630ac-6ff6-40a5-9f71-9c778337c1d0\") "
May 25 12:00:33.069425 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:00:33.069333    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/453630ac-6ff6-40a5-9f71-9c778337c1d0-proxy-tls\") pod \"453630ac-6ff6-40a5-9f71-9c778337c1d0\" (UID: \"453630ac-6ff6-40a5-9f71-9c778337c1d0\") "
May 25 12:00:33.069425 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:00:33.069392    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v7ss8\" (UniqueName: \"kubernetes.io/projected/453630ac-6ff6-40a5-9f71-9c778337c1d0-kube-api-access-v7ss8\") pod \"453630ac-6ff6-40a5-9f71-9c778337c1d0\" (UID: \"453630ac-6ff6-40a5-9f71-9c778337c1d0\") "
May 25 12:00:33.069601 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:00:33.069564    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/453630ac-6ff6-40a5-9f71-9c778337c1d0-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "453630ac-6ff6-40a5-9f71-9c778337c1d0" (UID: "453630ac-6ff6-40a5-9f71-9c778337c1d0"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 12:00:33.069667 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:00:33.069627    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/453630ac-6ff6-40a5-9f71-9c778337c1d0-sklearn-v2-mlserver-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "sklearn-v2-mlserver-kube-rbac-proxy-sar-config") pod "453630ac-6ff6-40a5-9f71-9c778337c1d0" (UID: "453630ac-6ff6-40a5-9f71-9c778337c1d0"). InnerVolumeSpecName "sklearn-v2-mlserver-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 12:00:33.071448 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:00:33.071421    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/453630ac-6ff6-40a5-9f71-9c778337c1d0-kube-api-access-v7ss8" (OuterVolumeSpecName: "kube-api-access-v7ss8") pod "453630ac-6ff6-40a5-9f71-9c778337c1d0" (UID: "453630ac-6ff6-40a5-9f71-9c778337c1d0"). InnerVolumeSpecName "kube-api-access-v7ss8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 12:00:33.071575 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:00:33.071512    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/453630ac-6ff6-40a5-9f71-9c778337c1d0-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "453630ac-6ff6-40a5-9f71-9c778337c1d0" (UID: "453630ac-6ff6-40a5-9f71-9c778337c1d0"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 12:00:33.131882 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:00:33.131860    2561 generic.go:358] "Generic (PLEG): container finished" podID="7b78e3b3-ee79-4279-add7-6c18558f0d67" containerID="e8f9d7bfa16dd0fddba8878542496c141d0a69f545f98a2645a6da9197c860cd" exitCode=0
May 25 12:00:33.131998 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:00:33.131934    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6b9cb59477-pf6d7" event={"ID":"7b78e3b3-ee79-4279-add7-6c18558f0d67","Type":"ContainerDied","Data":"e8f9d7bfa16dd0fddba8878542496c141d0a69f545f98a2645a6da9197c860cd"}
May 25 12:00:33.133741 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:00:33.133683    2561 generic.go:358] "Generic (PLEG): container finished" podID="453630ac-6ff6-40a5-9f71-9c778337c1d0" containerID="ed681b11b1901f0480efed4a66cd852116d6055f233893c88be21da61e2355ff" exitCode=0
May 25 12:00:33.133741 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:00:33.133722    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-qq58x" event={"ID":"453630ac-6ff6-40a5-9f71-9c778337c1d0","Type":"ContainerDied","Data":"ed681b11b1901f0480efed4a66cd852116d6055f233893c88be21da61e2355ff"}
May 25 12:00:33.133741 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:00:33.133739    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-qq58x" event={"ID":"453630ac-6ff6-40a5-9f71-9c778337c1d0","Type":"ContainerDied","Data":"862c30f1d5c6396399f932dedf0301d57c0bb1eae8c5538d1d7fda01dfd113ae"}
May 25 12:00:33.133910 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:00:33.133745    2561 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-qq58x"
May 25 12:00:33.133910 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:00:33.133756    2561 scope.go:117] "RemoveContainer" containerID="3fd3ece0682fc8414e301ada68a54d3424d8e80689c9c918588b77cd0a516ade"
May 25 12:00:33.141497 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:00:33.141482    2561 scope.go:117] "RemoveContainer" containerID="ed681b11b1901f0480efed4a66cd852116d6055f233893c88be21da61e2355ff"
May 25 12:00:33.148466 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:00:33.148447    2561 scope.go:117] "RemoveContainer" containerID="f5a00fd120056d50dcc5d5155ff305b49b02320120c0501976a819c42dcd3afd"
May 25 12:00:33.156722 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:00:33.156704    2561 scope.go:117] "RemoveContainer" containerID="3fd3ece0682fc8414e301ada68a54d3424d8e80689c9c918588b77cd0a516ade"
May 25 12:00:33.156964 ip-10-0-136-227 kubenswrapper[2561]: E0525 12:00:33.156946    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3fd3ece0682fc8414e301ada68a54d3424d8e80689c9c918588b77cd0a516ade\": container with ID starting with 3fd3ece0682fc8414e301ada68a54d3424d8e80689c9c918588b77cd0a516ade not found: ID does not exist" containerID="3fd3ece0682fc8414e301ada68a54d3424d8e80689c9c918588b77cd0a516ade"
May 25 12:00:33.157021 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:00:33.156972    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3fd3ece0682fc8414e301ada68a54d3424d8e80689c9c918588b77cd0a516ade"} err="failed to get container status \"3fd3ece0682fc8414e301ada68a54d3424d8e80689c9c918588b77cd0a516ade\": rpc error: code = NotFound desc = could not find container \"3fd3ece0682fc8414e301ada68a54d3424d8e80689c9c918588b77cd0a516ade\": container with ID starting with 3fd3ece0682fc8414e301ada68a54d3424d8e80689c9c918588b77cd0a516ade not found: ID does not exist"
May 25 12:00:33.157021 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:00:33.156994    2561 scope.go:117] "RemoveContainer" containerID="ed681b11b1901f0480efed4a66cd852116d6055f233893c88be21da61e2355ff"
May 25 12:00:33.157229 ip-10-0-136-227 kubenswrapper[2561]: E0525 12:00:33.157210    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed681b11b1901f0480efed4a66cd852116d6055f233893c88be21da61e2355ff\": container with ID starting with ed681b11b1901f0480efed4a66cd852116d6055f233893c88be21da61e2355ff not found: ID does not exist" containerID="ed681b11b1901f0480efed4a66cd852116d6055f233893c88be21da61e2355ff"
May 25 12:00:33.157315 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:00:33.157236    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed681b11b1901f0480efed4a66cd852116d6055f233893c88be21da61e2355ff"} err="failed to get container status \"ed681b11b1901f0480efed4a66cd852116d6055f233893c88be21da61e2355ff\": rpc error: code = NotFound desc = could not find container \"ed681b11b1901f0480efed4a66cd852116d6055f233893c88be21da61e2355ff\": container with ID starting with ed681b11b1901f0480efed4a66cd852116d6055f233893c88be21da61e2355ff not found: ID does not exist"
May 25 12:00:33.157315 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:00:33.157250    2561 scope.go:117] "RemoveContainer" containerID="f5a00fd120056d50dcc5d5155ff305b49b02320120c0501976a819c42dcd3afd"
May 25 12:00:33.157487 ip-10-0-136-227 kubenswrapper[2561]: E0525 12:00:33.157472    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5a00fd120056d50dcc5d5155ff305b49b02320120c0501976a819c42dcd3afd\": container with ID starting with f5a00fd120056d50dcc5d5155ff305b49b02320120c0501976a819c42dcd3afd not found: ID does not exist" containerID="f5a00fd120056d50dcc5d5155ff305b49b02320120c0501976a819c42dcd3afd"
May 25 12:00:33.157533 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:00:33.157491    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5a00fd120056d50dcc5d5155ff305b49b02320120c0501976a819c42dcd3afd"} err="failed to get container status \"f5a00fd120056d50dcc5d5155ff305b49b02320120c0501976a819c42dcd3afd\": rpc error: code = NotFound desc = could not find container \"f5a00fd120056d50dcc5d5155ff305b49b02320120c0501976a819c42dcd3afd\": container with ID starting with f5a00fd120056d50dcc5d5155ff305b49b02320120c0501976a819c42dcd3afd not found: ID does not exist"
May 25 12:00:33.166258 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:00:33.166236    2561 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-qq58x"]
May 25 12:00:33.170460 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:00:33.170440    2561 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-65d8664766-qq58x"]
May 25 12:00:33.170543 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:00:33.170505    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/453630ac-6ff6-40a5-9f71-9c778337c1d0-proxy-tls\") on node \"ip-10-0-136-227.ec2.internal\" DevicePath \"\""
May 25 12:00:33.170543 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:00:33.170530    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v7ss8\" (UniqueName: \"kubernetes.io/projected/453630ac-6ff6-40a5-9f71-9c778337c1d0-kube-api-access-v7ss8\") on node \"ip-10-0-136-227.ec2.internal\" DevicePath \"\""
May 25 12:00:33.170641 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:00:33.170546    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/453630ac-6ff6-40a5-9f71-9c778337c1d0-kserve-provision-location\") on node \"ip-10-0-136-227.ec2.internal\" DevicePath \"\""
May 25 12:00:33.170641 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:00:33.170562    2561 reconciler_common.go:299] "Volume detached for volume \"sklearn-v2-mlserver-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/453630ac-6ff6-40a5-9f71-9c778337c1d0-sklearn-v2-mlserver-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-227.ec2.internal\" DevicePath \"\""
May 25 12:00:34.139500 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:00:34.139465    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6b9cb59477-pf6d7" event={"ID":"7b78e3b3-ee79-4279-add7-6c18558f0d67","Type":"ContainerStarted","Data":"12e9fe72688a00c862fe21c22d6b7034ccf3dd6d466850ec5254a8ab484c4b88"}
May 25 12:00:34.139930 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:00:34.139507    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6b9cb59477-pf6d7" event={"ID":"7b78e3b3-ee79-4279-add7-6c18558f0d67","Type":"ContainerStarted","Data":"b7d882a0dac35854113eb5dee757eda9fcbf546d95c913d6db7eab095fc73d27"}
May 25 12:00:34.139930 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:00:34.139705    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6b9cb59477-pf6d7"
May 25 12:00:34.160297 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:00:34.160239    2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6b9cb59477-pf6d7" podStartSLOduration=8.160226891 podStartE2EDuration="8.160226891s" podCreationTimestamp="2026-05-25 12:00:26 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 12:00:34.158452393 +0000 UTC m=+2607.703630022" watchObservedRunningTime="2026-05-25 12:00:34.160226891 +0000 UTC m=+2607.705404550"
May 25 12:00:35.117724 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:00:35.117695    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="453630ac-6ff6-40a5-9f71-9c778337c1d0" path="/var/lib/kubelet/pods/453630ac-6ff6-40a5-9f71-9c778337c1d0/volumes"
May 25 12:00:35.143110 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:00:35.143081    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6b9cb59477-pf6d7"
May 25 12:00:35.144325 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:00:35.144302    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6b9cb59477-pf6d7" podUID="7b78e3b3-ee79-4279-add7-6c18558f0d67" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.49:8080: connect: connection refused"
May 25 12:00:36.146254 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:00:36.146211    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6b9cb59477-pf6d7" podUID="7b78e3b3-ee79-4279-add7-6c18558f0d67" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.49:8080: connect: connection refused"
May 25 12:00:41.150513 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:00:41.150481    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6b9cb59477-pf6d7"
May 25 12:00:41.151077 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:00:41.150930    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6b9cb59477-pf6d7" podUID="7b78e3b3-ee79-4279-add7-6c18558f0d67" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.49:8080: connect: connection refused"
May 25 12:00:51.151499 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:00:51.151468    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6b9cb59477-pf6d7"
May 25 12:01:03.386157 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:01:03.386119    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-runtime-predictor-6b9cb59477-pf6d7_7b78e3b3-ee79-4279-add7-6c18558f0d67/kserve-container/0.log"
May 25 12:01:03.524371 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:01:03.524327    2561 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6b9cb59477-pf6d7"]
May 25 12:01:03.524783 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:01:03.524732    2561 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6b9cb59477-pf6d7" podUID="7b78e3b3-ee79-4279-add7-6c18558f0d67" containerName="kserve-container" containerID="cri-o://b7d882a0dac35854113eb5dee757eda9fcbf546d95c913d6db7eab095fc73d27" gracePeriod=30
May 25 12:01:03.525016 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:01:03.524799    2561 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6b9cb59477-pf6d7" podUID="7b78e3b3-ee79-4279-add7-6c18558f0d67" containerName="kube-rbac-proxy" containerID="cri-o://12e9fe72688a00c862fe21c22d6b7034ccf3dd6d466850ec5254a8ab484c4b88" gracePeriod=30
May 25 12:01:03.606768 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:01:03.606731    2561 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-bzsvt"]
May 25 12:01:03.607118 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:01:03.607106    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="453630ac-6ff6-40a5-9f71-9c778337c1d0" containerName="kube-rbac-proxy"
May 25 12:01:03.607170 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:01:03.607120    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="453630ac-6ff6-40a5-9f71-9c778337c1d0" containerName="kube-rbac-proxy"
May 25 12:01:03.607170 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:01:03.607130    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="453630ac-6ff6-40a5-9f71-9c778337c1d0" containerName="kserve-container"
May 25 12:01:03.607170 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:01:03.607135    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="453630ac-6ff6-40a5-9f71-9c778337c1d0" containerName="kserve-container"
May 25 12:01:03.607170 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:01:03.607155    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="453630ac-6ff6-40a5-9f71-9c778337c1d0" containerName="storage-initializer"
May 25 12:01:03.607170 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:01:03.607161    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="453630ac-6ff6-40a5-9f71-9c778337c1d0" containerName="storage-initializer"
May 25 12:01:03.607403 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:01:03.607227    2561 memory_manager.go:356] "RemoveStaleState removing state" podUID="453630ac-6ff6-40a5-9f71-9c778337c1d0" containerName="kube-rbac-proxy"
May 25 12:01:03.607403 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:01:03.607239    2561 memory_manager.go:356] "RemoveStaleState removing state" podUID="453630ac-6ff6-40a5-9f71-9c778337c1d0" containerName="kserve-container"
May 25 12:01:03.611011 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:01:03.610988    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-bzsvt"
May 25 12:01:03.613710 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:01:03.613676    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-runtime-predictor-serving-cert\""
May 25 12:01:03.613710 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:01:03.613676    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config\""
May 25 12:01:03.622062 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:01:03.622037    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-bzsvt"]
May 25 12:01:03.650747 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:01:03.650716    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xf8p8\" (UniqueName: \"kubernetes.io/projected/e1a828b8-77fa-4048-9536-9681a05cbb9b-kube-api-access-xf8p8\") pod \"isvc-sklearn-v2-runtime-predictor-6d84c876f4-bzsvt\" (UID: \"e1a828b8-77fa-4048-9536-9681a05cbb9b\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-bzsvt"
May 25 12:01:03.650925 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:01:03.650866    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e1a828b8-77fa-4048-9536-9681a05cbb9b-isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-runtime-predictor-6d84c876f4-bzsvt\" (UID: \"e1a828b8-77fa-4048-9536-9681a05cbb9b\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-bzsvt"
May 25 12:01:03.650925 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:01:03.650909    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e1a828b8-77fa-4048-9536-9681a05cbb9b-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-6d84c876f4-bzsvt\" (UID: \"e1a828b8-77fa-4048-9536-9681a05cbb9b\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-bzsvt"
May 25 12:01:03.651018 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:01:03.650937    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e1a828b8-77fa-4048-9536-9681a05cbb9b-kserve-provision-location\") pod \"isvc-sklearn-v2-runtime-predictor-6d84c876f4-bzsvt\" (UID: \"e1a828b8-77fa-4048-9536-9681a05cbb9b\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-bzsvt"
May 25 12:01:03.752361 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:01:03.752329    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e1a828b8-77fa-4048-9536-9681a05cbb9b-isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-runtime-predictor-6d84c876f4-bzsvt\" (UID: \"e1a828b8-77fa-4048-9536-9681a05cbb9b\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-bzsvt"
May 25 12:01:03.752551 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:01:03.752366    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e1a828b8-77fa-4048-9536-9681a05cbb9b-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-6d84c876f4-bzsvt\" (UID: \"e1a828b8-77fa-4048-9536-9681a05cbb9b\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-bzsvt"
May 25 12:01:03.752551 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:01:03.752388    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e1a828b8-77fa-4048-9536-9681a05cbb9b-kserve-provision-location\") pod \"isvc-sklearn-v2-runtime-predictor-6d84c876f4-bzsvt\" (UID: \"e1a828b8-77fa-4048-9536-9681a05cbb9b\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-bzsvt"
May 25 12:01:03.752551 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:01:03.752436    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-xf8p8\" (UniqueName: \"kubernetes.io/projected/e1a828b8-77fa-4048-9536-9681a05cbb9b-kube-api-access-xf8p8\") pod \"isvc-sklearn-v2-runtime-predictor-6d84c876f4-bzsvt\" (UID: \"e1a828b8-77fa-4048-9536-9681a05cbb9b\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-bzsvt"
May 25 12:01:03.752551 ip-10-0-136-227 kubenswrapper[2561]: E0525 12:01:03.752499    2561 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-serving-cert: secret "isvc-sklearn-v2-runtime-predictor-serving-cert" not found
May 25 12:01:03.752765 ip-10-0-136-227 kubenswrapper[2561]: E0525 12:01:03.752604    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e1a828b8-77fa-4048-9536-9681a05cbb9b-proxy-tls podName:e1a828b8-77fa-4048-9536-9681a05cbb9b nodeName:}" failed. No retries permitted until 2026-05-25 12:01:04.252583555 +0000 UTC m=+2637.797761136 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/e1a828b8-77fa-4048-9536-9681a05cbb9b-proxy-tls") pod "isvc-sklearn-v2-runtime-predictor-6d84c876f4-bzsvt" (UID: "e1a828b8-77fa-4048-9536-9681a05cbb9b") : secret "isvc-sklearn-v2-runtime-predictor-serving-cert" not found
May 25 12:01:03.752957 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:01:03.752935    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e1a828b8-77fa-4048-9536-9681a05cbb9b-kserve-provision-location\") pod \"isvc-sklearn-v2-runtime-predictor-6d84c876f4-bzsvt\" (UID: \"e1a828b8-77fa-4048-9536-9681a05cbb9b\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-bzsvt"
May 25 12:01:03.753036 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:01:03.753020    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e1a828b8-77fa-4048-9536-9681a05cbb9b-isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-runtime-predictor-6d84c876f4-bzsvt\" (UID: \"e1a828b8-77fa-4048-9536-9681a05cbb9b\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-bzsvt"
May 25 12:01:03.761458 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:01:03.761433    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xf8p8\" (UniqueName: \"kubernetes.io/projected/e1a828b8-77fa-4048-9536-9681a05cbb9b-kube-api-access-xf8p8\") pod \"isvc-sklearn-v2-runtime-predictor-6d84c876f4-bzsvt\" (UID: \"e1a828b8-77fa-4048-9536-9681a05cbb9b\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-bzsvt"
May 25 12:01:04.254330 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:01:04.254297    2561 generic.go:358] "Generic (PLEG): container finished" podID="7b78e3b3-ee79-4279-add7-6c18558f0d67" containerID="12e9fe72688a00c862fe21c22d6b7034ccf3dd6d466850ec5254a8ab484c4b88" exitCode=2
May 25 12:01:04.254548 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:01:04.254381    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6b9cb59477-pf6d7" event={"ID":"7b78e3b3-ee79-4279-add7-6c18558f0d67","Type":"ContainerDied","Data":"12e9fe72688a00c862fe21c22d6b7034ccf3dd6d466850ec5254a8ab484c4b88"}
May 25 12:01:04.257731 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:01:04.257703    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e1a828b8-77fa-4048-9536-9681a05cbb9b-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-6d84c876f4-bzsvt\" (UID: \"e1a828b8-77fa-4048-9536-9681a05cbb9b\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-bzsvt"
May 25 12:01:04.257878 ip-10-0-136-227 kubenswrapper[2561]: E0525 12:01:04.257819    2561 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-serving-cert: secret "isvc-sklearn-v2-runtime-predictor-serving-cert" not found
May 25 12:01:04.257878 ip-10-0-136-227 kubenswrapper[2561]: E0525 12:01:04.257871    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e1a828b8-77fa-4048-9536-9681a05cbb9b-proxy-tls podName:e1a828b8-77fa-4048-9536-9681a05cbb9b nodeName:}" failed. No retries permitted until 2026-05-25 12:01:05.257856395 +0000 UTC m=+2638.803033976 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/e1a828b8-77fa-4048-9536-9681a05cbb9b-proxy-tls") pod "isvc-sklearn-v2-runtime-predictor-6d84c876f4-bzsvt" (UID: "e1a828b8-77fa-4048-9536-9681a05cbb9b") : secret "isvc-sklearn-v2-runtime-predictor-serving-cert" not found
May 25 12:01:04.479897 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:01:04.479876    2561 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6b9cb59477-pf6d7"
May 25 12:01:04.560793 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:01:04.560704    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-727nn\" (UniqueName: \"kubernetes.io/projected/7b78e3b3-ee79-4279-add7-6c18558f0d67-kube-api-access-727nn\") pod \"7b78e3b3-ee79-4279-add7-6c18558f0d67\" (UID: \"7b78e3b3-ee79-4279-add7-6c18558f0d67\") "
May 25 12:01:04.560793 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:01:04.560772    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-sklearn-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7b78e3b3-ee79-4279-add7-6c18558f0d67-isvc-sklearn-runtime-kube-rbac-proxy-sar-config\") pod \"7b78e3b3-ee79-4279-add7-6c18558f0d67\" (UID: \"7b78e3b3-ee79-4279-add7-6c18558f0d67\") "
May 25 12:01:04.561021 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:01:04.560844    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7b78e3b3-ee79-4279-add7-6c18558f0d67-proxy-tls\") pod \"7b78e3b3-ee79-4279-add7-6c18558f0d67\" (UID: \"7b78e3b3-ee79-4279-add7-6c18558f0d67\") "
May 25 12:01:04.561021 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:01:04.560941    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7b78e3b3-ee79-4279-add7-6c18558f0d67-kserve-provision-location\") pod \"7b78e3b3-ee79-4279-add7-6c18558f0d67\" (UID: \"7b78e3b3-ee79-4279-add7-6c18558f0d67\") "
May 25 12:01:04.561225 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:01:04.561198    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b78e3b3-ee79-4279-add7-6c18558f0d67-isvc-sklearn-runtime-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-sklearn-runtime-kube-rbac-proxy-sar-config") pod "7b78e3b3-ee79-4279-add7-6c18558f0d67" (UID: "7b78e3b3-ee79-4279-add7-6c18558f0d67"). InnerVolumeSpecName "isvc-sklearn-runtime-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 12:01:04.562864 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:01:04.562840    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b78e3b3-ee79-4279-add7-6c18558f0d67-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7b78e3b3-ee79-4279-add7-6c18558f0d67" (UID: "7b78e3b3-ee79-4279-add7-6c18558f0d67"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 12:01:04.562962 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:01:04.562878    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b78e3b3-ee79-4279-add7-6c18558f0d67-kube-api-access-727nn" (OuterVolumeSpecName: "kube-api-access-727nn") pod "7b78e3b3-ee79-4279-add7-6c18558f0d67" (UID: "7b78e3b3-ee79-4279-add7-6c18558f0d67"). InnerVolumeSpecName "kube-api-access-727nn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 12:01:04.586748 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:01:04.586710    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b78e3b3-ee79-4279-add7-6c18558f0d67-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7b78e3b3-ee79-4279-add7-6c18558f0d67" (UID: "7b78e3b3-ee79-4279-add7-6c18558f0d67"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 12:01:04.662464 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:01:04.662434    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7b78e3b3-ee79-4279-add7-6c18558f0d67-proxy-tls\") on node \"ip-10-0-136-227.ec2.internal\" DevicePath \"\""
May 25 12:01:04.662464 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:01:04.662461    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7b78e3b3-ee79-4279-add7-6c18558f0d67-kserve-provision-location\") on node \"ip-10-0-136-227.ec2.internal\" DevicePath \"\""
May 25 12:01:04.662643 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:01:04.662474    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-727nn\" (UniqueName: \"kubernetes.io/projected/7b78e3b3-ee79-4279-add7-6c18558f0d67-kube-api-access-727nn\") on node \"ip-10-0-136-227.ec2.internal\" DevicePath \"\""
May 25 12:01:04.662643 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:01:04.662484    2561 reconciler_common.go:299] "Volume detached for volume \"isvc-sklearn-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7b78e3b3-ee79-4279-add7-6c18558f0d67-isvc-sklearn-runtime-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-227.ec2.internal\" DevicePath \"\""
May 25 12:01:05.259486 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:01:05.259442    2561 generic.go:358] "Generic (PLEG): container finished" podID="7b78e3b3-ee79-4279-add7-6c18558f0d67" containerID="b7d882a0dac35854113eb5dee757eda9fcbf546d95c913d6db7eab095fc73d27" exitCode=0
May 25 12:01:05.259659 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:01:05.259506    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6b9cb59477-pf6d7" event={"ID":"7b78e3b3-ee79-4279-add7-6c18558f0d67","Type":"ContainerDied","Data":"b7d882a0dac35854113eb5dee757eda9fcbf546d95c913d6db7eab095fc73d27"}
May 25 12:01:05.259659 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:01:05.259548    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6b9cb59477-pf6d7" event={"ID":"7b78e3b3-ee79-4279-add7-6c18558f0d67","Type":"ContainerDied","Data":"84ace76928274c36ffc8219bbaa5b7b1d2589d79b2ccaec9e6d9af640ae475a2"}
May 25 12:01:05.259659 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:01:05.259563    2561 scope.go:117] "RemoveContainer" containerID="12e9fe72688a00c862fe21c22d6b7034ccf3dd6d466850ec5254a8ab484c4b88"
May 25 12:01:05.259659 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:01:05.259565    2561 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6b9cb59477-pf6d7"
May 25 12:01:05.267908 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:01:05.267885    2561 scope.go:117] "RemoveContainer" containerID="b7d882a0dac35854113eb5dee757eda9fcbf546d95c913d6db7eab095fc73d27"
May 25 12:01:05.268287 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:01:05.268246    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e1a828b8-77fa-4048-9536-9681a05cbb9b-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-6d84c876f4-bzsvt\" (UID: \"e1a828b8-77fa-4048-9536-9681a05cbb9b\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-bzsvt"
May 25 12:01:05.271113 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:01:05.271085    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e1a828b8-77fa-4048-9536-9681a05cbb9b-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-6d84c876f4-bzsvt\" (UID: \"e1a828b8-77fa-4048-9536-9681a05cbb9b\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-bzsvt"
May 25 12:01:05.282195 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:01:05.282173    2561 scope.go:117] "RemoveContainer" containerID="e8f9d7bfa16dd0fddba8878542496c141d0a69f545f98a2645a6da9197c860cd"
May 25 12:01:05.283000 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:01:05.282972    2561 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6b9cb59477-pf6d7"]
May 25 12:01:05.288349 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:01:05.288328    2561 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6b9cb59477-pf6d7"]
May 25 12:01:05.290905 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:01:05.290884    2561 scope.go:117] "RemoveContainer" containerID="12e9fe72688a00c862fe21c22d6b7034ccf3dd6d466850ec5254a8ab484c4b88"
May 25 12:01:05.291185 ip-10-0-136-227 kubenswrapper[2561]: E0525 12:01:05.291169    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12e9fe72688a00c862fe21c22d6b7034ccf3dd6d466850ec5254a8ab484c4b88\": container with ID starting with 12e9fe72688a00c862fe21c22d6b7034ccf3dd6d466850ec5254a8ab484c4b88 not found: ID does not exist" containerID="12e9fe72688a00c862fe21c22d6b7034ccf3dd6d466850ec5254a8ab484c4b88"
May 25 12:01:05.291250 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:01:05.291194    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12e9fe72688a00c862fe21c22d6b7034ccf3dd6d466850ec5254a8ab484c4b88"} err="failed to get container status \"12e9fe72688a00c862fe21c22d6b7034ccf3dd6d466850ec5254a8ab484c4b88\": rpc error: code = NotFound desc = could not find container \"12e9fe72688a00c862fe21c22d6b7034ccf3dd6d466850ec5254a8ab484c4b88\": container with ID starting with 12e9fe72688a00c862fe21c22d6b7034ccf3dd6d466850ec5254a8ab484c4b88 not found: ID does not exist"
May 25 12:01:05.291250 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:01:05.291212    2561 scope.go:117] "RemoveContainer" containerID="b7d882a0dac35854113eb5dee757eda9fcbf546d95c913d6db7eab095fc73d27"
May 25 12:01:05.291481 ip-10-0-136-227 kubenswrapper[2561]: E0525 12:01:05.291464    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7d882a0dac35854113eb5dee757eda9fcbf546d95c913d6db7eab095fc73d27\": container with ID starting with b7d882a0dac35854113eb5dee757eda9fcbf546d95c913d6db7eab095fc73d27 not found: ID does not exist" containerID="b7d882a0dac35854113eb5dee757eda9fcbf546d95c913d6db7eab095fc73d27"
May 25 12:01:05.291531 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:01:05.291486    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7d882a0dac35854113eb5dee757eda9fcbf546d95c913d6db7eab095fc73d27"} err="failed to get container status \"b7d882a0dac35854113eb5dee757eda9fcbf546d95c913d6db7eab095fc73d27\": rpc error: code = NotFound desc = could not find container \"b7d882a0dac35854113eb5dee757eda9fcbf546d95c913d6db7eab095fc73d27\": container with ID starting with b7d882a0dac35854113eb5dee757eda9fcbf546d95c913d6db7eab095fc73d27 not found: ID does not exist"
May 25 12:01:05.291531 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:01:05.291498    2561 scope.go:117] "RemoveContainer" containerID="e8f9d7bfa16dd0fddba8878542496c141d0a69f545f98a2645a6da9197c860cd"
May 25 12:01:05.291718 ip-10-0-136-227 kubenswrapper[2561]: E0525 12:01:05.291705    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8f9d7bfa16dd0fddba8878542496c141d0a69f545f98a2645a6da9197c860cd\": container with ID starting with e8f9d7bfa16dd0fddba8878542496c141d0a69f545f98a2645a6da9197c860cd not found: ID does not exist" containerID="e8f9d7bfa16dd0fddba8878542496c141d0a69f545f98a2645a6da9197c860cd"
May 25 12:01:05.291767 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:01:05.291723    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8f9d7bfa16dd0fddba8878542496c141d0a69f545f98a2645a6da9197c860cd"} err="failed to get container status \"e8f9d7bfa16dd0fddba8878542496c141d0a69f545f98a2645a6da9197c860cd\": rpc error: code = NotFound desc = could not find container \"e8f9d7bfa16dd0fddba8878542496c141d0a69f545f98a2645a6da9197c860cd\": container with ID starting with e8f9d7bfa16dd0fddba8878542496c141d0a69f545f98a2645a6da9197c860cd not found: ID does not exist"
May 25 12:01:05.422501 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:01:05.422406    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-bzsvt"
May 25 12:01:05.552512 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:01:05.552487    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-bzsvt"]
May 25 12:01:05.556520 ip-10-0-136-227 kubenswrapper[2561]: W0525 12:01:05.556487    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1a828b8_77fa_4048_9536_9681a05cbb9b.slice/crio-5a6308b1aae1d34bf73afedf60af1d60157e406a68713d477b210c6ac43a5763 WatchSource:0}: Error finding container 5a6308b1aae1d34bf73afedf60af1d60157e406a68713d477b210c6ac43a5763: Status 404 returned error can't find the container with id 5a6308b1aae1d34bf73afedf60af1d60157e406a68713d477b210c6ac43a5763
May 25 12:01:06.264983 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:01:06.264947    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-bzsvt" event={"ID":"e1a828b8-77fa-4048-9536-9681a05cbb9b","Type":"ContainerStarted","Data":"af086304278e63b909dbad26e8ebb95cec328a8e810cc2ea8aad7eada5589bda"}
May 25 12:01:06.264983 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:01:06.264986    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-bzsvt" event={"ID":"e1a828b8-77fa-4048-9536-9681a05cbb9b","Type":"ContainerStarted","Data":"5a6308b1aae1d34bf73afedf60af1d60157e406a68713d477b210c6ac43a5763"}
May 25 12:01:07.117257 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:01:07.117228    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7b78e3b3-ee79-4279-add7-6c18558f0d67" path="/var/lib/kubelet/pods/7b78e3b3-ee79-4279-add7-6c18558f0d67/volumes"
May 25 12:01:10.280007 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:01:10.279970    2561 generic.go:358] "Generic (PLEG): container finished" podID="e1a828b8-77fa-4048-9536-9681a05cbb9b" containerID="af086304278e63b909dbad26e8ebb95cec328a8e810cc2ea8aad7eada5589bda" exitCode=0
May 25 12:01:10.280417 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:01:10.280047    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-bzsvt" event={"ID":"e1a828b8-77fa-4048-9536-9681a05cbb9b","Type":"ContainerDied","Data":"af086304278e63b909dbad26e8ebb95cec328a8e810cc2ea8aad7eada5589bda"}
May 25 12:01:11.285190 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:01:11.285155    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-bzsvt" event={"ID":"e1a828b8-77fa-4048-9536-9681a05cbb9b","Type":"ContainerStarted","Data":"ae0c246d6f68d932ba85c1ed88faea3b7a43799b713c9dda0ef6fddfaccbd138"}
May 25 12:01:11.285190 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:01:11.285194    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-bzsvt" event={"ID":"e1a828b8-77fa-4048-9536-9681a05cbb9b","Type":"ContainerStarted","Data":"55008edac9a07a25e7e7f8bd5670db4730de279c39e002182b3a59bb73f9c989"}
May 25 12:01:11.285674 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:01:11.285516    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-bzsvt"
May 25 12:01:11.285674 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:01:11.285543    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-bzsvt"
May 25 12:01:11.305952 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:01:11.305912    2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-bzsvt" podStartSLOduration=8.305899858 podStartE2EDuration="8.305899858s" podCreationTimestamp="2026-05-25 12:01:03 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 12:01:11.304932864 +0000 UTC m=+2644.850110471" watchObservedRunningTime="2026-05-25 12:01:11.305899858 +0000 UTC m=+2644.851077458"
May 25 12:01:17.293734 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:01:17.293706    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-bzsvt"
May 25 12:01:47.363686 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:01:47.363640    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-bzsvt" podUID="e1a828b8-77fa-4048-9536-9681a05cbb9b" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400"
May 25 12:01:57.297214 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:01:57.297134    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-bzsvt"
May 25 12:02:03.702146 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:02:03.702111    2561 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-bzsvt"]
May 25 12:02:03.702687 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:02:03.702469    2561 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-bzsvt" podUID="e1a828b8-77fa-4048-9536-9681a05cbb9b" containerName="kserve-container" containerID="cri-o://55008edac9a07a25e7e7f8bd5670db4730de279c39e002182b3a59bb73f9c989" gracePeriod=30
May 25 12:02:03.702687 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:02:03.702531    2561 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-bzsvt" podUID="e1a828b8-77fa-4048-9536-9681a05cbb9b" containerName="kube-rbac-proxy" containerID="cri-o://ae0c246d6f68d932ba85c1ed88faea3b7a43799b713c9dda0ef6fddfaccbd138" gracePeriod=30
May 25 12:02:03.781246 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:02:03.781202    2561 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-58dfb5f9bc-hr4n2"]
May 25 12:02:03.781745 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:02:03.781711    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b78e3b3-ee79-4279-add7-6c18558f0d67" containerName="kube-rbac-proxy"
May 25 12:02:03.781745 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:02:03.781734    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b78e3b3-ee79-4279-add7-6c18558f0d67" containerName="kube-rbac-proxy"
May 25 12:02:03.781941 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:02:03.781758    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b78e3b3-ee79-4279-add7-6c18558f0d67" containerName="storage-initializer"
May 25 12:02:03.781941 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:02:03.781765    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b78e3b3-ee79-4279-add7-6c18558f0d67" containerName="storage-initializer"
May 25 12:02:03.781941 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:02:03.781790    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b78e3b3-ee79-4279-add7-6c18558f0d67" containerName="kserve-container"
May 25 12:02:03.781941 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:02:03.781797    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b78e3b3-ee79-4279-add7-6c18558f0d67" containerName="kserve-container"
May 25 12:02:03.781941 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:02:03.781866    2561 memory_manager.go:356] "RemoveStaleState removing state" podUID="7b78e3b3-ee79-4279-add7-6c18558f0d67" containerName="kube-rbac-proxy"
May 25 12:02:03.781941 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:02:03.781877    2561 memory_manager.go:356] "RemoveStaleState removing state" podUID="7b78e3b3-ee79-4279-add7-6c18558f0d67" containerName="kserve-container"
May 25 12:02:03.785384 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:02:03.785366    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-58dfb5f9bc-hr4n2"
May 25 12:02:03.788020 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:02:03.787997    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-kube-rbac-proxy-sar-config\""
May 25 12:02:03.788290 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:02:03.788254    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-predictor-serving-cert\""
May 25 12:02:03.794469 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:02:03.794447    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-58dfb5f9bc-hr4n2"]
May 25 12:02:03.831972 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:02:03.831944    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7bwt\" (UniqueName: \"kubernetes.io/projected/ca1b946e-29ff-4f5c-966b-764da5e4a57e-kube-api-access-k7bwt\") pod \"isvc-sklearn-v2-predictor-58dfb5f9bc-hr4n2\" (UID: \"ca1b946e-29ff-4f5c-966b-764da5e4a57e\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-58dfb5f9bc-hr4n2"
May 25 12:02:03.832091 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:02:03.832008    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ca1b946e-29ff-4f5c-966b-764da5e4a57e-proxy-tls\") pod \"isvc-sklearn-v2-predictor-58dfb5f9bc-hr4n2\" (UID: \"ca1b946e-29ff-4f5c-966b-764da5e4a57e\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-58dfb5f9bc-hr4n2"
May 25 12:02:03.832091 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:02:03.832056    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-sklearn-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ca1b946e-29ff-4f5c-966b-764da5e4a57e-isvc-sklearn-v2-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-predictor-58dfb5f9bc-hr4n2\" (UID: \"ca1b946e-29ff-4f5c-966b-764da5e4a57e\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-58dfb5f9bc-hr4n2"
May 25 12:02:03.832091 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:02:03.832075    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ca1b946e-29ff-4f5c-966b-764da5e4a57e-kserve-provision-location\") pod \"isvc-sklearn-v2-predictor-58dfb5f9bc-hr4n2\" (UID: \"ca1b946e-29ff-4f5c-966b-764da5e4a57e\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-58dfb5f9bc-hr4n2"
May 25 12:02:03.933107 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:02:03.933054    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-k7bwt\" (UniqueName: \"kubernetes.io/projected/ca1b946e-29ff-4f5c-966b-764da5e4a57e-kube-api-access-k7bwt\") pod \"isvc-sklearn-v2-predictor-58dfb5f9bc-hr4n2\" (UID: \"ca1b946e-29ff-4f5c-966b-764da5e4a57e\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-58dfb5f9bc-hr4n2"
May 25 12:02:03.933326 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:02:03.933181    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ca1b946e-29ff-4f5c-966b-764da5e4a57e-proxy-tls\") pod \"isvc-sklearn-v2-predictor-58dfb5f9bc-hr4n2\" (UID: \"ca1b946e-29ff-4f5c-966b-764da5e4a57e\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-58dfb5f9bc-hr4n2"
May 25 12:02:03.933326 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:02:03.933305    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-sklearn-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ca1b946e-29ff-4f5c-966b-764da5e4a57e-isvc-sklearn-v2-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-predictor-58dfb5f9bc-hr4n2\" (UID: \"ca1b946e-29ff-4f5c-966b-764da5e4a57e\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-58dfb5f9bc-hr4n2"
May 25 12:02:03.933476 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:02:03.933340    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ca1b946e-29ff-4f5c-966b-764da5e4a57e-kserve-provision-location\") pod \"isvc-sklearn-v2-predictor-58dfb5f9bc-hr4n2\" (UID: \"ca1b946e-29ff-4f5c-966b-764da5e4a57e\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-58dfb5f9bc-hr4n2"
May 25 12:02:03.933814 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:02:03.933786    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ca1b946e-29ff-4f5c-966b-764da5e4a57e-kserve-provision-location\") pod \"isvc-sklearn-v2-predictor-58dfb5f9bc-hr4n2\" (UID: \"ca1b946e-29ff-4f5c-966b-764da5e4a57e\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-58dfb5f9bc-hr4n2"
May 25 12:02:03.934107 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:02:03.934086    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-sklearn-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ca1b946e-29ff-4f5c-966b-764da5e4a57e-isvc-sklearn-v2-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-predictor-58dfb5f9bc-hr4n2\" (UID: \"ca1b946e-29ff-4f5c-966b-764da5e4a57e\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-58dfb5f9bc-hr4n2"
May 25 12:02:03.935695 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:02:03.935678    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ca1b946e-29ff-4f5c-966b-764da5e4a57e-proxy-tls\") pod \"isvc-sklearn-v2-predictor-58dfb5f9bc-hr4n2\" (UID: \"ca1b946e-29ff-4f5c-966b-764da5e4a57e\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-58dfb5f9bc-hr4n2"
May 25 12:02:03.941207 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:02:03.941175    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7bwt\" (UniqueName: \"kubernetes.io/projected/ca1b946e-29ff-4f5c-966b-764da5e4a57e-kube-api-access-k7bwt\") pod \"isvc-sklearn-v2-predictor-58dfb5f9bc-hr4n2\" (UID: \"ca1b946e-29ff-4f5c-966b-764da5e4a57e\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-58dfb5f9bc-hr4n2"
May 25 12:02:04.096769 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:02:04.096744    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-58dfb5f9bc-hr4n2"
May 25 12:02:04.226184 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:02:04.226162    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-58dfb5f9bc-hr4n2"]
May 25 12:02:04.228252 ip-10-0-136-227 kubenswrapper[2561]: W0525 12:02:04.228227    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca1b946e_29ff_4f5c_966b_764da5e4a57e.slice/crio-a2747d783ea7f81ea33a13d578e173312aabffc59b721f7c14309671670046fd WatchSource:0}: Error finding container a2747d783ea7f81ea33a13d578e173312aabffc59b721f7c14309671670046fd: Status 404 returned error can't find the container with id a2747d783ea7f81ea33a13d578e173312aabffc59b721f7c14309671670046fd
May 25 12:02:04.460403 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:02:04.460330    2561 generic.go:358] "Generic (PLEG): container finished" podID="e1a828b8-77fa-4048-9536-9681a05cbb9b" containerID="ae0c246d6f68d932ba85c1ed88faea3b7a43799b713c9dda0ef6fddfaccbd138" exitCode=2
May 25 12:02:04.460551 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:02:04.460410    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-bzsvt" event={"ID":"e1a828b8-77fa-4048-9536-9681a05cbb9b","Type":"ContainerDied","Data":"ae0c246d6f68d932ba85c1ed88faea3b7a43799b713c9dda0ef6fddfaccbd138"}
May 25 12:02:04.461678 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:02:04.461654    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-58dfb5f9bc-hr4n2" event={"ID":"ca1b946e-29ff-4f5c-966b-764da5e4a57e","Type":"ContainerStarted","Data":"ebb96a500dd1f139aca7224086fb69d17c1f1cbb7a6d1a5b874ec0391bb423f4"}
May 25 12:02:04.461808 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:02:04.461683    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-58dfb5f9bc-hr4n2" event={"ID":"ca1b946e-29ff-4f5c-966b-764da5e4a57e","Type":"ContainerStarted","Data":"a2747d783ea7f81ea33a13d578e173312aabffc59b721f7c14309671670046fd"}
May 25 12:02:07.188066 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:02:07.188039    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-9lj95_10a66344-3304-49a1-bbc1-b885800f5acf/console-operator/1.log"
May 25 12:02:07.193838 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:02:07.193817    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-9lj95_10a66344-3304-49a1-bbc1-b885800f5acf/console-operator/1.log"
May 25 12:02:07.289480 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:02:07.289451    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-bzsvt" podUID="e1a828b8-77fa-4048-9536-9681a05cbb9b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.50:8643/healthz\": dial tcp 10.132.0.50:8643: connect: connection refused"
May 25 12:02:07.294878 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:02:07.294857    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-bzsvt" podUID="e1a828b8-77fa-4048-9536-9681a05cbb9b" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.50:8080/v2/models/isvc-sklearn-v2-runtime/ready\": dial tcp 10.132.0.50:8080: connect: connection refused"
May 25 12:02:08.476242 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:02:08.476211    2561 generic.go:358] "Generic (PLEG): container finished" podID="ca1b946e-29ff-4f5c-966b-764da5e4a57e" containerID="ebb96a500dd1f139aca7224086fb69d17c1f1cbb7a6d1a5b874ec0391bb423f4" exitCode=0
May 25 12:02:08.476615 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:02:08.476295    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-58dfb5f9bc-hr4n2" event={"ID":"ca1b946e-29ff-4f5c-966b-764da5e4a57e","Type":"ContainerDied","Data":"ebb96a500dd1f139aca7224086fb69d17c1f1cbb7a6d1a5b874ec0391bb423f4"}
May 25 12:02:09.482026 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:02:09.481994    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-58dfb5f9bc-hr4n2" event={"ID":"ca1b946e-29ff-4f5c-966b-764da5e4a57e","Type":"ContainerStarted","Data":"1a03ecc28a11c4f2d2b1d5f68484a1f9a57bb43416a8c3217bcc1c074684416c"}
May 25 12:02:09.482026 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:02:09.482032    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-58dfb5f9bc-hr4n2" event={"ID":"ca1b946e-29ff-4f5c-966b-764da5e4a57e","Type":"ContainerStarted","Data":"76b8ccd9aa605f963bc90a89a84c57ba893d8bcead68eb89dcf1b66af6381351"}
May 25 12:02:09.482445 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:02:09.482355    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-58dfb5f9bc-hr4n2"
May 25 12:02:09.482505 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:02:09.482486    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-58dfb5f9bc-hr4n2"
May 25 12:02:09.483495 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:02:09.483471    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-58dfb5f9bc-hr4n2" podUID="ca1b946e-29ff-4f5c-966b-764da5e4a57e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.51:8080: connect: connection refused"
May 25 12:02:09.513574 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:02:09.513527    2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-58dfb5f9bc-hr4n2" podStartSLOduration=6.513512923 podStartE2EDuration="6.513512923s" podCreationTimestamp="2026-05-25 12:02:03 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 12:02:09.513456989 +0000 UTC m=+2703.058634589" watchObservedRunningTime="2026-05-25 12:02:09.513512923 +0000 UTC m=+2703.058690526"
May 25 12:02:09.947508 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:02:09.947483    2561 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-bzsvt"
May 25 12:02:10.080602 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:02:10.080582    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e1a828b8-77fa-4048-9536-9681a05cbb9b-proxy-tls\") pod \"e1a828b8-77fa-4048-9536-9681a05cbb9b\" (UID: \"e1a828b8-77fa-4048-9536-9681a05cbb9b\") "
May 25 12:02:10.080719 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:02:10.080644    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xf8p8\" (UniqueName: \"kubernetes.io/projected/e1a828b8-77fa-4048-9536-9681a05cbb9b-kube-api-access-xf8p8\") pod \"e1a828b8-77fa-4048-9536-9681a05cbb9b\" (UID: \"e1a828b8-77fa-4048-9536-9681a05cbb9b\") "
May 25 12:02:10.080858 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:02:10.080839    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e1a828b8-77fa-4048-9536-9681a05cbb9b-isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config\") pod \"e1a828b8-77fa-4048-9536-9681a05cbb9b\" (UID: \"e1a828b8-77fa-4048-9536-9681a05cbb9b\") "
May 25 12:02:10.080910 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:02:10.080893    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e1a828b8-77fa-4048-9536-9681a05cbb9b-kserve-provision-location\") pod \"e1a828b8-77fa-4048-9536-9681a05cbb9b\" (UID: \"e1a828b8-77fa-4048-9536-9681a05cbb9b\") "
May 25 12:02:10.081185 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:02:10.081162    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1a828b8-77fa-4048-9536-9681a05cbb9b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e1a828b8-77fa-4048-9536-9681a05cbb9b" (UID: "e1a828b8-77fa-4048-9536-9681a05cbb9b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 12:02:10.081284 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:02:10.081190    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1a828b8-77fa-4048-9536-9681a05cbb9b-isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config") pod "e1a828b8-77fa-4048-9536-9681a05cbb9b" (UID: "e1a828b8-77fa-4048-9536-9681a05cbb9b"). InnerVolumeSpecName "isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 12:02:10.082670 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:02:10.082646    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1a828b8-77fa-4048-9536-9681a05cbb9b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e1a828b8-77fa-4048-9536-9681a05cbb9b" (UID: "e1a828b8-77fa-4048-9536-9681a05cbb9b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 12:02:10.082760 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:02:10.082730    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1a828b8-77fa-4048-9536-9681a05cbb9b-kube-api-access-xf8p8" (OuterVolumeSpecName: "kube-api-access-xf8p8") pod "e1a828b8-77fa-4048-9536-9681a05cbb9b" (UID: "e1a828b8-77fa-4048-9536-9681a05cbb9b"). InnerVolumeSpecName "kube-api-access-xf8p8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 12:02:10.182138 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:02:10.182116    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e1a828b8-77fa-4048-9536-9681a05cbb9b-proxy-tls\") on node \"ip-10-0-136-227.ec2.internal\" DevicePath \"\""
May 25 12:02:10.182138 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:02:10.182137    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xf8p8\" (UniqueName: \"kubernetes.io/projected/e1a828b8-77fa-4048-9536-9681a05cbb9b-kube-api-access-xf8p8\") on node \"ip-10-0-136-227.ec2.internal\" DevicePath \"\""
May 25 12:02:10.182262 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:02:10.182148    2561 reconciler_common.go:299] "Volume detached for volume \"isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e1a828b8-77fa-4048-9536-9681a05cbb9b-isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-227.ec2.internal\" DevicePath \"\""
May 25 12:02:10.182262 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:02:10.182158    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e1a828b8-77fa-4048-9536-9681a05cbb9b-kserve-provision-location\") on node \"ip-10-0-136-227.ec2.internal\" DevicePath \"\""
May 25 12:02:10.489515 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:02:10.489417    2561 generic.go:358] "Generic (PLEG): container finished" podID="e1a828b8-77fa-4048-9536-9681a05cbb9b" containerID="55008edac9a07a25e7e7f8bd5670db4730de279c39e002182b3a59bb73f9c989" exitCode=0
May 25 12:02:10.489515 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:02:10.489449    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-bzsvt" event={"ID":"e1a828b8-77fa-4048-9536-9681a05cbb9b","Type":"ContainerDied","Data":"55008edac9a07a25e7e7f8bd5670db4730de279c39e002182b3a59bb73f9c989"}
May 25 12:02:10.489515 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:02:10.489493    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-bzsvt" event={"ID":"e1a828b8-77fa-4048-9536-9681a05cbb9b","Type":"ContainerDied","Data":"5a6308b1aae1d34bf73afedf60af1d60157e406a68713d477b210c6ac43a5763"}
May 25 12:02:10.489515 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:02:10.489510    2561 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-bzsvt"
May 25 12:02:10.490086 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:02:10.489513    2561 scope.go:117] "RemoveContainer" containerID="ae0c246d6f68d932ba85c1ed88faea3b7a43799b713c9dda0ef6fddfaccbd138"
May 25 12:02:10.490149 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:02:10.490076    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-58dfb5f9bc-hr4n2" podUID="ca1b946e-29ff-4f5c-966b-764da5e4a57e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.51:8080: connect: connection refused"
May 25 12:02:10.498128 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:02:10.498111    2561 scope.go:117] "RemoveContainer" containerID="55008edac9a07a25e7e7f8bd5670db4730de279c39e002182b3a59bb73f9c989"
May 25 12:02:10.505387 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:02:10.505370    2561 scope.go:117] "RemoveContainer" containerID="af086304278e63b909dbad26e8ebb95cec328a8e810cc2ea8aad7eada5589bda"
May 25 12:02:10.511003 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:02:10.510980    2561 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-bzsvt"]
May 25 12:02:10.513013 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:02:10.512999    2561 scope.go:117] "RemoveContainer" containerID="ae0c246d6f68d932ba85c1ed88faea3b7a43799b713c9dda0ef6fddfaccbd138"
May 25 12:02:10.513285 ip-10-0-136-227 kubenswrapper[2561]: E0525 12:02:10.513246    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae0c246d6f68d932ba85c1ed88faea3b7a43799b713c9dda0ef6fddfaccbd138\": container with ID starting with ae0c246d6f68d932ba85c1ed88faea3b7a43799b713c9dda0ef6fddfaccbd138 not found: ID does not exist" containerID="ae0c246d6f68d932ba85c1ed88faea3b7a43799b713c9dda0ef6fddfaccbd138"
May 25 12:02:10.513329 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:02:10.513288    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae0c246d6f68d932ba85c1ed88faea3b7a43799b713c9dda0ef6fddfaccbd138"} err="failed to get container status \"ae0c246d6f68d932ba85c1ed88faea3b7a43799b713c9dda0ef6fddfaccbd138\": rpc error: code = NotFound desc = could not find container \"ae0c246d6f68d932ba85c1ed88faea3b7a43799b713c9dda0ef6fddfaccbd138\": container with ID starting with ae0c246d6f68d932ba85c1ed88faea3b7a43799b713c9dda0ef6fddfaccbd138 not found: ID does not exist"
May 25 12:02:10.513329 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:02:10.513305    2561 scope.go:117] "RemoveContainer" containerID="55008edac9a07a25e7e7f8bd5670db4730de279c39e002182b3a59bb73f9c989"
May 25 12:02:10.513543 ip-10-0-136-227 kubenswrapper[2561]: E0525 12:02:10.513528    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55008edac9a07a25e7e7f8bd5670db4730de279c39e002182b3a59bb73f9c989\": container with ID starting with 55008edac9a07a25e7e7f8bd5670db4730de279c39e002182b3a59bb73f9c989 not found: ID does not exist" containerID="55008edac9a07a25e7e7f8bd5670db4730de279c39e002182b3a59bb73f9c989"
May 25 12:02:10.513590 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:02:10.513546    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55008edac9a07a25e7e7f8bd5670db4730de279c39e002182b3a59bb73f9c989"} err="failed to get container status \"55008edac9a07a25e7e7f8bd5670db4730de279c39e002182b3a59bb73f9c989\": rpc error: code = NotFound desc = could not find container \"55008edac9a07a25e7e7f8bd5670db4730de279c39e002182b3a59bb73f9c989\": container with ID starting with 55008edac9a07a25e7e7f8bd5670db4730de279c39e002182b3a59bb73f9c989 not found: ID does not exist"
May 25 12:02:10.513590 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:02:10.513558    2561 scope.go:117] "RemoveContainer" containerID="af086304278e63b909dbad26e8ebb95cec328a8e810cc2ea8aad7eada5589bda"
May 25 12:02:10.513780 ip-10-0-136-227 kubenswrapper[2561]: E0525 12:02:10.513762    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af086304278e63b909dbad26e8ebb95cec328a8e810cc2ea8aad7eada5589bda\": container with ID starting with af086304278e63b909dbad26e8ebb95cec328a8e810cc2ea8aad7eada5589bda not found: ID does not exist" containerID="af086304278e63b909dbad26e8ebb95cec328a8e810cc2ea8aad7eada5589bda"
May 25 12:02:10.513820 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:02:10.513788    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af086304278e63b909dbad26e8ebb95cec328a8e810cc2ea8aad7eada5589bda"} err="failed to get container status \"af086304278e63b909dbad26e8ebb95cec328a8e810cc2ea8aad7eada5589bda\": rpc error: code = NotFound desc = could not find container \"af086304278e63b909dbad26e8ebb95cec328a8e810cc2ea8aad7eada5589bda\": container with ID starting with af086304278e63b909dbad26e8ebb95cec328a8e810cc2ea8aad7eada5589bda not found: ID does not exist"
May 25 12:02:10.516667 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:02:10.516647    2561 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-6d84c876f4-bzsvt"]
May 25 12:02:11.117138 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:02:11.117106    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e1a828b8-77fa-4048-9536-9681a05cbb9b" path="/var/lib/kubelet/pods/e1a828b8-77fa-4048-9536-9681a05cbb9b/volumes"
May 25 12:02:11.492862 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:02:11.492777    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-58dfb5f9bc-hr4n2" podUID="ca1b946e-29ff-4f5c-966b-764da5e4a57e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.51:8080: connect: connection refused"
May 25 12:02:16.497280 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:02:16.497243    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-58dfb5f9bc-hr4n2"
May 25 12:02:16.497827 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:02:16.497797    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-58dfb5f9bc-hr4n2" podUID="ca1b946e-29ff-4f5c-966b-764da5e4a57e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.51:8080: connect: connection refused"
May 25 12:02:26.497834 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:02:26.497787    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-58dfb5f9bc-hr4n2" podUID="ca1b946e-29ff-4f5c-966b-764da5e4a57e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.51:8080: connect: connection refused"
May 25 12:02:36.497882 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:02:36.497841    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-58dfb5f9bc-hr4n2" podUID="ca1b946e-29ff-4f5c-966b-764da5e4a57e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.51:8080: connect: connection refused"
May 25 12:02:46.498248 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:02:46.498201    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-58dfb5f9bc-hr4n2" podUID="ca1b946e-29ff-4f5c-966b-764da5e4a57e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.51:8080: connect: connection refused"
May 25 12:02:56.498643 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:02:56.498602    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-58dfb5f9bc-hr4n2" podUID="ca1b946e-29ff-4f5c-966b-764da5e4a57e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.51:8080: connect: connection refused"
May 25 12:03:06.497852 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:03:06.497813    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-58dfb5f9bc-hr4n2" podUID="ca1b946e-29ff-4f5c-966b-764da5e4a57e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.51:8080: connect: connection refused"
May 25 12:03:16.499378 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:03:16.499349    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-58dfb5f9bc-hr4n2"
May 25 12:03:23.953263 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:03:23.953220    2561 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-58dfb5f9bc-hr4n2"]
May 25 12:03:23.953737 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:03:23.953686    2561 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-58dfb5f9bc-hr4n2" podUID="ca1b946e-29ff-4f5c-966b-764da5e4a57e" containerName="kserve-container" containerID="cri-o://76b8ccd9aa605f963bc90a89a84c57ba893d8bcead68eb89dcf1b66af6381351" gracePeriod=30
May 25 12:03:23.953814 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:03:23.953760    2561 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-58dfb5f9bc-hr4n2" podUID="ca1b946e-29ff-4f5c-966b-764da5e4a57e" containerName="kube-rbac-proxy" containerID="cri-o://1a03ecc28a11c4f2d2b1d5f68484a1f9a57bb43416a8c3217bcc1c074684416c" gracePeriod=30
May 25 12:03:24.036543 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:03:24.036517    2561 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6bcc6c87db-pmg7w"]
May 25 12:03:24.036876 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:03:24.036865    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e1a828b8-77fa-4048-9536-9681a05cbb9b" containerName="kserve-container"
May 25 12:03:24.036943 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:03:24.036877    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="e1a828b8-77fa-4048-9536-9681a05cbb9b" containerName="kserve-container"
May 25 12:03:24.036943 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:03:24.036890    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e1a828b8-77fa-4048-9536-9681a05cbb9b" containerName="kube-rbac-proxy"
May 25 12:03:24.036943 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:03:24.036895    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="e1a828b8-77fa-4048-9536-9681a05cbb9b" containerName="kube-rbac-proxy"
May 25 12:03:24.036943 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:03:24.036908    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e1a828b8-77fa-4048-9536-9681a05cbb9b" containerName="storage-initializer"
May 25 12:03:24.036943 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:03:24.036914    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="e1a828b8-77fa-4048-9536-9681a05cbb9b" containerName="storage-initializer"
May 25 12:03:24.037102 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:03:24.036977    2561 memory_manager.go:356] "RemoveStaleState removing state" podUID="e1a828b8-77fa-4048-9536-9681a05cbb9b" containerName="kube-rbac-proxy"
May 25 12:03:24.037102 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:03:24.036989    2561 memory_manager.go:356] "RemoveStaleState removing state" podUID="e1a828b8-77fa-4048-9536-9681a05cbb9b" containerName="kserve-container"
May 25 12:03:24.040310 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:03:24.040290    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6bcc6c87db-pmg7w"
May 25 12:03:24.042753 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:03:24.042734    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config\""
May 25 12:03:24.042934 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:03:24.042921    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-mixed-predictor-serving-cert\""
May 25 12:03:24.052030 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:03:24.052012    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6bcc6c87db-pmg7w"]
May 25 12:03:24.169953 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:03:24.169911    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8fc95c62-9897-45b5-9e29-45ffef56d91c-kserve-provision-location\") pod \"isvc-sklearn-v2-mixed-predictor-6bcc6c87db-pmg7w\" (UID: \"8fc95c62-9897-45b5-9e29-45ffef56d91c\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6bcc6c87db-pmg7w"
May 25 12:03:24.170099 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:03:24.170023    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8fc95c62-9897-45b5-9e29-45ffef56d91c-isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-mixed-predictor-6bcc6c87db-pmg7w\" (UID: \"8fc95c62-9897-45b5-9e29-45ffef56d91c\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6bcc6c87db-pmg7w"
May 25 12:03:24.170099 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:03:24.170067    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8fc95c62-9897-45b5-9e29-45ffef56d91c-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-6bcc6c87db-pmg7w\" (UID: \"8fc95c62-9897-45b5-9e29-45ffef56d91c\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6bcc6c87db-pmg7w"
May 25 12:03:24.170210 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:03:24.170104    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtvd4\" (UniqueName: \"kubernetes.io/projected/8fc95c62-9897-45b5-9e29-45ffef56d91c-kube-api-access-qtvd4\") pod \"isvc-sklearn-v2-mixed-predictor-6bcc6c87db-pmg7w\" (UID: \"8fc95c62-9897-45b5-9e29-45ffef56d91c\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6bcc6c87db-pmg7w"
May 25 12:03:24.270676 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:03:24.270612    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8fc95c62-9897-45b5-9e29-45ffef56d91c-kserve-provision-location\") pod \"isvc-sklearn-v2-mixed-predictor-6bcc6c87db-pmg7w\" (UID: \"8fc95c62-9897-45b5-9e29-45ffef56d91c\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6bcc6c87db-pmg7w"
May 25 12:03:24.270794 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:03:24.270684    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8fc95c62-9897-45b5-9e29-45ffef56d91c-isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-mixed-predictor-6bcc6c87db-pmg7w\" (UID: \"8fc95c62-9897-45b5-9e29-45ffef56d91c\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6bcc6c87db-pmg7w"
May 25 12:03:24.270794 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:03:24.270719    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8fc95c62-9897-45b5-9e29-45ffef56d91c-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-6bcc6c87db-pmg7w\" (UID: \"8fc95c62-9897-45b5-9e29-45ffef56d91c\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6bcc6c87db-pmg7w"
May 25 12:03:24.270794 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:03:24.270756    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-qtvd4\" (UniqueName: \"kubernetes.io/projected/8fc95c62-9897-45b5-9e29-45ffef56d91c-kube-api-access-qtvd4\") pod \"isvc-sklearn-v2-mixed-predictor-6bcc6c87db-pmg7w\" (UID: \"8fc95c62-9897-45b5-9e29-45ffef56d91c\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6bcc6c87db-pmg7w"
May 25 12:03:24.270962 ip-10-0-136-227 kubenswrapper[2561]: E0525 12:03:24.270862    2561 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-serving-cert: secret "isvc-sklearn-v2-mixed-predictor-serving-cert" not found
May 25 12:03:24.270962 ip-10-0-136-227 kubenswrapper[2561]: E0525 12:03:24.270930    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8fc95c62-9897-45b5-9e29-45ffef56d91c-proxy-tls podName:8fc95c62-9897-45b5-9e29-45ffef56d91c nodeName:}" failed. No retries permitted until 2026-05-25 12:03:24.77090871 +0000 UTC m=+2778.316086296 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/8fc95c62-9897-45b5-9e29-45ffef56d91c-proxy-tls") pod "isvc-sklearn-v2-mixed-predictor-6bcc6c87db-pmg7w" (UID: "8fc95c62-9897-45b5-9e29-45ffef56d91c") : secret "isvc-sklearn-v2-mixed-predictor-serving-cert" not found
May 25 12:03:24.271079 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:03:24.270997    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8fc95c62-9897-45b5-9e29-45ffef56d91c-kserve-provision-location\") pod \"isvc-sklearn-v2-mixed-predictor-6bcc6c87db-pmg7w\" (UID: \"8fc95c62-9897-45b5-9e29-45ffef56d91c\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6bcc6c87db-pmg7w"
May 25 12:03:24.271264 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:03:24.271245    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8fc95c62-9897-45b5-9e29-45ffef56d91c-isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-mixed-predictor-6bcc6c87db-pmg7w\" (UID: \"8fc95c62-9897-45b5-9e29-45ffef56d91c\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6bcc6c87db-pmg7w"
May 25 12:03:24.280030 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:03:24.280007    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtvd4\" (UniqueName: \"kubernetes.io/projected/8fc95c62-9897-45b5-9e29-45ffef56d91c-kube-api-access-qtvd4\") pod \"isvc-sklearn-v2-mixed-predictor-6bcc6c87db-pmg7w\" (UID: \"8fc95c62-9897-45b5-9e29-45ffef56d91c\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6bcc6c87db-pmg7w"
May 25 12:03:24.734833 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:03:24.734802    2561 generic.go:358] "Generic (PLEG): container finished" podID="ca1b946e-29ff-4f5c-966b-764da5e4a57e" containerID="1a03ecc28a11c4f2d2b1d5f68484a1f9a57bb43416a8c3217bcc1c074684416c" exitCode=2
May 25 12:03:24.734983 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:03:24.734872    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-58dfb5f9bc-hr4n2" event={"ID":"ca1b946e-29ff-4f5c-966b-764da5e4a57e","Type":"ContainerDied","Data":"1a03ecc28a11c4f2d2b1d5f68484a1f9a57bb43416a8c3217bcc1c074684416c"}
May 25 12:03:24.774869 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:03:24.774848    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8fc95c62-9897-45b5-9e29-45ffef56d91c-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-6bcc6c87db-pmg7w\" (UID: \"8fc95c62-9897-45b5-9e29-45ffef56d91c\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6bcc6c87db-pmg7w"
May 25 12:03:24.777397 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:03:24.777372    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8fc95c62-9897-45b5-9e29-45ffef56d91c-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-6bcc6c87db-pmg7w\" (UID: \"8fc95c62-9897-45b5-9e29-45ffef56d91c\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6bcc6c87db-pmg7w"
May 25 12:03:24.951003 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:03:24.950973    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6bcc6c87db-pmg7w"
May 25 12:03:25.077820 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:03:25.077790    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6bcc6c87db-pmg7w"]
May 25 12:03:25.080771 ip-10-0-136-227 kubenswrapper[2561]: W0525 12:03:25.080745    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fc95c62_9897_45b5_9e29_45ffef56d91c.slice/crio-51b0b3509bd692bb909e831fa310ff527db20550e190d22a35234df469763681 WatchSource:0}: Error finding container 51b0b3509bd692bb909e831fa310ff527db20550e190d22a35234df469763681: Status 404 returned error can't find the container with id 51b0b3509bd692bb909e831fa310ff527db20550e190d22a35234df469763681
May 25 12:03:25.082864 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:03:25.082845    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 25 12:03:25.739127 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:03:25.739095    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6bcc6c87db-pmg7w" event={"ID":"8fc95c62-9897-45b5-9e29-45ffef56d91c","Type":"ContainerStarted","Data":"c5ed700803c629a2f1c4ababff03d8a8d6773fa8a0c60d8b82f14121003ebc11"}
May 25 12:03:25.739127 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:03:25.739128    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6bcc6c87db-pmg7w" event={"ID":"8fc95c62-9897-45b5-9e29-45ffef56d91c","Type":"ContainerStarted","Data":"51b0b3509bd692bb909e831fa310ff527db20550e190d22a35234df469763681"}
May 25 12:03:26.493253 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:03:26.493159    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-58dfb5f9bc-hr4n2" podUID="ca1b946e-29ff-4f5c-966b-764da5e4a57e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.51:8643/healthz\": dial tcp 10.132.0.51:8643: connect: connection refused"
May 25 12:03:26.498500 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:03:26.498476    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-58dfb5f9bc-hr4n2" podUID="ca1b946e-29ff-4f5c-966b-764da5e4a57e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.51:8080: connect: connection refused"
May 25 12:03:28.008657 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:03:28.008626    2561 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-58dfb5f9bc-hr4n2"
May 25 12:03:28.103008 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:03:28.102974    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ca1b946e-29ff-4f5c-966b-764da5e4a57e-proxy-tls\") pod \"ca1b946e-29ff-4f5c-966b-764da5e4a57e\" (UID: \"ca1b946e-29ff-4f5c-966b-764da5e4a57e\") "
May 25 12:03:28.103177 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:03:28.103035    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-sklearn-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ca1b946e-29ff-4f5c-966b-764da5e4a57e-isvc-sklearn-v2-kube-rbac-proxy-sar-config\") pod \"ca1b946e-29ff-4f5c-966b-764da5e4a57e\" (UID: \"ca1b946e-29ff-4f5c-966b-764da5e4a57e\") "
May 25 12:03:28.103177 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:03:28.103090    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ca1b946e-29ff-4f5c-966b-764da5e4a57e-kserve-provision-location\") pod \"ca1b946e-29ff-4f5c-966b-764da5e4a57e\" (UID: \"ca1b946e-29ff-4f5c-966b-764da5e4a57e\") "
May 25 12:03:28.103177 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:03:28.103143    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k7bwt\" (UniqueName: \"kubernetes.io/projected/ca1b946e-29ff-4f5c-966b-764da5e4a57e-kube-api-access-k7bwt\") pod \"ca1b946e-29ff-4f5c-966b-764da5e4a57e\" (UID: \"ca1b946e-29ff-4f5c-966b-764da5e4a57e\") "
May 25 12:03:28.103440 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:03:28.103409    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca1b946e-29ff-4f5c-966b-764da5e4a57e-isvc-sklearn-v2-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-sklearn-v2-kube-rbac-proxy-sar-config") pod "ca1b946e-29ff-4f5c-966b-764da5e4a57e" (UID: "ca1b946e-29ff-4f5c-966b-764da5e4a57e"). InnerVolumeSpecName "isvc-sklearn-v2-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 12:03:28.103529 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:03:28.103446    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca1b946e-29ff-4f5c-966b-764da5e4a57e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ca1b946e-29ff-4f5c-966b-764da5e4a57e" (UID: "ca1b946e-29ff-4f5c-966b-764da5e4a57e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 12:03:28.105154 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:03:28.105123    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca1b946e-29ff-4f5c-966b-764da5e4a57e-kube-api-access-k7bwt" (OuterVolumeSpecName: "kube-api-access-k7bwt") pod "ca1b946e-29ff-4f5c-966b-764da5e4a57e" (UID: "ca1b946e-29ff-4f5c-966b-764da5e4a57e"). InnerVolumeSpecName "kube-api-access-k7bwt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 12:03:28.105259 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:03:28.105205    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca1b946e-29ff-4f5c-966b-764da5e4a57e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ca1b946e-29ff-4f5c-966b-764da5e4a57e" (UID: "ca1b946e-29ff-4f5c-966b-764da5e4a57e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 12:03:28.205032 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:03:28.204999    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k7bwt\" (UniqueName: \"kubernetes.io/projected/ca1b946e-29ff-4f5c-966b-764da5e4a57e-kube-api-access-k7bwt\") on node \"ip-10-0-136-227.ec2.internal\" DevicePath \"\""
May 25 12:03:28.205032 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:03:28.205028    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ca1b946e-29ff-4f5c-966b-764da5e4a57e-proxy-tls\") on node \"ip-10-0-136-227.ec2.internal\" DevicePath \"\""
May 25 12:03:28.205206 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:03:28.205043    2561 reconciler_common.go:299] "Volume detached for volume \"isvc-sklearn-v2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ca1b946e-29ff-4f5c-966b-764da5e4a57e-isvc-sklearn-v2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-227.ec2.internal\" DevicePath \"\""
May 25 12:03:28.205206 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:03:28.205059    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ca1b946e-29ff-4f5c-966b-764da5e4a57e-kserve-provision-location\") on node \"ip-10-0-136-227.ec2.internal\" DevicePath \"\""
May 25 12:03:28.750724 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:03:28.750686    2561 generic.go:358] "Generic (PLEG): container finished" podID="ca1b946e-29ff-4f5c-966b-764da5e4a57e" containerID="76b8ccd9aa605f963bc90a89a84c57ba893d8bcead68eb89dcf1b66af6381351" exitCode=0
May 25 12:03:28.750896 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:03:28.750741    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-58dfb5f9bc-hr4n2" event={"ID":"ca1b946e-29ff-4f5c-966b-764da5e4a57e","Type":"ContainerDied","Data":"76b8ccd9aa605f963bc90a89a84c57ba893d8bcead68eb89dcf1b66af6381351"}
May 25 12:03:28.750896 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:03:28.750784    2561 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-58dfb5f9bc-hr4n2"
May 25 12:03:28.750896 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:03:28.750799    2561 scope.go:117] "RemoveContainer" containerID="1a03ecc28a11c4f2d2b1d5f68484a1f9a57bb43416a8c3217bcc1c074684416c"
May 25 12:03:28.751081 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:03:28.750788    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-58dfb5f9bc-hr4n2" event={"ID":"ca1b946e-29ff-4f5c-966b-764da5e4a57e","Type":"ContainerDied","Data":"a2747d783ea7f81ea33a13d578e173312aabffc59b721f7c14309671670046fd"}
May 25 12:03:28.760164 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:03:28.760146    2561 scope.go:117] "RemoveContainer" containerID="76b8ccd9aa605f963bc90a89a84c57ba893d8bcead68eb89dcf1b66af6381351"
May 25 12:03:28.767100 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:03:28.767086    2561 scope.go:117] "RemoveContainer" containerID="ebb96a500dd1f139aca7224086fb69d17c1f1cbb7a6d1a5b874ec0391bb423f4"
May 25 12:03:28.774310 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:03:28.774292    2561 scope.go:117] "RemoveContainer" containerID="1a03ecc28a11c4f2d2b1d5f68484a1f9a57bb43416a8c3217bcc1c074684416c"
May 25 12:03:28.774633 ip-10-0-136-227 kubenswrapper[2561]: E0525 12:03:28.774608    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a03ecc28a11c4f2d2b1d5f68484a1f9a57bb43416a8c3217bcc1c074684416c\": container with ID starting with 1a03ecc28a11c4f2d2b1d5f68484a1f9a57bb43416a8c3217bcc1c074684416c not found: ID does not exist" containerID="1a03ecc28a11c4f2d2b1d5f68484a1f9a57bb43416a8c3217bcc1c074684416c"
May 25 12:03:28.774727 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:03:28.774640    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a03ecc28a11c4f2d2b1d5f68484a1f9a57bb43416a8c3217bcc1c074684416c"} err="failed to get container status \"1a03ecc28a11c4f2d2b1d5f68484a1f9a57bb43416a8c3217bcc1c074684416c\": rpc error: code = NotFound desc = could not find container \"1a03ecc28a11c4f2d2b1d5f68484a1f9a57bb43416a8c3217bcc1c074684416c\": container with ID starting with 1a03ecc28a11c4f2d2b1d5f68484a1f9a57bb43416a8c3217bcc1c074684416c not found: ID does not exist"
May 25 12:03:28.774727 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:03:28.774658    2561 scope.go:117] "RemoveContainer" containerID="76b8ccd9aa605f963bc90a89a84c57ba893d8bcead68eb89dcf1b66af6381351"
May 25 12:03:28.774927 ip-10-0-136-227 kubenswrapper[2561]: E0525 12:03:28.774910    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76b8ccd9aa605f963bc90a89a84c57ba893d8bcead68eb89dcf1b66af6381351\": container with ID starting with 76b8ccd9aa605f963bc90a89a84c57ba893d8bcead68eb89dcf1b66af6381351 not found: ID does not exist" containerID="76b8ccd9aa605f963bc90a89a84c57ba893d8bcead68eb89dcf1b66af6381351"
May 25 12:03:28.774996 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:03:28.774937    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76b8ccd9aa605f963bc90a89a84c57ba893d8bcead68eb89dcf1b66af6381351"} err="failed to get container status \"76b8ccd9aa605f963bc90a89a84c57ba893d8bcead68eb89dcf1b66af6381351\": rpc error: code = NotFound desc = could not find container \"76b8ccd9aa605f963bc90a89a84c57ba893d8bcead68eb89dcf1b66af6381351\": container with ID starting with 76b8ccd9aa605f963bc90a89a84c57ba893d8bcead68eb89dcf1b66af6381351 not found: ID does not exist"
May 25 12:03:28.774996 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:03:28.774956    2561 scope.go:117] "RemoveContainer" containerID="ebb96a500dd1f139aca7224086fb69d17c1f1cbb7a6d1a5b874ec0391bb423f4"
May 25 12:03:28.775219 ip-10-0-136-227 kubenswrapper[2561]: E0525 12:03:28.775201    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ebb96a500dd1f139aca7224086fb69d17c1f1cbb7a6d1a5b874ec0391bb423f4\": container with ID starting with ebb96a500dd1f139aca7224086fb69d17c1f1cbb7a6d1a5b874ec0391bb423f4 not found: ID does not exist" containerID="ebb96a500dd1f139aca7224086fb69d17c1f1cbb7a6d1a5b874ec0391bb423f4"
May 25 12:03:28.775320 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:03:28.775222    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebb96a500dd1f139aca7224086fb69d17c1f1cbb7a6d1a5b874ec0391bb423f4"} err="failed to get container status \"ebb96a500dd1f139aca7224086fb69d17c1f1cbb7a6d1a5b874ec0391bb423f4\": rpc error: code = NotFound desc = could not find container \"ebb96a500dd1f139aca7224086fb69d17c1f1cbb7a6d1a5b874ec0391bb423f4\": container with ID starting with ebb96a500dd1f139aca7224086fb69d17c1f1cbb7a6d1a5b874ec0391bb423f4 not found: ID does not exist"
May 25 12:03:28.776368 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:03:28.776349    2561 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-58dfb5f9bc-hr4n2"]
May 25 12:03:28.781488 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:03:28.781467    2561 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-58dfb5f9bc-hr4n2"]
May 25 12:03:29.116987 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:03:29.116959    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ca1b946e-29ff-4f5c-966b-764da5e4a57e" path="/var/lib/kubelet/pods/ca1b946e-29ff-4f5c-966b-764da5e4a57e/volumes"
May 25 12:03:29.756600 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:03:29.756566    2561 generic.go:358] "Generic (PLEG): container finished" podID="8fc95c62-9897-45b5-9e29-45ffef56d91c" containerID="c5ed700803c629a2f1c4ababff03d8a8d6773fa8a0c60d8b82f14121003ebc11" exitCode=0
May 25 12:03:29.756770 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:03:29.756603    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6bcc6c87db-pmg7w" event={"ID":"8fc95c62-9897-45b5-9e29-45ffef56d91c","Type":"ContainerDied","Data":"c5ed700803c629a2f1c4ababff03d8a8d6773fa8a0c60d8b82f14121003ebc11"}
May 25 12:03:30.761848 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:03:30.761817    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6bcc6c87db-pmg7w" event={"ID":"8fc95c62-9897-45b5-9e29-45ffef56d91c","Type":"ContainerStarted","Data":"2558d3f8512f52f8da2ab06cd1215c35a03a69aa2a73f857a4a8ff48c2fbe8f4"}
May 25 12:03:30.761848 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:03:30.761850    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6bcc6c87db-pmg7w" event={"ID":"8fc95c62-9897-45b5-9e29-45ffef56d91c","Type":"ContainerStarted","Data":"4bc0775a133f09078e6ffc286ccdb61de441a6d7e768f3eb4735f642c6b36694"}
May 25 12:03:30.762416 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:03:30.762134    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6bcc6c87db-pmg7w"
May 25 12:03:30.762416 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:03:30.762256    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6bcc6c87db-pmg7w"
May 25 12:03:30.763513 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:03:30.763481    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6bcc6c87db-pmg7w" podUID="8fc95c62-9897-45b5-9e29-45ffef56d91c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.52:8080: connect: connection refused"
May 25 12:03:30.781989 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:03:30.781947    2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6bcc6c87db-pmg7w" podStartSLOduration=6.781936227 podStartE2EDuration="6.781936227s" podCreationTimestamp="2026-05-25 12:03:24 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 12:03:30.779636717 +0000 UTC m=+2784.324814335" watchObservedRunningTime="2026-05-25 12:03:30.781936227 +0000 UTC m=+2784.327113830"
May 25 12:03:31.765652 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:03:31.765614    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6bcc6c87db-pmg7w" podUID="8fc95c62-9897-45b5-9e29-45ffef56d91c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.52:8080: connect: connection refused"
May 25 12:03:36.770743 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:03:36.770714    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6bcc6c87db-pmg7w"
May 25 12:03:36.771341 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:03:36.771315    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6bcc6c87db-pmg7w" podUID="8fc95c62-9897-45b5-9e29-45ffef56d91c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.52:8080: connect: connection refused"
May 25 12:03:46.771906 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:03:46.771865    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6bcc6c87db-pmg7w" podUID="8fc95c62-9897-45b5-9e29-45ffef56d91c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.52:8080: connect: connection refused"
May 25 12:03:56.771925 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:03:56.771885    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6bcc6c87db-pmg7w" podUID="8fc95c62-9897-45b5-9e29-45ffef56d91c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.52:8080: connect: connection refused"
May 25 12:04:06.771454 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:04:06.771417    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6bcc6c87db-pmg7w" podUID="8fc95c62-9897-45b5-9e29-45ffef56d91c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.52:8080: connect: connection refused"
May 25 12:04:16.771589 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:04:16.771549    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6bcc6c87db-pmg7w" podUID="8fc95c62-9897-45b5-9e29-45ffef56d91c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.52:8080: connect: connection refused"
May 25 12:04:26.771665 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:04:26.771625    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6bcc6c87db-pmg7w" podUID="8fc95c62-9897-45b5-9e29-45ffef56d91c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.52:8080: connect: connection refused"
May 25 12:04:36.772746 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:04:36.772716    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6bcc6c87db-pmg7w"
May 25 12:04:44.131970 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:04:44.131934    2561 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6bcc6c87db-pmg7w"]
May 25 12:04:44.132552 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:04:44.132251    2561 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6bcc6c87db-pmg7w" podUID="8fc95c62-9897-45b5-9e29-45ffef56d91c" containerName="kserve-container" containerID="cri-o://4bc0775a133f09078e6ffc286ccdb61de441a6d7e768f3eb4735f642c6b36694" gracePeriod=30
May 25 12:04:44.132552 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:04:44.132361    2561 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6bcc6c87db-pmg7w" podUID="8fc95c62-9897-45b5-9e29-45ffef56d91c" containerName="kube-rbac-proxy" containerID="cri-o://2558d3f8512f52f8da2ab06cd1215c35a03a69aa2a73f857a4a8ff48c2fbe8f4" gracePeriod=30
May 25 12:04:45.012636 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:04:45.012598    2561 generic.go:358] "Generic (PLEG): container finished" podID="8fc95c62-9897-45b5-9e29-45ffef56d91c" containerID="2558d3f8512f52f8da2ab06cd1215c35a03a69aa2a73f857a4a8ff48c2fbe8f4" exitCode=2
May 25 12:04:45.012805 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:04:45.012675    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6bcc6c87db-pmg7w" event={"ID":"8fc95c62-9897-45b5-9e29-45ffef56d91c","Type":"ContainerDied","Data":"2558d3f8512f52f8da2ab06cd1215c35a03a69aa2a73f857a4a8ff48c2fbe8f4"}
May 25 12:04:46.766203 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:04:46.766159    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6bcc6c87db-pmg7w" podUID="8fc95c62-9897-45b5-9e29-45ffef56d91c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.52:8643/healthz\": dial tcp 10.132.0.52:8643: connect: connection refused"
May 25 12:04:46.771523 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:04:46.771494    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6bcc6c87db-pmg7w" podUID="8fc95c62-9897-45b5-9e29-45ffef56d91c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.52:8080: connect: connection refused"
May 25 12:04:48.025036 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:04:48.025008    2561 generic.go:358] "Generic (PLEG): container finished" podID="8fc95c62-9897-45b5-9e29-45ffef56d91c" containerID="4bc0775a133f09078e6ffc286ccdb61de441a6d7e768f3eb4735f642c6b36694" exitCode=0
May 25 12:04:48.025378 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:04:48.025080    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6bcc6c87db-pmg7w" event={"ID":"8fc95c62-9897-45b5-9e29-45ffef56d91c","Type":"ContainerDied","Data":"4bc0775a133f09078e6ffc286ccdb61de441a6d7e768f3eb4735f642c6b36694"}
May 25 12:04:48.082472 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:04:48.082451    2561 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6bcc6c87db-pmg7w"
May 25 12:04:48.247818 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:04:48.247753    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qtvd4\" (UniqueName: \"kubernetes.io/projected/8fc95c62-9897-45b5-9e29-45ffef56d91c-kube-api-access-qtvd4\") pod \"8fc95c62-9897-45b5-9e29-45ffef56d91c\" (UID: \"8fc95c62-9897-45b5-9e29-45ffef56d91c\") "
May 25 12:04:48.247818 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:04:48.247792    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8fc95c62-9897-45b5-9e29-45ffef56d91c-proxy-tls\") pod \"8fc95c62-9897-45b5-9e29-45ffef56d91c\" (UID: \"8fc95c62-9897-45b5-9e29-45ffef56d91c\") "
May 25 12:04:48.247982 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:04:48.247833    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8fc95c62-9897-45b5-9e29-45ffef56d91c-isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config\") pod \"8fc95c62-9897-45b5-9e29-45ffef56d91c\" (UID: \"8fc95c62-9897-45b5-9e29-45ffef56d91c\") "
May 25 12:04:48.247982 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:04:48.247857    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8fc95c62-9897-45b5-9e29-45ffef56d91c-kserve-provision-location\") pod \"8fc95c62-9897-45b5-9e29-45ffef56d91c\" (UID: \"8fc95c62-9897-45b5-9e29-45ffef56d91c\") "
May 25 12:04:48.248242 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:04:48.248198    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8fc95c62-9897-45b5-9e29-45ffef56d91c-isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config") pod "8fc95c62-9897-45b5-9e29-45ffef56d91c" (UID: "8fc95c62-9897-45b5-9e29-45ffef56d91c"). InnerVolumeSpecName "isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 12:04:48.248242 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:04:48.248225    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fc95c62-9897-45b5-9e29-45ffef56d91c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8fc95c62-9897-45b5-9e29-45ffef56d91c" (UID: "8fc95c62-9897-45b5-9e29-45ffef56d91c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 12:04:48.249845 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:04:48.249825    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8fc95c62-9897-45b5-9e29-45ffef56d91c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8fc95c62-9897-45b5-9e29-45ffef56d91c" (UID: "8fc95c62-9897-45b5-9e29-45ffef56d91c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 12:04:48.249845 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:04:48.249836    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fc95c62-9897-45b5-9e29-45ffef56d91c-kube-api-access-qtvd4" (OuterVolumeSpecName: "kube-api-access-qtvd4") pod "8fc95c62-9897-45b5-9e29-45ffef56d91c" (UID: "8fc95c62-9897-45b5-9e29-45ffef56d91c"). InnerVolumeSpecName "kube-api-access-qtvd4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 12:04:48.348539 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:04:48.348516    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qtvd4\" (UniqueName: \"kubernetes.io/projected/8fc95c62-9897-45b5-9e29-45ffef56d91c-kube-api-access-qtvd4\") on node \"ip-10-0-136-227.ec2.internal\" DevicePath \"\""
May 25 12:04:48.348539 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:04:48.348537    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8fc95c62-9897-45b5-9e29-45ffef56d91c-proxy-tls\") on node \"ip-10-0-136-227.ec2.internal\" DevicePath \"\""
May 25 12:04:48.348683 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:04:48.348547    2561 reconciler_common.go:299] "Volume detached for volume \"isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8fc95c62-9897-45b5-9e29-45ffef56d91c-isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-227.ec2.internal\" DevicePath \"\""
May 25 12:04:48.348683 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:04:48.348557    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8fc95c62-9897-45b5-9e29-45ffef56d91c-kserve-provision-location\") on node \"ip-10-0-136-227.ec2.internal\" DevicePath \"\""
May 25 12:04:49.029617 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:04:49.029585    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6bcc6c87db-pmg7w" event={"ID":"8fc95c62-9897-45b5-9e29-45ffef56d91c","Type":"ContainerDied","Data":"51b0b3509bd692bb909e831fa310ff527db20550e190d22a35234df469763681"}
May 25 12:04:49.030038 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:04:49.029630    2561 scope.go:117] "RemoveContainer" containerID="2558d3f8512f52f8da2ab06cd1215c35a03a69aa2a73f857a4a8ff48c2fbe8f4"
May 25 12:04:49.030038 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:04:49.029657    2561 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6bcc6c87db-pmg7w"
May 25 12:04:49.038850 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:04:49.038595    2561 scope.go:117] "RemoveContainer" containerID="4bc0775a133f09078e6ffc286ccdb61de441a6d7e768f3eb4735f642c6b36694"
May 25 12:04:49.047715 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:04:49.047694    2561 scope.go:117] "RemoveContainer" containerID="c5ed700803c629a2f1c4ababff03d8a8d6773fa8a0c60d8b82f14121003ebc11"
May 25 12:04:49.053671 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:04:49.053648    2561 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6bcc6c87db-pmg7w"]
May 25 12:04:49.058237 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:04:49.058216    2561 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-6bcc6c87db-pmg7w"]
May 25 12:04:49.117083 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:04:49.117059    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8fc95c62-9897-45b5-9e29-45ffef56d91c" path="/var/lib/kubelet/pods/8fc95c62-9897-45b5-9e29-45ffef56d91c/volumes"
May 25 12:07:07.218378 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:07:07.218301    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-9lj95_10a66344-3304-49a1-bbc1-b885800f5acf/console-operator/1.log"
May 25 12:07:07.223760 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:07:07.223739    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-9lj95_10a66344-3304-49a1-bbc1-b885800f5acf/console-operator/1.log"
May 25 12:09:58.051981 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:09:58.051901    2561 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-zvqkl"]
May 25 12:09:58.052496 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:09:58.052443    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca1b946e-29ff-4f5c-966b-764da5e4a57e" containerName="storage-initializer"
May 25 12:09:58.052496 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:09:58.052462    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca1b946e-29ff-4f5c-966b-764da5e4a57e" containerName="storage-initializer"
May 25 12:09:58.052496 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:09:58.052472    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8fc95c62-9897-45b5-9e29-45ffef56d91c" containerName="kube-rbac-proxy"
May 25 12:09:58.052496 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:09:58.052480    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="8fc95c62-9897-45b5-9e29-45ffef56d91c" containerName="kube-rbac-proxy"
May 25 12:09:58.052496 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:09:58.052493    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8fc95c62-9897-45b5-9e29-45ffef56d91c" containerName="storage-initializer"
May 25 12:09:58.052496 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:09:58.052499    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="8fc95c62-9897-45b5-9e29-45ffef56d91c" containerName="storage-initializer"
May 25 12:09:58.052742 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:09:58.052509    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8fc95c62-9897-45b5-9e29-45ffef56d91c" containerName="kserve-container"
May 25 12:09:58.052742 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:09:58.052514    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="8fc95c62-9897-45b5-9e29-45ffef56d91c" containerName="kserve-container"
May 25 12:09:58.052742 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:09:58.052523    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca1b946e-29ff-4f5c-966b-764da5e4a57e" containerName="kube-rbac-proxy"
May 25 12:09:58.052742 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:09:58.052532    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca1b946e-29ff-4f5c-966b-764da5e4a57e" containerName="kube-rbac-proxy"
May 25 12:09:58.052742 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:09:58.052543    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca1b946e-29ff-4f5c-966b-764da5e4a57e" containerName="kserve-container"
May 25 12:09:58.052742 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:09:58.052548    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca1b946e-29ff-4f5c-966b-764da5e4a57e" containerName="kserve-container"
May 25 12:09:58.052742 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:09:58.052612    2561 memory_manager.go:356] "RemoveStaleState removing state" podUID="ca1b946e-29ff-4f5c-966b-764da5e4a57e" containerName="kube-rbac-proxy"
May 25 12:09:58.052742 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:09:58.052623    2561 memory_manager.go:356] "RemoveStaleState removing state" podUID="8fc95c62-9897-45b5-9e29-45ffef56d91c" containerName="kube-rbac-proxy"
May 25 12:09:58.052742 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:09:58.052630    2561 memory_manager.go:356] "RemoveStaleState removing state" podUID="ca1b946e-29ff-4f5c-966b-764da5e4a57e" containerName="kserve-container"
May 25 12:09:58.052742 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:09:58.052637    2561 memory_manager.go:356] "RemoveStaleState removing state" podUID="8fc95c62-9897-45b5-9e29-45ffef56d91c" containerName="kserve-container"
May 25 12:09:58.060004 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:09:58.059977    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-zvqkl"
May 25 12:09:58.062644 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:09:58.062610    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\""
May 25 12:09:58.062788 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:09:58.062710    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
May 25 12:09:58.062788 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:09:58.062715    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
May 25 12:09:58.062989 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:09:58.062975    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-mlserver-predictor-serving-cert\""
May 25 12:09:58.063051 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:09:58.063013    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-svs62\""
May 25 12:09:58.067017 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:09:58.066996    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-zvqkl"]
May 25 12:09:58.098505 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:09:58.098477    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ea8e3e57-237e-4f7b-926c-c554dd3c0da0-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-67d4bc6646-zvqkl\" (UID: \"ea8e3e57-237e-4f7b-926c-c554dd3c0da0\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-zvqkl"
May 25 12:09:58.098646 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:09:58.098514    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ea8e3e57-237e-4f7b-926c-c554dd3c0da0-kserve-provision-location\") pod \"isvc-xgboost-v2-mlserver-predictor-67d4bc6646-zvqkl\" (UID: \"ea8e3e57-237e-4f7b-926c-c554dd3c0da0\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-zvqkl"
May 25 12:09:58.098646 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:09:58.098547    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ea8e3e57-237e-4f7b-926c-c554dd3c0da0-isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-mlserver-predictor-67d4bc6646-zvqkl\" (UID: \"ea8e3e57-237e-4f7b-926c-c554dd3c0da0\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-zvqkl"
May 25 12:09:58.098646 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:09:58.098592    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5v8z\" (UniqueName: \"kubernetes.io/projected/ea8e3e57-237e-4f7b-926c-c554dd3c0da0-kube-api-access-s5v8z\") pod \"isvc-xgboost-v2-mlserver-predictor-67d4bc6646-zvqkl\" (UID: \"ea8e3e57-237e-4f7b-926c-c554dd3c0da0\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-zvqkl"
May 25 12:09:58.199419 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:09:58.199377    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ea8e3e57-237e-4f7b-926c-c554dd3c0da0-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-67d4bc6646-zvqkl\" (UID: \"ea8e3e57-237e-4f7b-926c-c554dd3c0da0\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-zvqkl"
May 25 12:09:58.199419 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:09:58.199418    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ea8e3e57-237e-4f7b-926c-c554dd3c0da0-kserve-provision-location\") pod \"isvc-xgboost-v2-mlserver-predictor-67d4bc6646-zvqkl\" (UID: \"ea8e3e57-237e-4f7b-926c-c554dd3c0da0\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-zvqkl"
May 25 12:09:58.199636 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:09:58.199441    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ea8e3e57-237e-4f7b-926c-c554dd3c0da0-isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-mlserver-predictor-67d4bc6646-zvqkl\" (UID: \"ea8e3e57-237e-4f7b-926c-c554dd3c0da0\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-zvqkl"
May 25 12:09:58.199636 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:09:58.199472    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-s5v8z\" (UniqueName: \"kubernetes.io/projected/ea8e3e57-237e-4f7b-926c-c554dd3c0da0-kube-api-access-s5v8z\") pod \"isvc-xgboost-v2-mlserver-predictor-67d4bc6646-zvqkl\" (UID: \"ea8e3e57-237e-4f7b-926c-c554dd3c0da0\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-zvqkl"
May 25 12:09:58.199636 ip-10-0-136-227 kubenswrapper[2561]: E0525 12:09:58.199526    2561 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-serving-cert: secret "isvc-xgboost-v2-mlserver-predictor-serving-cert" not found
May 25 12:09:58.199636 ip-10-0-136-227 kubenswrapper[2561]: E0525 12:09:58.199602    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ea8e3e57-237e-4f7b-926c-c554dd3c0da0-proxy-tls podName:ea8e3e57-237e-4f7b-926c-c554dd3c0da0 nodeName:}" failed. No retries permitted until 2026-05-25 12:09:58.699586098 +0000 UTC m=+3172.244763681 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/ea8e3e57-237e-4f7b-926c-c554dd3c0da0-proxy-tls") pod "isvc-xgboost-v2-mlserver-predictor-67d4bc6646-zvqkl" (UID: "ea8e3e57-237e-4f7b-926c-c554dd3c0da0") : secret "isvc-xgboost-v2-mlserver-predictor-serving-cert" not found
May 25 12:09:58.199869 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:09:58.199849    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ea8e3e57-237e-4f7b-926c-c554dd3c0da0-kserve-provision-location\") pod \"isvc-xgboost-v2-mlserver-predictor-67d4bc6646-zvqkl\" (UID: \"ea8e3e57-237e-4f7b-926c-c554dd3c0da0\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-zvqkl"
May 25 12:09:58.200072 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:09:58.200056    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ea8e3e57-237e-4f7b-926c-c554dd3c0da0-isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-mlserver-predictor-67d4bc6646-zvqkl\" (UID: \"ea8e3e57-237e-4f7b-926c-c554dd3c0da0\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-zvqkl"
May 25 12:09:58.208794 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:09:58.208763    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5v8z\" (UniqueName: \"kubernetes.io/projected/ea8e3e57-237e-4f7b-926c-c554dd3c0da0-kube-api-access-s5v8z\") pod \"isvc-xgboost-v2-mlserver-predictor-67d4bc6646-zvqkl\" (UID: \"ea8e3e57-237e-4f7b-926c-c554dd3c0da0\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-zvqkl"
May 25 12:09:58.702114 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:09:58.702085    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ea8e3e57-237e-4f7b-926c-c554dd3c0da0-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-67d4bc6646-zvqkl\" (UID: \"ea8e3e57-237e-4f7b-926c-c554dd3c0da0\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-zvqkl"
May 25 12:09:58.704612 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:09:58.704579    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ea8e3e57-237e-4f7b-926c-c554dd3c0da0-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-67d4bc6646-zvqkl\" (UID: \"ea8e3e57-237e-4f7b-926c-c554dd3c0da0\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-zvqkl"
May 25 12:09:58.971941 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:09:58.971911    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-zvqkl"
May 25 12:09:59.100372 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:09:59.100348    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-zvqkl"]
May 25 12:09:59.102607 ip-10-0-136-227 kubenswrapper[2561]: W0525 12:09:59.102575    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea8e3e57_237e_4f7b_926c_c554dd3c0da0.slice/crio-b688b6e47eb8884feac06cb15adf0b2de8daa0b6390b6d9e82c89fdd9ce2e762 WatchSource:0}: Error finding container b688b6e47eb8884feac06cb15adf0b2de8daa0b6390b6d9e82c89fdd9ce2e762: Status 404 returned error can't find the container with id b688b6e47eb8884feac06cb15adf0b2de8daa0b6390b6d9e82c89fdd9ce2e762
May 25 12:09:59.104144 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:09:59.104128    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 25 12:10:00.047881 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:10:00.047846    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-zvqkl" event={"ID":"ea8e3e57-237e-4f7b-926c-c554dd3c0da0","Type":"ContainerStarted","Data":"a8a13ef6a8690551a92310478dbadcbba0ac3023dba5f68e7614683aa1b067d1"}
May 25 12:10:00.047881 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:10:00.047887    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-zvqkl" event={"ID":"ea8e3e57-237e-4f7b-926c-c554dd3c0da0","Type":"ContainerStarted","Data":"b688b6e47eb8884feac06cb15adf0b2de8daa0b6390b6d9e82c89fdd9ce2e762"}
May 25 12:10:03.059039 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:10:03.058997    2561 generic.go:358] "Generic (PLEG): container finished" podID="ea8e3e57-237e-4f7b-926c-c554dd3c0da0" containerID="a8a13ef6a8690551a92310478dbadcbba0ac3023dba5f68e7614683aa1b067d1" exitCode=0
May 25 12:10:03.059426 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:10:03.059050    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-zvqkl" event={"ID":"ea8e3e57-237e-4f7b-926c-c554dd3c0da0","Type":"ContainerDied","Data":"a8a13ef6a8690551a92310478dbadcbba0ac3023dba5f68e7614683aa1b067d1"}
May 25 12:10:04.064819 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:10:04.064784    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-zvqkl" event={"ID":"ea8e3e57-237e-4f7b-926c-c554dd3c0da0","Type":"ContainerStarted","Data":"cd6d16c5d38471e8f94b63958f3fbc555608381decd79b2a379cd6123bb94a37"}
May 25 12:10:04.065181 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:10:04.064826    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-zvqkl" event={"ID":"ea8e3e57-237e-4f7b-926c-c554dd3c0da0","Type":"ContainerStarted","Data":"f1acbc6cc1a9cb137b1ecacd772a3b48b6e2813098478901b3b910ba920f1187"}
May 25 12:10:04.065181 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:10:04.065029    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-zvqkl"
May 25 12:10:04.085482 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:10:04.085443    2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-zvqkl" podStartSLOduration=6.085429632 podStartE2EDuration="6.085429632s" podCreationTimestamp="2026-05-25 12:09:58 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 12:10:04.083183915 +0000 UTC m=+3177.628361552" watchObservedRunningTime="2026-05-25 12:10:04.085429632 +0000 UTC m=+3177.630607234"
May 25 12:10:05.068474 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:10:05.068446    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-zvqkl"
May 25 12:10:11.077435 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:10:11.077404    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-zvqkl"
May 25 12:10:41.081753 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:10:41.081725    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-zvqkl"
May 25 12:10:48.226233 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:10:48.226200    2561 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-vkcjk"]
May 25 12:10:48.230736 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:10:48.230713    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-vkcjk"
May 25 12:10:48.233767 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:10:48.233743    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"xgboost-v2-mlserver-predictor-serving-cert\""
May 25 12:10:48.233923 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:10:48.233747    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"xgboost-v2-mlserver-kube-rbac-proxy-sar-config\""
May 25 12:10:48.242983 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:10:48.242958    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-vkcjk"]
May 25 12:10:48.261449 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:10:48.261406    2561 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-zvqkl"]
May 25 12:10:48.261850 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:10:48.261797    2561 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-zvqkl" podUID="ea8e3e57-237e-4f7b-926c-c554dd3c0da0" containerName="kserve-container" containerID="cri-o://f1acbc6cc1a9cb137b1ecacd772a3b48b6e2813098478901b3b910ba920f1187" gracePeriod=30
May 25 12:10:48.261987 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:10:48.261851    2561 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-zvqkl" podUID="ea8e3e57-237e-4f7b-926c-c554dd3c0da0" containerName="kube-rbac-proxy" containerID="cri-o://cd6d16c5d38471e8f94b63958f3fbc555608381decd79b2a379cd6123bb94a37" gracePeriod=30
May 25 12:10:48.299664 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:10:48.299638    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6cd7349e-725f-42d7-9cbd-fdefe65d06d5-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-7799869d6f-vkcjk\" (UID: \"6cd7349e-725f-42d7-9cbd-fdefe65d06d5\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-vkcjk"
May 25 12:10:48.299776 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:10:48.299669    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfhpp\" (UniqueName: \"kubernetes.io/projected/6cd7349e-725f-42d7-9cbd-fdefe65d06d5-kube-api-access-xfhpp\") pod \"xgboost-v2-mlserver-predictor-7799869d6f-vkcjk\" (UID: \"6cd7349e-725f-42d7-9cbd-fdefe65d06d5\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-vkcjk"
May 25 12:10:48.299776 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:10:48.299688    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6cd7349e-725f-42d7-9cbd-fdefe65d06d5-kserve-provision-location\") pod \"xgboost-v2-mlserver-predictor-7799869d6f-vkcjk\" (UID: \"6cd7349e-725f-42d7-9cbd-fdefe65d06d5\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-vkcjk"
May 25 12:10:48.299776 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:10:48.299710    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"xgboost-v2-mlserver-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6cd7349e-725f-42d7-9cbd-fdefe65d06d5-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\") pod \"xgboost-v2-mlserver-predictor-7799869d6f-vkcjk\" (UID: \"6cd7349e-725f-42d7-9cbd-fdefe65d06d5\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-vkcjk"
May 25 12:10:48.400294 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:10:48.400257    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6cd7349e-725f-42d7-9cbd-fdefe65d06d5-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-7799869d6f-vkcjk\" (UID: \"6cd7349e-725f-42d7-9cbd-fdefe65d06d5\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-vkcjk"
May 25 12:10:48.400417 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:10:48.400304    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-xfhpp\" (UniqueName: \"kubernetes.io/projected/6cd7349e-725f-42d7-9cbd-fdefe65d06d5-kube-api-access-xfhpp\") pod \"xgboost-v2-mlserver-predictor-7799869d6f-vkcjk\" (UID: \"6cd7349e-725f-42d7-9cbd-fdefe65d06d5\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-vkcjk"
May 25 12:10:48.400417 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:10:48.400323    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6cd7349e-725f-42d7-9cbd-fdefe65d06d5-kserve-provision-location\") pod \"xgboost-v2-mlserver-predictor-7799869d6f-vkcjk\" (UID: \"6cd7349e-725f-42d7-9cbd-fdefe65d06d5\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-vkcjk"
May 25 12:10:48.400417 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:10:48.400349    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"xgboost-v2-mlserver-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6cd7349e-725f-42d7-9cbd-fdefe65d06d5-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\") pod \"xgboost-v2-mlserver-predictor-7799869d6f-vkcjk\" (UID: \"6cd7349e-725f-42d7-9cbd-fdefe65d06d5\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-vkcjk"
May 25 12:10:48.400652 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:10:48.400634    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6cd7349e-725f-42d7-9cbd-fdefe65d06d5-kserve-provision-location\") pod \"xgboost-v2-mlserver-predictor-7799869d6f-vkcjk\" (UID: \"6cd7349e-725f-42d7-9cbd-fdefe65d06d5\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-vkcjk"
May 25 12:10:48.400950 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:10:48.400922    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"xgboost-v2-mlserver-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6cd7349e-725f-42d7-9cbd-fdefe65d06d5-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\") pod \"xgboost-v2-mlserver-predictor-7799869d6f-vkcjk\" (UID: \"6cd7349e-725f-42d7-9cbd-fdefe65d06d5\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-vkcjk"
May 25 12:10:48.402416 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:10:48.402400    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6cd7349e-725f-42d7-9cbd-fdefe65d06d5-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-7799869d6f-vkcjk\" (UID: \"6cd7349e-725f-42d7-9cbd-fdefe65d06d5\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-vkcjk"
May 25 12:10:48.410156 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:10:48.410135    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfhpp\" (UniqueName: \"kubernetes.io/projected/6cd7349e-725f-42d7-9cbd-fdefe65d06d5-kube-api-access-xfhpp\") pod \"xgboost-v2-mlserver-predictor-7799869d6f-vkcjk\" (UID: \"6cd7349e-725f-42d7-9cbd-fdefe65d06d5\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-vkcjk"
May 25 12:10:48.544176 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:10:48.544118    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-vkcjk"
May 25 12:10:48.676226 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:10:48.676187    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-vkcjk"]
May 25 12:10:48.679282 ip-10-0-136-227 kubenswrapper[2561]: W0525 12:10:48.679242    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cd7349e_725f_42d7_9cbd_fdefe65d06d5.slice/crio-05f1f6cd6bc7d30623c514d48b7bdc263cae1eca37a872f4e551056b97ca8328 WatchSource:0}: Error finding container 05f1f6cd6bc7d30623c514d48b7bdc263cae1eca37a872f4e551056b97ca8328: Status 404 returned error can't find the container with id 05f1f6cd6bc7d30623c514d48b7bdc263cae1eca37a872f4e551056b97ca8328
May 25 12:10:49.212793 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:10:49.212757    2561 generic.go:358] "Generic (PLEG): container finished" podID="ea8e3e57-237e-4f7b-926c-c554dd3c0da0" containerID="cd6d16c5d38471e8f94b63958f3fbc555608381decd79b2a379cd6123bb94a37" exitCode=2
May 25 12:10:49.212982 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:10:49.212842    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-zvqkl" event={"ID":"ea8e3e57-237e-4f7b-926c-c554dd3c0da0","Type":"ContainerDied","Data":"cd6d16c5d38471e8f94b63958f3fbc555608381decd79b2a379cd6123bb94a37"}
May 25 12:10:49.214051 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:10:49.214029    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-vkcjk" event={"ID":"6cd7349e-725f-42d7-9cbd-fdefe65d06d5","Type":"ContainerStarted","Data":"60442d7468a33d3625b2c86a076b7710005505896169a337ad6c84be0d2a6112"}
May 25 12:10:49.214051 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:10:49.214053    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-vkcjk" event={"ID":"6cd7349e-725f-42d7-9cbd-fdefe65d06d5","Type":"ContainerStarted","Data":"05f1f6cd6bc7d30623c514d48b7bdc263cae1eca37a872f4e551056b97ca8328"}
May 25 12:10:51.073122 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:10:51.073082    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-zvqkl" podUID="ea8e3e57-237e-4f7b-926c-c554dd3c0da0" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.53:8643/healthz\": dial tcp 10.132.0.53:8643: connect: connection refused"
May 25 12:10:51.078322 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:10:51.078296    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-zvqkl" podUID="ea8e3e57-237e-4f7b-926c-c554dd3c0da0" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.53:8080/v2/models/isvc-xgboost-v2-mlserver/ready\": dial tcp 10.132.0.53:8080: connect: connection refused"
May 25 12:10:53.227988 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:10:53.227957    2561 generic.go:358] "Generic (PLEG): container finished" podID="6cd7349e-725f-42d7-9cbd-fdefe65d06d5" containerID="60442d7468a33d3625b2c86a076b7710005505896169a337ad6c84be0d2a6112" exitCode=0
May 25 12:10:53.228449 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:10:53.228029    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-vkcjk" event={"ID":"6cd7349e-725f-42d7-9cbd-fdefe65d06d5","Type":"ContainerDied","Data":"60442d7468a33d3625b2c86a076b7710005505896169a337ad6c84be0d2a6112"}
May 25 12:10:53.923248 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:10:53.923227    2561 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-zvqkl"
May 25 12:10:54.048775 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:10:54.048703    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ea8e3e57-237e-4f7b-926c-c554dd3c0da0-isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\") pod \"ea8e3e57-237e-4f7b-926c-c554dd3c0da0\" (UID: \"ea8e3e57-237e-4f7b-926c-c554dd3c0da0\") "
May 25 12:10:54.048775 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:10:54.048776    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ea8e3e57-237e-4f7b-926c-c554dd3c0da0-proxy-tls\") pod \"ea8e3e57-237e-4f7b-926c-c554dd3c0da0\" (UID: \"ea8e3e57-237e-4f7b-926c-c554dd3c0da0\") "
May 25 12:10:54.048971 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:10:54.048794    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ea8e3e57-237e-4f7b-926c-c554dd3c0da0-kserve-provision-location\") pod \"ea8e3e57-237e-4f7b-926c-c554dd3c0da0\" (UID: \"ea8e3e57-237e-4f7b-926c-c554dd3c0da0\") "
May 25 12:10:54.048971 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:10:54.048823    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s5v8z\" (UniqueName: \"kubernetes.io/projected/ea8e3e57-237e-4f7b-926c-c554dd3c0da0-kube-api-access-s5v8z\") pod \"ea8e3e57-237e-4f7b-926c-c554dd3c0da0\" (UID: \"ea8e3e57-237e-4f7b-926c-c554dd3c0da0\") "
May 25 12:10:54.049108 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:10:54.049084    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea8e3e57-237e-4f7b-926c-c554dd3c0da0-isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config") pod "ea8e3e57-237e-4f7b-926c-c554dd3c0da0" (UID: "ea8e3e57-237e-4f7b-926c-c554dd3c0da0"). InnerVolumeSpecName "isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 12:10:54.049184 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:10:54.049150    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea8e3e57-237e-4f7b-926c-c554dd3c0da0-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ea8e3e57-237e-4f7b-926c-c554dd3c0da0" (UID: "ea8e3e57-237e-4f7b-926c-c554dd3c0da0"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 12:10:54.050803 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:10:54.050784    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea8e3e57-237e-4f7b-926c-c554dd3c0da0-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ea8e3e57-237e-4f7b-926c-c554dd3c0da0" (UID: "ea8e3e57-237e-4f7b-926c-c554dd3c0da0"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 12:10:54.050920 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:10:54.050837    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea8e3e57-237e-4f7b-926c-c554dd3c0da0-kube-api-access-s5v8z" (OuterVolumeSpecName: "kube-api-access-s5v8z") pod "ea8e3e57-237e-4f7b-926c-c554dd3c0da0" (UID: "ea8e3e57-237e-4f7b-926c-c554dd3c0da0"). InnerVolumeSpecName "kube-api-access-s5v8z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 12:10:54.149694 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:10:54.149669    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ea8e3e57-237e-4f7b-926c-c554dd3c0da0-proxy-tls\") on node \"ip-10-0-136-227.ec2.internal\" DevicePath \"\""
May 25 12:10:54.149694 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:10:54.149691    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ea8e3e57-237e-4f7b-926c-c554dd3c0da0-kserve-provision-location\") on node \"ip-10-0-136-227.ec2.internal\" DevicePath \"\""
May 25 12:10:54.149694 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:10:54.149700    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s5v8z\" (UniqueName: \"kubernetes.io/projected/ea8e3e57-237e-4f7b-926c-c554dd3c0da0-kube-api-access-s5v8z\") on node \"ip-10-0-136-227.ec2.internal\" DevicePath \"\""
May 25 12:10:54.149880 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:10:54.149710    2561 reconciler_common.go:299] "Volume detached for volume \"isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ea8e3e57-237e-4f7b-926c-c554dd3c0da0-isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-227.ec2.internal\" DevicePath \"\""
May 25 12:10:54.233363 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:10:54.233324    2561 generic.go:358] "Generic (PLEG): container finished" podID="ea8e3e57-237e-4f7b-926c-c554dd3c0da0" containerID="f1acbc6cc1a9cb137b1ecacd772a3b48b6e2813098478901b3b910ba920f1187" exitCode=0
May 25 12:10:54.233779 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:10:54.233396    2561 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-zvqkl"
May 25 12:10:54.233779 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:10:54.233407    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-zvqkl" event={"ID":"ea8e3e57-237e-4f7b-926c-c554dd3c0da0","Type":"ContainerDied","Data":"f1acbc6cc1a9cb137b1ecacd772a3b48b6e2813098478901b3b910ba920f1187"}
May 25 12:10:54.233779 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:10:54.233454    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-zvqkl" event={"ID":"ea8e3e57-237e-4f7b-926c-c554dd3c0da0","Type":"ContainerDied","Data":"b688b6e47eb8884feac06cb15adf0b2de8daa0b6390b6d9e82c89fdd9ce2e762"}
May 25 12:10:54.233779 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:10:54.233478    2561 scope.go:117] "RemoveContainer" containerID="cd6d16c5d38471e8f94b63958f3fbc555608381decd79b2a379cd6123bb94a37"
May 25 12:10:54.235564 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:10:54.235540    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-vkcjk" event={"ID":"6cd7349e-725f-42d7-9cbd-fdefe65d06d5","Type":"ContainerStarted","Data":"d334ed3f82fe9a3604441a2c28afd866bab6bc9d8a1e6e575fd598eaafd40796"}
May 25 12:10:54.235679 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:10:54.235573    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-vkcjk" event={"ID":"6cd7349e-725f-42d7-9cbd-fdefe65d06d5","Type":"ContainerStarted","Data":"0e171207626f24f389e7ee4bcdae7a11c94f4aa6b9f7b837dee8ec8451d2559f"}
May 25 12:10:54.235781 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:10:54.235766    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-vkcjk"
May 25 12:10:54.235850 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:10:54.235793    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-vkcjk"
May 25 12:10:54.242170 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:10:54.242153    2561 scope.go:117] "RemoveContainer" containerID="f1acbc6cc1a9cb137b1ecacd772a3b48b6e2813098478901b3b910ba920f1187"
May 25 12:10:54.251071 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:10:54.251049    2561 scope.go:117] "RemoveContainer" containerID="a8a13ef6a8690551a92310478dbadcbba0ac3023dba5f68e7614683aa1b067d1"
May 25 12:10:54.257924 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:10:54.257890    2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-vkcjk" podStartSLOduration=6.257879035 podStartE2EDuration="6.257879035s" podCreationTimestamp="2026-05-25 12:10:48 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 12:10:54.25606331 +0000 UTC m=+3227.801240913" watchObservedRunningTime="2026-05-25 12:10:54.257879035 +0000 UTC m=+3227.803056638"
May 25 12:10:54.258381 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:10:54.258364    2561 scope.go:117] "RemoveContainer" containerID="cd6d16c5d38471e8f94b63958f3fbc555608381decd79b2a379cd6123bb94a37"
May 25 12:10:54.258610 ip-10-0-136-227 kubenswrapper[2561]: E0525 12:10:54.258589    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd6d16c5d38471e8f94b63958f3fbc555608381decd79b2a379cd6123bb94a37\": container with ID starting with cd6d16c5d38471e8f94b63958f3fbc555608381decd79b2a379cd6123bb94a37 not found: ID does not exist" containerID="cd6d16c5d38471e8f94b63958f3fbc555608381decd79b2a379cd6123bb94a37"
May 25 12:10:54.258703 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:10:54.258621    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd6d16c5d38471e8f94b63958f3fbc555608381decd79b2a379cd6123bb94a37"} err="failed to get container status \"cd6d16c5d38471e8f94b63958f3fbc555608381decd79b2a379cd6123bb94a37\": rpc error: code = NotFound desc = could not find container \"cd6d16c5d38471e8f94b63958f3fbc555608381decd79b2a379cd6123bb94a37\": container with ID starting with cd6d16c5d38471e8f94b63958f3fbc555608381decd79b2a379cd6123bb94a37 not found: ID does not exist"
May 25 12:10:54.258703 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:10:54.258643    2561 scope.go:117] "RemoveContainer" containerID="f1acbc6cc1a9cb137b1ecacd772a3b48b6e2813098478901b3b910ba920f1187"
May 25 12:10:54.258877 ip-10-0-136-227 kubenswrapper[2561]: E0525 12:10:54.258858    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1acbc6cc1a9cb137b1ecacd772a3b48b6e2813098478901b3b910ba920f1187\": container with ID starting with f1acbc6cc1a9cb137b1ecacd772a3b48b6e2813098478901b3b910ba920f1187 not found: ID does not exist" containerID="f1acbc6cc1a9cb137b1ecacd772a3b48b6e2813098478901b3b910ba920f1187"
May 25 12:10:54.258917 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:10:54.258884    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1acbc6cc1a9cb137b1ecacd772a3b48b6e2813098478901b3b910ba920f1187"} err="failed to get container status \"f1acbc6cc1a9cb137b1ecacd772a3b48b6e2813098478901b3b910ba920f1187\": rpc error: code = NotFound desc = could not find container \"f1acbc6cc1a9cb137b1ecacd772a3b48b6e2813098478901b3b910ba920f1187\": container with ID starting with f1acbc6cc1a9cb137b1ecacd772a3b48b6e2813098478901b3b910ba920f1187 not found: ID does not exist"
May 25 12:10:54.258917 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:10:54.258898    2561 scope.go:117] "RemoveContainer" containerID="a8a13ef6a8690551a92310478dbadcbba0ac3023dba5f68e7614683aa1b067d1"
May 25 12:10:54.259128 ip-10-0-136-227 kubenswrapper[2561]: E0525 12:10:54.259111    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8a13ef6a8690551a92310478dbadcbba0ac3023dba5f68e7614683aa1b067d1\": container with ID starting with a8a13ef6a8690551a92310478dbadcbba0ac3023dba5f68e7614683aa1b067d1 not found: ID does not exist" containerID="a8a13ef6a8690551a92310478dbadcbba0ac3023dba5f68e7614683aa1b067d1"
May 25 12:10:54.259209 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:10:54.259131    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8a13ef6a8690551a92310478dbadcbba0ac3023dba5f68e7614683aa1b067d1"} err="failed to get container status \"a8a13ef6a8690551a92310478dbadcbba0ac3023dba5f68e7614683aa1b067d1\": rpc error: code = NotFound desc = could not find container \"a8a13ef6a8690551a92310478dbadcbba0ac3023dba5f68e7614683aa1b067d1\": container with ID starting with a8a13ef6a8690551a92310478dbadcbba0ac3023dba5f68e7614683aa1b067d1 not found: ID does not exist"
May 25 12:10:54.272358 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:10:54.271813    2561 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-zvqkl"]
May 25 12:10:54.276547 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:10:54.276527    2561 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-67d4bc6646-zvqkl"]
May 25 12:10:55.117172 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:10:55.117135    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ea8e3e57-237e-4f7b-926c-c554dd3c0da0" path="/var/lib/kubelet/pods/ea8e3e57-237e-4f7b-926c-c554dd3c0da0/volumes"
May 25 12:11:00.245310 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:11:00.245262    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-vkcjk"
May 25 12:11:30.249355 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:11:30.249327    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-vkcjk"
May 25 12:11:38.281611 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:11:38.281565    2561 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-vkcjk"]
May 25 12:11:38.282083 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:11:38.282031    2561 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-vkcjk" podUID="6cd7349e-725f-42d7-9cbd-fdefe65d06d5" containerName="kserve-container" containerID="cri-o://0e171207626f24f389e7ee4bcdae7a11c94f4aa6b9f7b837dee8ec8451d2559f" gracePeriod=30
May 25 12:11:38.282168 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:11:38.282083    2561 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-vkcjk" podUID="6cd7349e-725f-42d7-9cbd-fdefe65d06d5" containerName="kube-rbac-proxy" containerID="cri-o://d334ed3f82fe9a3604441a2c28afd866bab6bc9d8a1e6e575fd598eaafd40796" gracePeriod=30
May 25 12:11:39.397523 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:11:39.397491    2561 generic.go:358] "Generic (PLEG): container finished" podID="6cd7349e-725f-42d7-9cbd-fdefe65d06d5" containerID="d334ed3f82fe9a3604441a2c28afd866bab6bc9d8a1e6e575fd598eaafd40796" exitCode=2
May 25 12:11:39.397885 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:11:39.397564    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-vkcjk" event={"ID":"6cd7349e-725f-42d7-9cbd-fdefe65d06d5","Type":"ContainerDied","Data":"d334ed3f82fe9a3604441a2c28afd866bab6bc9d8a1e6e575fd598eaafd40796"}
May 25 12:11:40.240567 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:11:40.240521    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-vkcjk" podUID="6cd7349e-725f-42d7-9cbd-fdefe65d06d5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.54:8643/healthz\": dial tcp 10.132.0.54:8643: connect: connection refused"
May 25 12:11:43.933380 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:11:43.933356    2561 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-vkcjk"
May 25 12:11:44.054782 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:11:44.054695    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6cd7349e-725f-42d7-9cbd-fdefe65d06d5-proxy-tls\") pod \"6cd7349e-725f-42d7-9cbd-fdefe65d06d5\" (UID: \"6cd7349e-725f-42d7-9cbd-fdefe65d06d5\") "
May 25 12:11:44.054782 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:11:44.054774    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6cd7349e-725f-42d7-9cbd-fdefe65d06d5-kserve-provision-location\") pod \"6cd7349e-725f-42d7-9cbd-fdefe65d06d5\" (UID: \"6cd7349e-725f-42d7-9cbd-fdefe65d06d5\") "
May 25 12:11:44.055004 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:11:44.054850    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"xgboost-v2-mlserver-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6cd7349e-725f-42d7-9cbd-fdefe65d06d5-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\") pod \"6cd7349e-725f-42d7-9cbd-fdefe65d06d5\" (UID: \"6cd7349e-725f-42d7-9cbd-fdefe65d06d5\") "
May 25 12:11:44.055004 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:11:44.054947    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xfhpp\" (UniqueName: \"kubernetes.io/projected/6cd7349e-725f-42d7-9cbd-fdefe65d06d5-kube-api-access-xfhpp\") pod \"6cd7349e-725f-42d7-9cbd-fdefe65d06d5\" (UID: \"6cd7349e-725f-42d7-9cbd-fdefe65d06d5\") "
May 25 12:11:44.055182 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:11:44.055065    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cd7349e-725f-42d7-9cbd-fdefe65d06d5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6cd7349e-725f-42d7-9cbd-fdefe65d06d5" (UID: "6cd7349e-725f-42d7-9cbd-fdefe65d06d5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 12:11:44.055182 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:11:44.055177    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6cd7349e-725f-42d7-9cbd-fdefe65d06d5-kserve-provision-location\") on node \"ip-10-0-136-227.ec2.internal\" DevicePath \"\""
May 25 12:11:44.055320 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:11:44.055240    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6cd7349e-725f-42d7-9cbd-fdefe65d06d5-xgboost-v2-mlserver-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "xgboost-v2-mlserver-kube-rbac-proxy-sar-config") pod "6cd7349e-725f-42d7-9cbd-fdefe65d06d5" (UID: "6cd7349e-725f-42d7-9cbd-fdefe65d06d5"). InnerVolumeSpecName "xgboost-v2-mlserver-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 12:11:44.056791 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:11:44.056769    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6cd7349e-725f-42d7-9cbd-fdefe65d06d5-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6cd7349e-725f-42d7-9cbd-fdefe65d06d5" (UID: "6cd7349e-725f-42d7-9cbd-fdefe65d06d5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 12:11:44.056898 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:11:44.056831    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cd7349e-725f-42d7-9cbd-fdefe65d06d5-kube-api-access-xfhpp" (OuterVolumeSpecName: "kube-api-access-xfhpp") pod "6cd7349e-725f-42d7-9cbd-fdefe65d06d5" (UID: "6cd7349e-725f-42d7-9cbd-fdefe65d06d5"). InnerVolumeSpecName "kube-api-access-xfhpp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 12:11:44.155572 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:11:44.155541    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6cd7349e-725f-42d7-9cbd-fdefe65d06d5-proxy-tls\") on node \"ip-10-0-136-227.ec2.internal\" DevicePath \"\""
May 25 12:11:44.155572 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:11:44.155568    2561 reconciler_common.go:299] "Volume detached for volume \"xgboost-v2-mlserver-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6cd7349e-725f-42d7-9cbd-fdefe65d06d5-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-227.ec2.internal\" DevicePath \"\""
May 25 12:11:44.155765 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:11:44.155580    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xfhpp\" (UniqueName: \"kubernetes.io/projected/6cd7349e-725f-42d7-9cbd-fdefe65d06d5-kube-api-access-xfhpp\") on node \"ip-10-0-136-227.ec2.internal\" DevicePath \"\""
May 25 12:11:44.415597 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:11:44.415560    2561 generic.go:358] "Generic (PLEG): container finished" podID="6cd7349e-725f-42d7-9cbd-fdefe65d06d5" containerID="0e171207626f24f389e7ee4bcdae7a11c94f4aa6b9f7b837dee8ec8451d2559f" exitCode=0
May 25 12:11:44.415783 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:11:44.415635    2561 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-vkcjk"
May 25 12:11:44.415783 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:11:44.415650    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-vkcjk" event={"ID":"6cd7349e-725f-42d7-9cbd-fdefe65d06d5","Type":"ContainerDied","Data":"0e171207626f24f389e7ee4bcdae7a11c94f4aa6b9f7b837dee8ec8451d2559f"}
May 25 12:11:44.415783 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:11:44.415693    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-vkcjk" event={"ID":"6cd7349e-725f-42d7-9cbd-fdefe65d06d5","Type":"ContainerDied","Data":"05f1f6cd6bc7d30623c514d48b7bdc263cae1eca37a872f4e551056b97ca8328"}
May 25 12:11:44.415783 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:11:44.415709    2561 scope.go:117] "RemoveContainer" containerID="d334ed3f82fe9a3604441a2c28afd866bab6bc9d8a1e6e575fd598eaafd40796"
May 25 12:11:44.425310 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:11:44.425292    2561 scope.go:117] "RemoveContainer" containerID="0e171207626f24f389e7ee4bcdae7a11c94f4aa6b9f7b837dee8ec8451d2559f"
May 25 12:11:44.432437 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:11:44.432421    2561 scope.go:117] "RemoveContainer" containerID="60442d7468a33d3625b2c86a076b7710005505896169a337ad6c84be0d2a6112"
May 25 12:11:44.439513 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:11:44.439494    2561 scope.go:117] "RemoveContainer" containerID="d334ed3f82fe9a3604441a2c28afd866bab6bc9d8a1e6e575fd598eaafd40796"
May 25 12:11:44.439764 ip-10-0-136-227 kubenswrapper[2561]: E0525 12:11:44.439745    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d334ed3f82fe9a3604441a2c28afd866bab6bc9d8a1e6e575fd598eaafd40796\": container with ID starting with d334ed3f82fe9a3604441a2c28afd866bab6bc9d8a1e6e575fd598eaafd40796 not found: ID does not exist" containerID="d334ed3f82fe9a3604441a2c28afd866bab6bc9d8a1e6e575fd598eaafd40796"
May 25 12:11:44.439850 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:11:44.439774    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d334ed3f82fe9a3604441a2c28afd866bab6bc9d8a1e6e575fd598eaafd40796"} err="failed to get container status \"d334ed3f82fe9a3604441a2c28afd866bab6bc9d8a1e6e575fd598eaafd40796\": rpc error: code = NotFound desc = could not find container \"d334ed3f82fe9a3604441a2c28afd866bab6bc9d8a1e6e575fd598eaafd40796\": container with ID starting with d334ed3f82fe9a3604441a2c28afd866bab6bc9d8a1e6e575fd598eaafd40796 not found: ID does not exist"
May 25 12:11:44.439850 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:11:44.439791    2561 scope.go:117] "RemoveContainer" containerID="0e171207626f24f389e7ee4bcdae7a11c94f4aa6b9f7b837dee8ec8451d2559f"
May 25 12:11:44.439850 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:11:44.439824    2561 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-vkcjk"]
May 25 12:11:44.440079 ip-10-0-136-227 kubenswrapper[2561]: E0525 12:11:44.440059    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e171207626f24f389e7ee4bcdae7a11c94f4aa6b9f7b837dee8ec8451d2559f\": container with ID starting with 0e171207626f24f389e7ee4bcdae7a11c94f4aa6b9f7b837dee8ec8451d2559f not found: ID does not exist" containerID="0e171207626f24f389e7ee4bcdae7a11c94f4aa6b9f7b837dee8ec8451d2559f"
May 25 12:11:44.440131 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:11:44.440084    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e171207626f24f389e7ee4bcdae7a11c94f4aa6b9f7b837dee8ec8451d2559f"} err="failed to get container status \"0e171207626f24f389e7ee4bcdae7a11c94f4aa6b9f7b837dee8ec8451d2559f\": rpc error: code = NotFound desc = could not find container \"0e171207626f24f389e7ee4bcdae7a11c94f4aa6b9f7b837dee8ec8451d2559f\": container with ID starting with 0e171207626f24f389e7ee4bcdae7a11c94f4aa6b9f7b837dee8ec8451d2559f not found: ID does not exist"
May 25 12:11:44.440131 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:11:44.440099    2561 scope.go:117] "RemoveContainer" containerID="60442d7468a33d3625b2c86a076b7710005505896169a337ad6c84be0d2a6112"
May 25 12:11:44.440398 ip-10-0-136-227 kubenswrapper[2561]: E0525 12:11:44.440379    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60442d7468a33d3625b2c86a076b7710005505896169a337ad6c84be0d2a6112\": container with ID starting with 60442d7468a33d3625b2c86a076b7710005505896169a337ad6c84be0d2a6112 not found: ID does not exist" containerID="60442d7468a33d3625b2c86a076b7710005505896169a337ad6c84be0d2a6112"
May 25 12:11:44.440457 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:11:44.440403    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60442d7468a33d3625b2c86a076b7710005505896169a337ad6c84be0d2a6112"} err="failed to get container status \"60442d7468a33d3625b2c86a076b7710005505896169a337ad6c84be0d2a6112\": rpc error: code = NotFound desc = could not find container \"60442d7468a33d3625b2c86a076b7710005505896169a337ad6c84be0d2a6112\": container with ID starting with 60442d7468a33d3625b2c86a076b7710005505896169a337ad6c84be0d2a6112 not found: ID does not exist"
May 25 12:11:44.443730 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:11:44.443709    2561 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-7799869d6f-vkcjk"]
May 25 12:11:45.116918 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:11:45.116885    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6cd7349e-725f-42d7-9cbd-fdefe65d06d5" path="/var/lib/kubelet/pods/6cd7349e-725f-42d7-9cbd-fdefe65d06d5/volumes"
May 25 12:12:07.244227 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:12:07.244191    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-9lj95_10a66344-3304-49a1-bbc1-b885800f5acf/console-operator/1.log"
May 25 12:12:07.250113 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:12:07.250078    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-9lj95_10a66344-3304-49a1-bbc1-b885800f5acf/console-operator/1.log"
May 25 12:12:58.557589 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:12:58.557516    2561 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-mctcc"]
May 25 12:12:58.558016 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:12:58.557849    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea8e3e57-237e-4f7b-926c-c554dd3c0da0" containerName="storage-initializer"
May 25 12:12:58.558016 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:12:58.557860    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea8e3e57-237e-4f7b-926c-c554dd3c0da0" containerName="storage-initializer"
May 25 12:12:58.558016 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:12:58.557868    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6cd7349e-725f-42d7-9cbd-fdefe65d06d5" containerName="kserve-container"
May 25 12:12:58.558016 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:12:58.557875    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="6cd7349e-725f-42d7-9cbd-fdefe65d06d5" containerName="kserve-container"
May 25 12:12:58.558016 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:12:58.557885    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6cd7349e-725f-42d7-9cbd-fdefe65d06d5" containerName="kube-rbac-proxy"
May 25 12:12:58.558016 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:12:58.557891    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="6cd7349e-725f-42d7-9cbd-fdefe65d06d5" containerName="kube-rbac-proxy"
May 25 12:12:58.558016 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:12:58.557901    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6cd7349e-725f-42d7-9cbd-fdefe65d06d5" containerName="storage-initializer"
May 25 12:12:58.558016 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:12:58.557906    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="6cd7349e-725f-42d7-9cbd-fdefe65d06d5" containerName="storage-initializer"
May 25 12:12:58.558016 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:12:58.557916    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea8e3e57-237e-4f7b-926c-c554dd3c0da0" containerName="kube-rbac-proxy"
May 25 12:12:58.558016 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:12:58.557921    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea8e3e57-237e-4f7b-926c-c554dd3c0da0" containerName="kube-rbac-proxy"
May 25 12:12:58.558016 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:12:58.557931    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea8e3e57-237e-4f7b-926c-c554dd3c0da0" containerName="kserve-container"
May 25 12:12:58.558016 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:12:58.557936    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea8e3e57-237e-4f7b-926c-c554dd3c0da0" containerName="kserve-container"
May 25 12:12:58.558016 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:12:58.557991    2561 memory_manager.go:356] "RemoveStaleState removing state" podUID="ea8e3e57-237e-4f7b-926c-c554dd3c0da0" containerName="kube-rbac-proxy"
May 25 12:12:58.558016 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:12:58.557998    2561 memory_manager.go:356] "RemoveStaleState removing state" podUID="6cd7349e-725f-42d7-9cbd-fdefe65d06d5" containerName="kserve-container"
May 25 12:12:58.558016 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:12:58.558005    2561 memory_manager.go:356] "RemoveStaleState removing state" podUID="ea8e3e57-237e-4f7b-926c-c554dd3c0da0" containerName="kserve-container"
May 25 12:12:58.558016 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:12:58.558014    2561 memory_manager.go:356] "RemoveStaleState removing state" podUID="6cd7349e-725f-42d7-9cbd-fdefe65d06d5" containerName="kube-rbac-proxy"
May 25 12:12:58.561258 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:12:58.561234    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-mctcc"
May 25 12:12:58.564220 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:12:58.564191    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config\""
May 25 12:12:58.564220 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:12:58.564214    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-svs62\""
May 25 12:12:58.564476 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:12:58.564191    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
May 25 12:12:58.564476 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:12:58.564290    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
May 25 12:12:58.564476 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:12:58.564217    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-runtime-predictor-serving-cert\""
May 25 12:12:58.575735 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:12:58.575713    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-mctcc"]
May 25 12:12:58.661821 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:12:58.661796    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/012ffa82-136c-4072-9531-916607acdf92-kserve-provision-location\") pod \"isvc-xgboost-v2-runtime-predictor-6dc5954dc-mctcc\" (UID: \"012ffa82-136c-4072-9531-916607acdf92\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-mctcc"
May 25 12:12:58.661913 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:12:58.661866    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/012ffa82-136c-4072-9531-916607acdf92-isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-runtime-predictor-6dc5954dc-mctcc\" (UID: \"012ffa82-136c-4072-9531-916607acdf92\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-mctcc"
May 25 12:12:58.661967 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:12:58.661921    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/012ffa82-136c-4072-9531-916607acdf92-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-6dc5954dc-mctcc\" (UID: \"012ffa82-136c-4072-9531-916607acdf92\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-mctcc"
May 25 12:12:58.661967 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:12:58.661946    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bv2b\" (UniqueName: \"kubernetes.io/projected/012ffa82-136c-4072-9531-916607acdf92-kube-api-access-7bv2b\") pod \"isvc-xgboost-v2-runtime-predictor-6dc5954dc-mctcc\" (UID: \"012ffa82-136c-4072-9531-916607acdf92\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-mctcc"
May 25 12:12:58.762301 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:12:58.762242    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/012ffa82-136c-4072-9531-916607acdf92-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-6dc5954dc-mctcc\" (UID: \"012ffa82-136c-4072-9531-916607acdf92\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-mctcc"
May 25 12:12:58.762446 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:12:58.762309    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-7bv2b\" (UniqueName: \"kubernetes.io/projected/012ffa82-136c-4072-9531-916607acdf92-kube-api-access-7bv2b\") pod \"isvc-xgboost-v2-runtime-predictor-6dc5954dc-mctcc\" (UID: \"012ffa82-136c-4072-9531-916607acdf92\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-mctcc"
May 25 12:12:58.762446 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:12:58.762330    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/012ffa82-136c-4072-9531-916607acdf92-kserve-provision-location\") pod \"isvc-xgboost-v2-runtime-predictor-6dc5954dc-mctcc\" (UID: \"012ffa82-136c-4072-9531-916607acdf92\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-mctcc"
May 25 12:12:58.762446 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:12:58.762382    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/012ffa82-136c-4072-9531-916607acdf92-isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-runtime-predictor-6dc5954dc-mctcc\" (UID: \"012ffa82-136c-4072-9531-916607acdf92\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-mctcc"
May 25 12:12:58.762853 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:12:58.762830    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/012ffa82-136c-4072-9531-916607acdf92-kserve-provision-location\") pod \"isvc-xgboost-v2-runtime-predictor-6dc5954dc-mctcc\" (UID: \"012ffa82-136c-4072-9531-916607acdf92\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-mctcc"
May 25 12:12:58.763120 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:12:58.763100    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/012ffa82-136c-4072-9531-916607acdf92-isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-runtime-predictor-6dc5954dc-mctcc\" (UID: \"012ffa82-136c-4072-9531-916607acdf92\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-mctcc"
May 25 12:12:58.764743 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:12:58.764721    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/012ffa82-136c-4072-9531-916607acdf92-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-6dc5954dc-mctcc\" (UID: \"012ffa82-136c-4072-9531-916607acdf92\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-mctcc"
May 25 12:12:58.770476 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:12:58.770451    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bv2b\" (UniqueName: \"kubernetes.io/projected/012ffa82-136c-4072-9531-916607acdf92-kube-api-access-7bv2b\") pod \"isvc-xgboost-v2-runtime-predictor-6dc5954dc-mctcc\" (UID: \"012ffa82-136c-4072-9531-916607acdf92\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-mctcc"
May 25 12:12:58.873009 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:12:58.872982    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-mctcc"
May 25 12:12:59.006641 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:12:59.006599    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-mctcc"]
May 25 12:12:59.009978 ip-10-0-136-227 kubenswrapper[2561]: W0525 12:12:59.009949    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod012ffa82_136c_4072_9531_916607acdf92.slice/crio-d44f242adb6085a060d58eed42c566d89afb08a01de77ebe6a35e6cd4d060b96 WatchSource:0}: Error finding container d44f242adb6085a060d58eed42c566d89afb08a01de77ebe6a35e6cd4d060b96: Status 404 returned error can't find the container with id d44f242adb6085a060d58eed42c566d89afb08a01de77ebe6a35e6cd4d060b96
May 25 12:12:59.665397 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:12:59.665359    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-mctcc" event={"ID":"012ffa82-136c-4072-9531-916607acdf92","Type":"ContainerStarted","Data":"eb4e95cf2ce6245f1ed6b9574ae943fb66ceea27d553536238c499b1d75b65af"}
May 25 12:12:59.665397 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:12:59.665400    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-mctcc" event={"ID":"012ffa82-136c-4072-9531-916607acdf92","Type":"ContainerStarted","Data":"d44f242adb6085a060d58eed42c566d89afb08a01de77ebe6a35e6cd4d060b96"}
May 25 12:13:03.679341 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:13:03.679304    2561 generic.go:358] "Generic (PLEG): container finished" podID="012ffa82-136c-4072-9531-916607acdf92" containerID="eb4e95cf2ce6245f1ed6b9574ae943fb66ceea27d553536238c499b1d75b65af" exitCode=0
May 25 12:13:03.679789 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:13:03.679386    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-mctcc" event={"ID":"012ffa82-136c-4072-9531-916607acdf92","Type":"ContainerDied","Data":"eb4e95cf2ce6245f1ed6b9574ae943fb66ceea27d553536238c499b1d75b65af"}
May 25 12:13:04.684393 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:13:04.684358    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-mctcc" event={"ID":"012ffa82-136c-4072-9531-916607acdf92","Type":"ContainerStarted","Data":"97e20a09dc0f92d36e2ec7e29bda6aa61e6c6c8f9ff27523494e67cb1f681433"}
May 25 12:13:04.684393 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:13:04.684395    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-mctcc" event={"ID":"012ffa82-136c-4072-9531-916607acdf92","Type":"ContainerStarted","Data":"212233f3fe85f4a72c75dc6c05ff4c7c13f95af9ab283bd81d53d4b7bc6e4a04"}
May 25 12:13:04.684841 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:13:04.684612    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-mctcc"
May 25 12:13:04.684841 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:13:04.684733    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-mctcc"
May 25 12:13:04.705325 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:13:04.705264    2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-mctcc" podStartSLOduration=6.705250565 podStartE2EDuration="6.705250565s" podCreationTimestamp="2026-05-25 12:12:58 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 12:13:04.704144635 +0000 UTC m=+3358.249322239" watchObservedRunningTime="2026-05-25 12:13:04.705250565 +0000 UTC m=+3358.250428168"
May 25 12:13:10.693740 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:13:10.693709    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-mctcc"
May 25 12:13:40.762986 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:13:40.762934    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-mctcc" podUID="012ffa82-136c-4072-9531-916607acdf92" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400"
May 25 12:13:50.696217 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:13:50.696186    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-mctcc"
May 25 12:13:58.636116 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:13:58.636034    2561 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-mctcc"]
May 25 12:13:58.636530 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:13:58.636381    2561 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-mctcc" podUID="012ffa82-136c-4072-9531-916607acdf92" containerName="kserve-container" containerID="cri-o://212233f3fe85f4a72c75dc6c05ff4c7c13f95af9ab283bd81d53d4b7bc6e4a04" gracePeriod=30
May 25 12:13:58.636530 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:13:58.636396    2561 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-mctcc" podUID="012ffa82-136c-4072-9531-916607acdf92" containerName="kube-rbac-proxy" containerID="cri-o://97e20a09dc0f92d36e2ec7e29bda6aa61e6c6c8f9ff27523494e67cb1f681433" gracePeriod=30
May 25 12:13:58.867932 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:13:58.867896    2561 generic.go:358] "Generic (PLEG): container finished" podID="012ffa82-136c-4072-9531-916607acdf92" containerID="97e20a09dc0f92d36e2ec7e29bda6aa61e6c6c8f9ff27523494e67cb1f681433" exitCode=2
May 25 12:13:58.868086 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:13:58.867969    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-mctcc" event={"ID":"012ffa82-136c-4072-9531-916607acdf92","Type":"ContainerDied","Data":"97e20a09dc0f92d36e2ec7e29bda6aa61e6c6c8f9ff27523494e67cb1f681433"}
May 25 12:14:00.689527 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:14:00.689479    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-mctcc" podUID="012ffa82-136c-4072-9531-916607acdf92" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.55:8643/healthz\": dial tcp 10.132.0.55:8643: connect: connection refused"
May 25 12:14:05.086094 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:14:05.086064    2561 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-mctcc"
May 25 12:14:05.205611 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:14:05.205546    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/012ffa82-136c-4072-9531-916607acdf92-proxy-tls\") pod \"012ffa82-136c-4072-9531-916607acdf92\" (UID: \"012ffa82-136c-4072-9531-916607acdf92\") "
May 25 12:14:05.205611 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:14:05.205599    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/012ffa82-136c-4072-9531-916607acdf92-isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config\") pod \"012ffa82-136c-4072-9531-916607acdf92\" (UID: \"012ffa82-136c-4072-9531-916607acdf92\") "
May 25 12:14:05.205764 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:14:05.205638    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7bv2b\" (UniqueName: \"kubernetes.io/projected/012ffa82-136c-4072-9531-916607acdf92-kube-api-access-7bv2b\") pod \"012ffa82-136c-4072-9531-916607acdf92\" (UID: \"012ffa82-136c-4072-9531-916607acdf92\") "
May 25 12:14:05.205764 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:14:05.205663    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/012ffa82-136c-4072-9531-916607acdf92-kserve-provision-location\") pod \"012ffa82-136c-4072-9531-916607acdf92\" (UID: \"012ffa82-136c-4072-9531-916607acdf92\") "
May 25 12:14:05.206007 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:14:05.205983    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/012ffa82-136c-4072-9531-916607acdf92-isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config") pod "012ffa82-136c-4072-9531-916607acdf92" (UID: "012ffa82-136c-4072-9531-916607acdf92"). InnerVolumeSpecName "isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 12:14:05.206007 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:14:05.205992    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/012ffa82-136c-4072-9531-916607acdf92-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "012ffa82-136c-4072-9531-916607acdf92" (UID: "012ffa82-136c-4072-9531-916607acdf92"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 12:14:05.207675 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:14:05.207655    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/012ffa82-136c-4072-9531-916607acdf92-kube-api-access-7bv2b" (OuterVolumeSpecName: "kube-api-access-7bv2b") pod "012ffa82-136c-4072-9531-916607acdf92" (UID: "012ffa82-136c-4072-9531-916607acdf92"). InnerVolumeSpecName "kube-api-access-7bv2b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 12:14:05.207734 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:14:05.207720    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/012ffa82-136c-4072-9531-916607acdf92-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "012ffa82-136c-4072-9531-916607acdf92" (UID: "012ffa82-136c-4072-9531-916607acdf92"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 12:14:05.306809 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:14:05.306788    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/012ffa82-136c-4072-9531-916607acdf92-proxy-tls\") on node \"ip-10-0-136-227.ec2.internal\" DevicePath \"\""
May 25 12:14:05.306809 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:14:05.306810    2561 reconciler_common.go:299] "Volume detached for volume \"isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/012ffa82-136c-4072-9531-916607acdf92-isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-227.ec2.internal\" DevicePath \"\""
May 25 12:14:05.306942 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:14:05.306820    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7bv2b\" (UniqueName: \"kubernetes.io/projected/012ffa82-136c-4072-9531-916607acdf92-kube-api-access-7bv2b\") on node \"ip-10-0-136-227.ec2.internal\" DevicePath \"\""
May 25 12:14:05.306942 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:14:05.306829    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/012ffa82-136c-4072-9531-916607acdf92-kserve-provision-location\") on node \"ip-10-0-136-227.ec2.internal\" DevicePath \"\""
May 25 12:14:05.892950 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:14:05.892918    2561 generic.go:358] "Generic (PLEG): container finished" podID="012ffa82-136c-4072-9531-916607acdf92" containerID="212233f3fe85f4a72c75dc6c05ff4c7c13f95af9ab283bd81d53d4b7bc6e4a04" exitCode=0
May 25 12:14:05.893086 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:14:05.892966    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-mctcc" event={"ID":"012ffa82-136c-4072-9531-916607acdf92","Type":"ContainerDied","Data":"212233f3fe85f4a72c75dc6c05ff4c7c13f95af9ab283bd81d53d4b7bc6e4a04"}
May 25 12:14:05.893086 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:14:05.893009    2561 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-mctcc"
May 25 12:14:05.893086 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:14:05.893000    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-mctcc" event={"ID":"012ffa82-136c-4072-9531-916607acdf92","Type":"ContainerDied","Data":"d44f242adb6085a060d58eed42c566d89afb08a01de77ebe6a35e6cd4d060b96"}
May 25 12:14:05.893086 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:14:05.893039    2561 scope.go:117] "RemoveContainer" containerID="97e20a09dc0f92d36e2ec7e29bda6aa61e6c6c8f9ff27523494e67cb1f681433"
May 25 12:14:05.902128 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:14:05.902108    2561 scope.go:117] "RemoveContainer" containerID="212233f3fe85f4a72c75dc6c05ff4c7c13f95af9ab283bd81d53d4b7bc6e4a04"
May 25 12:14:05.909667 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:14:05.909644    2561 scope.go:117] "RemoveContainer" containerID="eb4e95cf2ce6245f1ed6b9574ae943fb66ceea27d553536238c499b1d75b65af"
May 25 12:14:05.916317 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:14:05.916292    2561 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-mctcc"]
May 25 12:14:05.916792 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:14:05.916774    2561 scope.go:117] "RemoveContainer" containerID="97e20a09dc0f92d36e2ec7e29bda6aa61e6c6c8f9ff27523494e67cb1f681433"
May 25 12:14:05.917025 ip-10-0-136-227 kubenswrapper[2561]: E0525 12:14:05.917005    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97e20a09dc0f92d36e2ec7e29bda6aa61e6c6c8f9ff27523494e67cb1f681433\": container with ID starting with 97e20a09dc0f92d36e2ec7e29bda6aa61e6c6c8f9ff27523494e67cb1f681433 not found: ID does not exist" containerID="97e20a09dc0f92d36e2ec7e29bda6aa61e6c6c8f9ff27523494e67cb1f681433"
May 25 12:14:05.917070 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:14:05.917034    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97e20a09dc0f92d36e2ec7e29bda6aa61e6c6c8f9ff27523494e67cb1f681433"} err="failed to get container status \"97e20a09dc0f92d36e2ec7e29bda6aa61e6c6c8f9ff27523494e67cb1f681433\": rpc error: code = NotFound desc = could not find container \"97e20a09dc0f92d36e2ec7e29bda6aa61e6c6c8f9ff27523494e67cb1f681433\": container with ID starting with 97e20a09dc0f92d36e2ec7e29bda6aa61e6c6c8f9ff27523494e67cb1f681433 not found: ID does not exist"
May 25 12:14:05.917070 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:14:05.917051    2561 scope.go:117] "RemoveContainer" containerID="212233f3fe85f4a72c75dc6c05ff4c7c13f95af9ab283bd81d53d4b7bc6e4a04"
May 25 12:14:05.917336 ip-10-0-136-227 kubenswrapper[2561]: E0525 12:14:05.917298    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"212233f3fe85f4a72c75dc6c05ff4c7c13f95af9ab283bd81d53d4b7bc6e4a04\": container with ID starting with 212233f3fe85f4a72c75dc6c05ff4c7c13f95af9ab283bd81d53d4b7bc6e4a04 not found: ID does not exist" containerID="212233f3fe85f4a72c75dc6c05ff4c7c13f95af9ab283bd81d53d4b7bc6e4a04"
May 25 12:14:05.917391 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:14:05.917329    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"212233f3fe85f4a72c75dc6c05ff4c7c13f95af9ab283bd81d53d4b7bc6e4a04"} err="failed to get container status \"212233f3fe85f4a72c75dc6c05ff4c7c13f95af9ab283bd81d53d4b7bc6e4a04\": rpc error: code = NotFound desc = could not find container \"212233f3fe85f4a72c75dc6c05ff4c7c13f95af9ab283bd81d53d4b7bc6e4a04\": container with ID starting with 212233f3fe85f4a72c75dc6c05ff4c7c13f95af9ab283bd81d53d4b7bc6e4a04 not found: ID does not exist"
May 25 12:14:05.917391 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:14:05.917351    2561 scope.go:117] "RemoveContainer" containerID="eb4e95cf2ce6245f1ed6b9574ae943fb66ceea27d553536238c499b1d75b65af"
May 25 12:14:05.917985 ip-10-0-136-227 kubenswrapper[2561]: E0525 12:14:05.917683    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb4e95cf2ce6245f1ed6b9574ae943fb66ceea27d553536238c499b1d75b65af\": container with ID starting with eb4e95cf2ce6245f1ed6b9574ae943fb66ceea27d553536238c499b1d75b65af not found: ID does not exist" containerID="eb4e95cf2ce6245f1ed6b9574ae943fb66ceea27d553536238c499b1d75b65af"
May 25 12:14:05.917985 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:14:05.917715    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb4e95cf2ce6245f1ed6b9574ae943fb66ceea27d553536238c499b1d75b65af"} err="failed to get container status \"eb4e95cf2ce6245f1ed6b9574ae943fb66ceea27d553536238c499b1d75b65af\": rpc error: code = NotFound desc = could not find container \"eb4e95cf2ce6245f1ed6b9574ae943fb66ceea27d553536238c499b1d75b65af\": container with ID starting with eb4e95cf2ce6245f1ed6b9574ae943fb66ceea27d553536238c499b1d75b65af not found: ID does not exist"
May 25 12:14:05.920193 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:14:05.920175    2561 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-6dc5954dc-mctcc"]
May 25 12:14:07.117769 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:14:07.117733    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="012ffa82-136c-4072-9531-916607acdf92" path="/var/lib/kubelet/pods/012ffa82-136c-4072-9531-916607acdf92/volumes"
May 25 12:15:08.890515 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:15:08.890478    2561 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7cbbdf44db-qbbth"]
May 25 12:15:08.890983 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:15:08.890816    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="012ffa82-136c-4072-9531-916607acdf92" containerName="kube-rbac-proxy"
May 25 12:15:08.890983 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:15:08.890827    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="012ffa82-136c-4072-9531-916607acdf92" containerName="kube-rbac-proxy"
May 25 12:15:08.890983 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:15:08.890848    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="012ffa82-136c-4072-9531-916607acdf92" containerName="storage-initializer"
May 25 12:15:08.890983 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:15:08.890857    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="012ffa82-136c-4072-9531-916607acdf92" containerName="storage-initializer"
May 25 12:15:08.890983 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:15:08.890879    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="012ffa82-136c-4072-9531-916607acdf92" containerName="kserve-container"
May 25 12:15:08.890983 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:15:08.890898    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="012ffa82-136c-4072-9531-916607acdf92" containerName="kserve-container"
May 25 12:15:08.890983 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:15:08.890972    2561 memory_manager.go:356] "RemoveStaleState removing state" podUID="012ffa82-136c-4072-9531-916607acdf92" containerName="kube-rbac-proxy"
May 25 12:15:08.890983 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:15:08.890984    2561 memory_manager.go:356] "RemoveStaleState removing state" podUID="012ffa82-136c-4072-9531-916607acdf92" containerName="kserve-container"
May 25 12:15:08.894279 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:15:08.894240    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7cbbdf44db-qbbth"
May 25 12:15:08.897223 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:15:08.897204    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
May 25 12:15:08.897349 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:15:08.897322    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
May 25 12:15:08.897565 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:15:08.897548    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
May 25 12:15:08.897648 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:15:08.897549    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-kube-rbac-proxy-sar-config\""
May 25 12:15:08.897648 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:15:08.897577    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-predictor-serving-cert\""
May 25 12:15:08.897648 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:15:08.897604    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-svs62\""
May 25 12:15:08.906215 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:15:08.906197    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7cbbdf44db-qbbth"]
May 25 12:15:08.987345 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:15:08.987316    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2a3176d7-66c5-41ec-a9d3-80e3216ea3b2-proxy-tls\") pod \"isvc-sklearn-s3-predictor-7cbbdf44db-qbbth\" (UID: \"2a3176d7-66c5-41ec-a9d3-80e3216ea3b2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7cbbdf44db-qbbth"
May 25 12:15:08.987461 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:15:08.987352    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-sklearn-s3-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2a3176d7-66c5-41ec-a9d3-80e3216ea3b2-isvc-sklearn-s3-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-predictor-7cbbdf44db-qbbth\" (UID: \"2a3176d7-66c5-41ec-a9d3-80e3216ea3b2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7cbbdf44db-qbbth"
May 25 12:15:08.987461 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:15:08.987436    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2a3176d7-66c5-41ec-a9d3-80e3216ea3b2-kserve-provision-location\") pod \"isvc-sklearn-s3-predictor-7cbbdf44db-qbbth\" (UID: \"2a3176d7-66c5-41ec-a9d3-80e3216ea3b2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7cbbdf44db-qbbth"
May 25 12:15:08.987547 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:15:08.987488    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2fhb\" (UniqueName: \"kubernetes.io/projected/2a3176d7-66c5-41ec-a9d3-80e3216ea3b2-kube-api-access-h2fhb\") pod \"isvc-sklearn-s3-predictor-7cbbdf44db-qbbth\" (UID: \"2a3176d7-66c5-41ec-a9d3-80e3216ea3b2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7cbbdf44db-qbbth"
May 25 12:15:09.088016 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:15:09.087991    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-h2fhb\" (UniqueName: \"kubernetes.io/projected/2a3176d7-66c5-41ec-a9d3-80e3216ea3b2-kube-api-access-h2fhb\") pod \"isvc-sklearn-s3-predictor-7cbbdf44db-qbbth\" (UID: \"2a3176d7-66c5-41ec-a9d3-80e3216ea3b2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7cbbdf44db-qbbth"
May 25 12:15:09.088111 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:15:09.088057    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2a3176d7-66c5-41ec-a9d3-80e3216ea3b2-proxy-tls\") pod \"isvc-sklearn-s3-predictor-7cbbdf44db-qbbth\" (UID: \"2a3176d7-66c5-41ec-a9d3-80e3216ea3b2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7cbbdf44db-qbbth"
May 25 12:15:09.088111 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:15:09.088078    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-sklearn-s3-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2a3176d7-66c5-41ec-a9d3-80e3216ea3b2-isvc-sklearn-s3-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-predictor-7cbbdf44db-qbbth\" (UID: \"2a3176d7-66c5-41ec-a9d3-80e3216ea3b2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7cbbdf44db-qbbth"
May 25 12:15:09.088180 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:15:09.088112    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2a3176d7-66c5-41ec-a9d3-80e3216ea3b2-kserve-provision-location\") pod \"isvc-sklearn-s3-predictor-7cbbdf44db-qbbth\" (UID: \"2a3176d7-66c5-41ec-a9d3-80e3216ea3b2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7cbbdf44db-qbbth"
May 25 12:15:09.088214 ip-10-0-136-227 kubenswrapper[2561]: E0525 12:15:09.088195    2561 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-s3-predictor-serving-cert: secret "isvc-sklearn-s3-predictor-serving-cert" not found
May 25 12:15:09.088262 ip-10-0-136-227 kubenswrapper[2561]: E0525 12:15:09.088251    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a3176d7-66c5-41ec-a9d3-80e3216ea3b2-proxy-tls podName:2a3176d7-66c5-41ec-a9d3-80e3216ea3b2 nodeName:}" failed. No retries permitted until 2026-05-25 12:15:09.588236121 +0000 UTC m=+3483.133413703 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/2a3176d7-66c5-41ec-a9d3-80e3216ea3b2-proxy-tls") pod "isvc-sklearn-s3-predictor-7cbbdf44db-qbbth" (UID: "2a3176d7-66c5-41ec-a9d3-80e3216ea3b2") : secret "isvc-sklearn-s3-predictor-serving-cert" not found
May 25 12:15:09.088561 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:15:09.088543    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2a3176d7-66c5-41ec-a9d3-80e3216ea3b2-kserve-provision-location\") pod \"isvc-sklearn-s3-predictor-7cbbdf44db-qbbth\" (UID: \"2a3176d7-66c5-41ec-a9d3-80e3216ea3b2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7cbbdf44db-qbbth"
May 25 12:15:09.088786 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:15:09.088771    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-sklearn-s3-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2a3176d7-66c5-41ec-a9d3-80e3216ea3b2-isvc-sklearn-s3-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-predictor-7cbbdf44db-qbbth\" (UID: \"2a3176d7-66c5-41ec-a9d3-80e3216ea3b2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7cbbdf44db-qbbth"
May 25 12:15:09.097754 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:15:09.097730    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2fhb\" (UniqueName: \"kubernetes.io/projected/2a3176d7-66c5-41ec-a9d3-80e3216ea3b2-kube-api-access-h2fhb\") pod \"isvc-sklearn-s3-predictor-7cbbdf44db-qbbth\" (UID: \"2a3176d7-66c5-41ec-a9d3-80e3216ea3b2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7cbbdf44db-qbbth"
May 25 12:15:09.593364 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:15:09.593335    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2a3176d7-66c5-41ec-a9d3-80e3216ea3b2-proxy-tls\") pod \"isvc-sklearn-s3-predictor-7cbbdf44db-qbbth\" (UID: \"2a3176d7-66c5-41ec-a9d3-80e3216ea3b2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7cbbdf44db-qbbth"
May 25 12:15:09.595564 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:15:09.595536    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2a3176d7-66c5-41ec-a9d3-80e3216ea3b2-proxy-tls\") pod \"isvc-sklearn-s3-predictor-7cbbdf44db-qbbth\" (UID: \"2a3176d7-66c5-41ec-a9d3-80e3216ea3b2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7cbbdf44db-qbbth"
May 25 12:15:09.805531 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:15:09.805502    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7cbbdf44db-qbbth"
May 25 12:15:09.932122 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:15:09.932006    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7cbbdf44db-qbbth"]
May 25 12:15:09.935010 ip-10-0-136-227 kubenswrapper[2561]: W0525 12:15:09.934983    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a3176d7_66c5_41ec_a9d3_80e3216ea3b2.slice/crio-6bdfd656bc7a320c8aad24e22aab9e63b5c4b6d4a85cf8be29b69e60127eace9 WatchSource:0}: Error finding container 6bdfd656bc7a320c8aad24e22aab9e63b5c4b6d4a85cf8be29b69e60127eace9: Status 404 returned error can't find the container with id 6bdfd656bc7a320c8aad24e22aab9e63b5c4b6d4a85cf8be29b69e60127eace9
May 25 12:15:09.936860 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:15:09.936844    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 25 12:15:10.118800 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:15:10.118712    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7cbbdf44db-qbbth" event={"ID":"2a3176d7-66c5-41ec-a9d3-80e3216ea3b2","Type":"ContainerStarted","Data":"3df266d9cf226aea068339243d6f0d324a14805fed4fc823921081ae093a47f0"}
May 25 12:15:10.118800 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:15:10.118744    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7cbbdf44db-qbbth" event={"ID":"2a3176d7-66c5-41ec-a9d3-80e3216ea3b2","Type":"ContainerStarted","Data":"6bdfd656bc7a320c8aad24e22aab9e63b5c4b6d4a85cf8be29b69e60127eace9"}
May 25 12:15:11.122470 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:15:11.122364    2561 generic.go:358] "Generic (PLEG): container finished" podID="2a3176d7-66c5-41ec-a9d3-80e3216ea3b2" containerID="3df266d9cf226aea068339243d6f0d324a14805fed4fc823921081ae093a47f0" exitCode=0
May 25 12:15:11.122470 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:15:11.122401    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7cbbdf44db-qbbth" event={"ID":"2a3176d7-66c5-41ec-a9d3-80e3216ea3b2","Type":"ContainerDied","Data":"3df266d9cf226aea068339243d6f0d324a14805fed4fc823921081ae093a47f0"}
May 25 12:15:12.127486 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:15:12.127454    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7cbbdf44db-qbbth" event={"ID":"2a3176d7-66c5-41ec-a9d3-80e3216ea3b2","Type":"ContainerStarted","Data":"6f7e36c0d545a51890c690987dce41ea5291ea394e3892a254e41f5d77aa58a4"}
May 25 12:15:12.127486 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:15:12.127484    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7cbbdf44db-qbbth" event={"ID":"2a3176d7-66c5-41ec-a9d3-80e3216ea3b2","Type":"ContainerStarted","Data":"9db93d40c977a878b8c806af51bc2fd1be4562a55a560b5d4b60bf9d1cd302d4"}
May 25 12:15:12.127897 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:15:12.127615    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7cbbdf44db-qbbth"
May 25 12:15:12.147415 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:15:12.147356    2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7cbbdf44db-qbbth" podStartSLOduration=4.147344273 podStartE2EDuration="4.147344273s" podCreationTimestamp="2026-05-25 12:15:08 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 12:15:12.146213946 +0000 UTC m=+3485.691391549" watchObservedRunningTime="2026-05-25 12:15:12.147344273 +0000 UTC m=+3485.692521875"
May 25 12:15:13.130301 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:15:13.130251    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7cbbdf44db-qbbth"
May 25 12:15:13.131449 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:15:13.131420    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7cbbdf44db-qbbth" podUID="2a3176d7-66c5-41ec-a9d3-80e3216ea3b2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.56:8080: connect: connection refused"
May 25 12:15:14.134072 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:15:14.134028    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7cbbdf44db-qbbth" podUID="2a3176d7-66c5-41ec-a9d3-80e3216ea3b2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.56:8080: connect: connection refused"
May 25 12:15:19.138826 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:15:19.138795    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7cbbdf44db-qbbth"
May 25 12:15:19.139378 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:15:19.139350    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7cbbdf44db-qbbth" podUID="2a3176d7-66c5-41ec-a9d3-80e3216ea3b2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.56:8080: connect: connection refused"
May 25 12:15:29.139440 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:15:29.139358    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7cbbdf44db-qbbth" podUID="2a3176d7-66c5-41ec-a9d3-80e3216ea3b2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.56:8080: connect: connection refused"
May 25 12:15:39.143528 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:15:39.143489    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7cbbdf44db-qbbth" podUID="2a3176d7-66c5-41ec-a9d3-80e3216ea3b2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.56:8080: connect: connection refused"
May 25 12:15:49.139192 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:15:49.139155    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7cbbdf44db-qbbth" podUID="2a3176d7-66c5-41ec-a9d3-80e3216ea3b2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.56:8080: connect: connection refused"
May 25 12:15:59.140119 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:15:59.140081    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7cbbdf44db-qbbth" podUID="2a3176d7-66c5-41ec-a9d3-80e3216ea3b2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.56:8080: connect: connection refused"
May 25 12:16:09.139305 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:16:09.139250    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7cbbdf44db-qbbth" podUID="2a3176d7-66c5-41ec-a9d3-80e3216ea3b2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.56:8080: connect: connection refused"
May 25 12:16:19.140407 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:16:19.140379    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7cbbdf44db-qbbth"
May 25 12:16:28.978554 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:16:28.978520    2561 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7cbbdf44db-qbbth"]
May 25 12:16:28.979138 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:16:28.978912    2561 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7cbbdf44db-qbbth" podUID="2a3176d7-66c5-41ec-a9d3-80e3216ea3b2" containerName="kserve-container" containerID="cri-o://9db93d40c977a878b8c806af51bc2fd1be4562a55a560b5d4b60bf9d1cd302d4" gracePeriod=30
May 25 12:16:28.979138 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:16:28.979004    2561 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7cbbdf44db-qbbth" podUID="2a3176d7-66c5-41ec-a9d3-80e3216ea3b2" containerName="kube-rbac-proxy" containerID="cri-o://6f7e36c0d545a51890c690987dce41ea5291ea394e3892a254e41f5d77aa58a4" gracePeriod=30
May 25 12:16:29.111445 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:16:29.111417    2561 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-754b5dc854-lmdhs"]
May 25 12:16:29.115095 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:16:29.115070    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-754b5dc854-lmdhs"
May 25 12:16:29.117633 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:16:29.117610    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-pass-predictor-serving-cert\""
May 25 12:16:29.117761 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:16:29.117657    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config\""
May 25 12:16:29.117862 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:16:29.117840    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
May 25 12:16:29.124321 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:16:29.124298    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-754b5dc854-lmdhs"]
May 25 12:16:29.134555 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:16:29.134521    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7cbbdf44db-qbbth" podUID="2a3176d7-66c5-41ec-a9d3-80e3216ea3b2" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.56:8643/healthz\": dial tcp 10.132.0.56:8643: connect: connection refused"
May 25 12:16:29.139387 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:16:29.139358    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7cbbdf44db-qbbth" podUID="2a3176d7-66c5-41ec-a9d3-80e3216ea3b2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.56:8080: connect: connection refused"
May 25 12:16:29.169519 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:16:29.169482    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/93f67758-c701-4c1a-98da-8cd15668858e-isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-754b5dc854-lmdhs\" (UID: \"93f67758-c701-4c1a-98da-8cd15668858e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-754b5dc854-lmdhs"
May 25 12:16:29.169610 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:16:29.169528    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/93f67758-c701-4c1a-98da-8cd15668858e-cabundle-cert\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-754b5dc854-lmdhs\" (UID: \"93f67758-c701-4c1a-98da-8cd15668858e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-754b5dc854-lmdhs"
May 25 12:16:29.169657 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:16:29.169624    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/93f67758-c701-4c1a-98da-8cd15668858e-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-754b5dc854-lmdhs\" (UID: \"93f67758-c701-4c1a-98da-8cd15668858e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-754b5dc854-lmdhs"
May 25 12:16:29.169657 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:16:29.169651    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4zbp\" (UniqueName: \"kubernetes.io/projected/93f67758-c701-4c1a-98da-8cd15668858e-kube-api-access-q4zbp\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-754b5dc854-lmdhs\" (UID: \"93f67758-c701-4c1a-98da-8cd15668858e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-754b5dc854-lmdhs"
May 25 12:16:29.169745 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:16:29.169733    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/93f67758-c701-4c1a-98da-8cd15668858e-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-754b5dc854-lmdhs\" (UID: \"93f67758-c701-4c1a-98da-8cd15668858e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-754b5dc854-lmdhs"
May 25 12:16:29.270443 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:16:29.270387    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/93f67758-c701-4c1a-98da-8cd15668858e-cabundle-cert\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-754b5dc854-lmdhs\" (UID: \"93f67758-c701-4c1a-98da-8cd15668858e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-754b5dc854-lmdhs"
May 25 12:16:29.270537 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:16:29.270446    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/93f67758-c701-4c1a-98da-8cd15668858e-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-754b5dc854-lmdhs\" (UID: \"93f67758-c701-4c1a-98da-8cd15668858e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-754b5dc854-lmdhs"
May 25 12:16:29.270537 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:16:29.270465    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-q4zbp\" (UniqueName: \"kubernetes.io/projected/93f67758-c701-4c1a-98da-8cd15668858e-kube-api-access-q4zbp\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-754b5dc854-lmdhs\" (UID: \"93f67758-c701-4c1a-98da-8cd15668858e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-754b5dc854-lmdhs"
May 25 12:16:29.270537 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:16:29.270527    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/93f67758-c701-4c1a-98da-8cd15668858e-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-754b5dc854-lmdhs\" (UID: \"93f67758-c701-4c1a-98da-8cd15668858e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-754b5dc854-lmdhs"
May 25 12:16:29.270695 ip-10-0-136-227 kubenswrapper[2561]: E0525 12:16:29.270558    2561 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-serving-cert: secret "isvc-sklearn-s3-tls-global-pass-predictor-serving-cert" not found
May 25 12:16:29.270695 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:16:29.270563    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/93f67758-c701-4c1a-98da-8cd15668858e-isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-754b5dc854-lmdhs\" (UID: \"93f67758-c701-4c1a-98da-8cd15668858e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-754b5dc854-lmdhs"
May 25 12:16:29.270695 ip-10-0-136-227 kubenswrapper[2561]: E0525 12:16:29.270620    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/93f67758-c701-4c1a-98da-8cd15668858e-proxy-tls podName:93f67758-c701-4c1a-98da-8cd15668858e nodeName:}" failed. No retries permitted until 2026-05-25 12:16:29.770599954 +0000 UTC m=+3563.315777547 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/93f67758-c701-4c1a-98da-8cd15668858e-proxy-tls") pod "isvc-sklearn-s3-tls-global-pass-predictor-754b5dc854-lmdhs" (UID: "93f67758-c701-4c1a-98da-8cd15668858e") : secret "isvc-sklearn-s3-tls-global-pass-predictor-serving-cert" not found
May 25 12:16:29.270934 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:16:29.270916    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/93f67758-c701-4c1a-98da-8cd15668858e-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-754b5dc854-lmdhs\" (UID: \"93f67758-c701-4c1a-98da-8cd15668858e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-754b5dc854-lmdhs"
May 25 12:16:29.271044 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:16:29.271004    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/93f67758-c701-4c1a-98da-8cd15668858e-cabundle-cert\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-754b5dc854-lmdhs\" (UID: \"93f67758-c701-4c1a-98da-8cd15668858e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-754b5dc854-lmdhs"
May 25 12:16:29.271083 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:16:29.271071    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/93f67758-c701-4c1a-98da-8cd15668858e-isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-754b5dc854-lmdhs\" (UID: \"93f67758-c701-4c1a-98da-8cd15668858e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-754b5dc854-lmdhs"
May 25 12:16:29.281741 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:16:29.281715    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4zbp\" (UniqueName: \"kubernetes.io/projected/93f67758-c701-4c1a-98da-8cd15668858e-kube-api-access-q4zbp\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-754b5dc854-lmdhs\" (UID: \"93f67758-c701-4c1a-98da-8cd15668858e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-754b5dc854-lmdhs"
May 25 12:16:29.372382 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:16:29.372357    2561 generic.go:358] "Generic (PLEG): container finished" podID="2a3176d7-66c5-41ec-a9d3-80e3216ea3b2" containerID="6f7e36c0d545a51890c690987dce41ea5291ea394e3892a254e41f5d77aa58a4" exitCode=2
May 25 12:16:29.372469 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:16:29.372431    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7cbbdf44db-qbbth" event={"ID":"2a3176d7-66c5-41ec-a9d3-80e3216ea3b2","Type":"ContainerDied","Data":"6f7e36c0d545a51890c690987dce41ea5291ea394e3892a254e41f5d77aa58a4"}
May 25 12:16:29.774309 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:16:29.774253    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/93f67758-c701-4c1a-98da-8cd15668858e-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-754b5dc854-lmdhs\" (UID: \"93f67758-c701-4c1a-98da-8cd15668858e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-754b5dc854-lmdhs"
May 25 12:16:29.776644 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:16:29.776625    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/93f67758-c701-4c1a-98da-8cd15668858e-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-754b5dc854-lmdhs\" (UID: \"93f67758-c701-4c1a-98da-8cd15668858e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-754b5dc854-lmdhs"
May 25 12:16:30.026385 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:16:30.026308    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-754b5dc854-lmdhs"
May 25 12:16:30.154251 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:16:30.154219    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-754b5dc854-lmdhs"]
May 25 12:16:30.156576 ip-10-0-136-227 kubenswrapper[2561]: W0525 12:16:30.156542    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93f67758_c701_4c1a_98da_8cd15668858e.slice/crio-90922510307a335a0a6eb369144bb3b5b8e00268417b5cc2a0604f9be7251eb3 WatchSource:0}: Error finding container 90922510307a335a0a6eb369144bb3b5b8e00268417b5cc2a0604f9be7251eb3: Status 404 returned error can't find the container with id 90922510307a335a0a6eb369144bb3b5b8e00268417b5cc2a0604f9be7251eb3
May 25 12:16:30.376949 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:16:30.376908    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-754b5dc854-lmdhs" event={"ID":"93f67758-c701-4c1a-98da-8cd15668858e","Type":"ContainerStarted","Data":"18d225e83120249d65a0373b4b8507c14957be716f95474b7f0f76c78667a0cc"}
May 25 12:16:30.376949 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:16:30.376950    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-754b5dc854-lmdhs" event={"ID":"93f67758-c701-4c1a-98da-8cd15668858e","Type":"ContainerStarted","Data":"90922510307a335a0a6eb369144bb3b5b8e00268417b5cc2a0604f9be7251eb3"}
May 25 12:16:31.381866 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:16:31.381835    2561 generic.go:358] "Generic (PLEG): container finished" podID="93f67758-c701-4c1a-98da-8cd15668858e" containerID="18d225e83120249d65a0373b4b8507c14957be716f95474b7f0f76c78667a0cc" exitCode=0
May 25 12:16:31.382195 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:16:31.381890    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-754b5dc854-lmdhs" event={"ID":"93f67758-c701-4c1a-98da-8cd15668858e","Type":"ContainerDied","Data":"18d225e83120249d65a0373b4b8507c14957be716f95474b7f0f76c78667a0cc"}
May 25 12:16:32.387587 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:16:32.387545    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-754b5dc854-lmdhs" event={"ID":"93f67758-c701-4c1a-98da-8cd15668858e","Type":"ContainerStarted","Data":"e1012f262bd7cfe55e2ceee69203b384397a224ac9cc8ade6429930046388a79"}
May 25 12:16:32.387587 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:16:32.387580    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-754b5dc854-lmdhs" event={"ID":"93f67758-c701-4c1a-98da-8cd15668858e","Type":"ContainerStarted","Data":"43e9418b3000c1fabc67fc426873ef34ee59baab776f8fa7d8a73f9b85b8231a"}
May 25 12:16:32.388000 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:16:32.387763    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-754b5dc854-lmdhs"
May 25 12:16:32.388000 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:16:32.387792    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-754b5dc854-lmdhs"
May 25 12:16:32.389097 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:16:32.389073    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-754b5dc854-lmdhs" podUID="93f67758-c701-4c1a-98da-8cd15668858e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.57:8080: connect: connection refused"
May 25 12:16:32.413102 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:16:32.413036    2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-754b5dc854-lmdhs" podStartSLOduration=3.413024827 podStartE2EDuration="3.413024827s" podCreationTimestamp="2026-05-25 12:16:29 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 12:16:32.409224797 +0000 UTC m=+3565.954402411" watchObservedRunningTime="2026-05-25 12:16:32.413024827 +0000 UTC m=+3565.958202424"
May 25 12:16:32.725945 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:16:32.725922    2561 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7cbbdf44db-qbbth"
May 25 12:16:32.800955 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:16:32.800929    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2a3176d7-66c5-41ec-a9d3-80e3216ea3b2-proxy-tls\") pod \"2a3176d7-66c5-41ec-a9d3-80e3216ea3b2\" (UID: \"2a3176d7-66c5-41ec-a9d3-80e3216ea3b2\") "
May 25 12:16:32.801112 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:16:32.800965    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-sklearn-s3-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2a3176d7-66c5-41ec-a9d3-80e3216ea3b2-isvc-sklearn-s3-kube-rbac-proxy-sar-config\") pod \"2a3176d7-66c5-41ec-a9d3-80e3216ea3b2\" (UID: \"2a3176d7-66c5-41ec-a9d3-80e3216ea3b2\") "
May 25 12:16:32.801112 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:16:32.801081    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2a3176d7-66c5-41ec-a9d3-80e3216ea3b2-kserve-provision-location\") pod \"2a3176d7-66c5-41ec-a9d3-80e3216ea3b2\" (UID: \"2a3176d7-66c5-41ec-a9d3-80e3216ea3b2\") "
May 25 12:16:32.801112 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:16:32.801105    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h2fhb\" (UniqueName: \"kubernetes.io/projected/2a3176d7-66c5-41ec-a9d3-80e3216ea3b2-kube-api-access-h2fhb\") pod \"2a3176d7-66c5-41ec-a9d3-80e3216ea3b2\" (UID: \"2a3176d7-66c5-41ec-a9d3-80e3216ea3b2\") "
May 25 12:16:32.801379 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:16:32.801356    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a3176d7-66c5-41ec-a9d3-80e3216ea3b2-isvc-sklearn-s3-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-sklearn-s3-kube-rbac-proxy-sar-config") pod "2a3176d7-66c5-41ec-a9d3-80e3216ea3b2" (UID: "2a3176d7-66c5-41ec-a9d3-80e3216ea3b2"). InnerVolumeSpecName "isvc-sklearn-s3-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 12:16:32.801437 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:16:32.801422    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a3176d7-66c5-41ec-a9d3-80e3216ea3b2-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2a3176d7-66c5-41ec-a9d3-80e3216ea3b2" (UID: "2a3176d7-66c5-41ec-a9d3-80e3216ea3b2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 12:16:32.803008 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:16:32.802987    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a3176d7-66c5-41ec-a9d3-80e3216ea3b2-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2a3176d7-66c5-41ec-a9d3-80e3216ea3b2" (UID: "2a3176d7-66c5-41ec-a9d3-80e3216ea3b2"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 12:16:32.803128 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:16:32.803109    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a3176d7-66c5-41ec-a9d3-80e3216ea3b2-kube-api-access-h2fhb" (OuterVolumeSpecName: "kube-api-access-h2fhb") pod "2a3176d7-66c5-41ec-a9d3-80e3216ea3b2" (UID: "2a3176d7-66c5-41ec-a9d3-80e3216ea3b2"). InnerVolumeSpecName "kube-api-access-h2fhb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 12:16:32.901944 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:16:32.901886    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2a3176d7-66c5-41ec-a9d3-80e3216ea3b2-proxy-tls\") on node \"ip-10-0-136-227.ec2.internal\" DevicePath \"\""
May 25 12:16:32.901944 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:16:32.901907    2561 reconciler_common.go:299] "Volume detached for volume \"isvc-sklearn-s3-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2a3176d7-66c5-41ec-a9d3-80e3216ea3b2-isvc-sklearn-s3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-227.ec2.internal\" DevicePath \"\""
May 25 12:16:32.901944 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:16:32.901917    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2a3176d7-66c5-41ec-a9d3-80e3216ea3b2-kserve-provision-location\") on node \"ip-10-0-136-227.ec2.internal\" DevicePath \"\""
May 25 12:16:32.901944 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:16:32.901928    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2fhb\" (UniqueName: \"kubernetes.io/projected/2a3176d7-66c5-41ec-a9d3-80e3216ea3b2-kube-api-access-h2fhb\") on node \"ip-10-0-136-227.ec2.internal\" DevicePath \"\""
May 25 12:16:33.392847 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:16:33.392814    2561 generic.go:358] "Generic (PLEG): container finished" podID="2a3176d7-66c5-41ec-a9d3-80e3216ea3b2" containerID="9db93d40c977a878b8c806af51bc2fd1be4562a55a560b5d4b60bf9d1cd302d4" exitCode=0
May 25 12:16:33.393229 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:16:33.392892    2561 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7cbbdf44db-qbbth"
May 25 12:16:33.393229 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:16:33.392901    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7cbbdf44db-qbbth" event={"ID":"2a3176d7-66c5-41ec-a9d3-80e3216ea3b2","Type":"ContainerDied","Data":"9db93d40c977a878b8c806af51bc2fd1be4562a55a560b5d4b60bf9d1cd302d4"}
May 25 12:16:33.393229 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:16:33.392935    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7cbbdf44db-qbbth" event={"ID":"2a3176d7-66c5-41ec-a9d3-80e3216ea3b2","Type":"ContainerDied","Data":"6bdfd656bc7a320c8aad24e22aab9e63b5c4b6d4a85cf8be29b69e60127eace9"}
May 25 12:16:33.393229 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:16:33.392951    2561 scope.go:117] "RemoveContainer" containerID="6f7e36c0d545a51890c690987dce41ea5291ea394e3892a254e41f5d77aa58a4"
May 25 12:16:33.393468 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:16:33.393380    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-754b5dc854-lmdhs" podUID="93f67758-c701-4c1a-98da-8cd15668858e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.57:8080: connect: connection refused"
May 25 12:16:33.400980 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:16:33.400963    2561 scope.go:117] "RemoveContainer" containerID="9db93d40c977a878b8c806af51bc2fd1be4562a55a560b5d4b60bf9d1cd302d4"
May 25 12:16:33.407966 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:16:33.407949    2561 scope.go:117] "RemoveContainer" containerID="3df266d9cf226aea068339243d6f0d324a14805fed4fc823921081ae093a47f0"
May 25 12:16:33.412095 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:16:33.412073    2561 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7cbbdf44db-qbbth"]
May 25 12:16:33.415612 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:16:33.415592    2561 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-7cbbdf44db-qbbth"]
May 25 12:16:33.416207 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:16:33.416190    2561 scope.go:117] "RemoveContainer" containerID="6f7e36c0d545a51890c690987dce41ea5291ea394e3892a254e41f5d77aa58a4"
May 25 12:16:33.416471 ip-10-0-136-227 kubenswrapper[2561]: E0525 12:16:33.416446    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f7e36c0d545a51890c690987dce41ea5291ea394e3892a254e41f5d77aa58a4\": container with ID starting with 6f7e36c0d545a51890c690987dce41ea5291ea394e3892a254e41f5d77aa58a4 not found: ID does not exist" containerID="6f7e36c0d545a51890c690987dce41ea5291ea394e3892a254e41f5d77aa58a4"
May 25 12:16:33.416569 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:16:33.416479    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f7e36c0d545a51890c690987dce41ea5291ea394e3892a254e41f5d77aa58a4"} err="failed to get container status \"6f7e36c0d545a51890c690987dce41ea5291ea394e3892a254e41f5d77aa58a4\": rpc error: code = NotFound desc = could not find container \"6f7e36c0d545a51890c690987dce41ea5291ea394e3892a254e41f5d77aa58a4\": container with ID starting with 6f7e36c0d545a51890c690987dce41ea5291ea394e3892a254e41f5d77aa58a4 not found: ID does not exist"
May 25 12:16:33.416569 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:16:33.416504    2561 scope.go:117] "RemoveContainer" containerID="9db93d40c977a878b8c806af51bc2fd1be4562a55a560b5d4b60bf9d1cd302d4"
May 25 12:16:33.416707 ip-10-0-136-227 kubenswrapper[2561]: E0525 12:16:33.416696    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9db93d40c977a878b8c806af51bc2fd1be4562a55a560b5d4b60bf9d1cd302d4\": container with ID starting with 9db93d40c977a878b8c806af51bc2fd1be4562a55a560b5d4b60bf9d1cd302d4 not found: ID does not exist" containerID="9db93d40c977a878b8c806af51bc2fd1be4562a55a560b5d4b60bf9d1cd302d4"
May 25 12:16:33.416749 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:16:33.416714    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9db93d40c977a878b8c806af51bc2fd1be4562a55a560b5d4b60bf9d1cd302d4"} err="failed to get container status \"9db93d40c977a878b8c806af51bc2fd1be4562a55a560b5d4b60bf9d1cd302d4\": rpc error: code = NotFound desc = could not find container \"9db93d40c977a878b8c806af51bc2fd1be4562a55a560b5d4b60bf9d1cd302d4\": container with ID starting with 9db93d40c977a878b8c806af51bc2fd1be4562a55a560b5d4b60bf9d1cd302d4 not found: ID does not exist"
May 25 12:16:33.416749 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:16:33.416730    2561 scope.go:117] "RemoveContainer" containerID="3df266d9cf226aea068339243d6f0d324a14805fed4fc823921081ae093a47f0"
May 25 12:16:33.416962 ip-10-0-136-227 kubenswrapper[2561]: E0525 12:16:33.416945    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3df266d9cf226aea068339243d6f0d324a14805fed4fc823921081ae093a47f0\": container with ID starting with 3df266d9cf226aea068339243d6f0d324a14805fed4fc823921081ae093a47f0 not found: ID does not exist" containerID="3df266d9cf226aea068339243d6f0d324a14805fed4fc823921081ae093a47f0"
May 25 12:16:33.417018 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:16:33.416969    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3df266d9cf226aea068339243d6f0d324a14805fed4fc823921081ae093a47f0"} err="failed to get container status \"3df266d9cf226aea068339243d6f0d324a14805fed4fc823921081ae093a47f0\": rpc error: code = NotFound desc = could not find container \"3df266d9cf226aea068339243d6f0d324a14805fed4fc823921081ae093a47f0\": container with ID starting with 3df266d9cf226aea068339243d6f0d324a14805fed4fc823921081ae093a47f0 not found: ID does not exist"
May 25 12:16:35.117964 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:16:35.117933    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2a3176d7-66c5-41ec-a9d3-80e3216ea3b2" path="/var/lib/kubelet/pods/2a3176d7-66c5-41ec-a9d3-80e3216ea3b2/volumes"
May 25 12:16:38.398059 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:16:38.398030    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-754b5dc854-lmdhs"
May 25 12:16:38.398636 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:16:38.398612    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-754b5dc854-lmdhs" podUID="93f67758-c701-4c1a-98da-8cd15668858e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.57:8080: connect: connection refused"
May 25 12:16:48.399099 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:16:48.399053    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-754b5dc854-lmdhs" podUID="93f67758-c701-4c1a-98da-8cd15668858e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.57:8080: connect: connection refused"
May 25 12:16:58.399090 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:16:58.399007    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-754b5dc854-lmdhs" podUID="93f67758-c701-4c1a-98da-8cd15668858e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.57:8080: connect: connection refused"
May 25 12:17:07.267847 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:17:07.267818    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-9lj95_10a66344-3304-49a1-bbc1-b885800f5acf/console-operator/1.log"
May 25 12:17:07.275078 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:17:07.275059    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-9lj95_10a66344-3304-49a1-bbc1-b885800f5acf/console-operator/1.log"
May 25 12:17:08.399247 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:17:08.399207    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-754b5dc854-lmdhs" podUID="93f67758-c701-4c1a-98da-8cd15668858e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.57:8080: connect: connection refused"
May 25 12:17:18.398606 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:17:18.398570    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-754b5dc854-lmdhs" podUID="93f67758-c701-4c1a-98da-8cd15668858e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.57:8080: connect: connection refused"
May 25 12:17:28.399312 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:17:28.399260    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-754b5dc854-lmdhs" podUID="93f67758-c701-4c1a-98da-8cd15668858e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.57:8080: connect: connection refused"
May 25 12:17:38.399458 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:17:38.399428    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-754b5dc854-lmdhs"
May 25 12:17:39.143261 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:17:39.143227    2561 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-754b5dc854-lmdhs"]
May 25 12:17:39.143587 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:17:39.143562    2561 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-754b5dc854-lmdhs" podUID="93f67758-c701-4c1a-98da-8cd15668858e" containerName="kserve-container" containerID="cri-o://43e9418b3000c1fabc67fc426873ef34ee59baab776f8fa7d8a73f9b85b8231a" gracePeriod=30
May 25 12:17:39.143683 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:17:39.143611    2561 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-754b5dc854-lmdhs" podUID="93f67758-c701-4c1a-98da-8cd15668858e" containerName="kube-rbac-proxy" containerID="cri-o://e1012f262bd7cfe55e2ceee69203b384397a224ac9cc8ade6429930046388a79" gracePeriod=30
May 25 12:17:39.611465 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:17:39.611436    2561 generic.go:358] "Generic (PLEG): container finished" podID="93f67758-c701-4c1a-98da-8cd15668858e" containerID="e1012f262bd7cfe55e2ceee69203b384397a224ac9cc8ade6429930046388a79" exitCode=2
May 25 12:17:39.611799 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:17:39.611475    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-754b5dc854-lmdhs" event={"ID":"93f67758-c701-4c1a-98da-8cd15668858e","Type":"ContainerDied","Data":"e1012f262bd7cfe55e2ceee69203b384397a224ac9cc8ade6429930046388a79"}
May 25 12:17:40.220661 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:17:40.220625    2561 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7cf8df44c6-mssq5"]
May 25 12:17:40.221001 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:17:40.220988    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a3176d7-66c5-41ec-a9d3-80e3216ea3b2" containerName="kube-rbac-proxy"
May 25 12:17:40.221070 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:17:40.221002    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a3176d7-66c5-41ec-a9d3-80e3216ea3b2" containerName="kube-rbac-proxy"
May 25 12:17:40.221070 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:17:40.221018    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a3176d7-66c5-41ec-a9d3-80e3216ea3b2" containerName="kserve-container"
May 25 12:17:40.221070 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:17:40.221023    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a3176d7-66c5-41ec-a9d3-80e3216ea3b2" containerName="kserve-container"
May 25 12:17:40.221070 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:17:40.221035    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a3176d7-66c5-41ec-a9d3-80e3216ea3b2" containerName="storage-initializer"
May 25 12:17:40.221070 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:17:40.221043    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a3176d7-66c5-41ec-a9d3-80e3216ea3b2" containerName="storage-initializer"
May 25 12:17:40.221522 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:17:40.221152    2561 memory_manager.go:356] "RemoveStaleState removing state" podUID="2a3176d7-66c5-41ec-a9d3-80e3216ea3b2" containerName="kube-rbac-proxy"
May 25 12:17:40.221522 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:17:40.221167    2561 memory_manager.go:356] "RemoveStaleState removing state" podUID="2a3176d7-66c5-41ec-a9d3-80e3216ea3b2" containerName="kserve-container"
May 25 12:17:40.224617 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:17:40.224598    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7cf8df44c6-mssq5"
May 25 12:17:40.226901 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:17:40.226875    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config\""
May 25 12:17:40.227000 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:17:40.226878    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-fail-predictor-serving-cert\""
May 25 12:17:40.235607 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:17:40.235588    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7cf8df44c6-mssq5"]
May 25 12:17:40.316889 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:17:40.316865    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/10e6245c-3cae-4bcc-9532-5c9d67d98210-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-7cf8df44c6-mssq5\" (UID: \"10e6245c-3cae-4bcc-9532-5c9d67d98210\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7cf8df44c6-mssq5"
May 25 12:17:40.317004 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:17:40.316974    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/10e6245c-3cae-4bcc-9532-5c9d67d98210-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-7cf8df44c6-mssq5\" (UID: \"10e6245c-3cae-4bcc-9532-5c9d67d98210\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7cf8df44c6-mssq5"
May 25 12:17:40.317070 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:17:40.317011    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rd8jk\" (UniqueName: \"kubernetes.io/projected/10e6245c-3cae-4bcc-9532-5c9d67d98210-kube-api-access-rd8jk\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-7cf8df44c6-mssq5\" (UID: \"10e6245c-3cae-4bcc-9532-5c9d67d98210\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7cf8df44c6-mssq5"
May 25 12:17:40.317131 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:17:40.317114    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/10e6245c-3cae-4bcc-9532-5c9d67d98210-isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-7cf8df44c6-mssq5\" (UID: \"10e6245c-3cae-4bcc-9532-5c9d67d98210\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7cf8df44c6-mssq5"
May 25 12:17:40.417652 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:17:40.417614    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/10e6245c-3cae-4bcc-9532-5c9d67d98210-isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-7cf8df44c6-mssq5\" (UID: \"10e6245c-3cae-4bcc-9532-5c9d67d98210\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7cf8df44c6-mssq5"
May 25 12:17:40.417652 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:17:40.417659    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/10e6245c-3cae-4bcc-9532-5c9d67d98210-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-7cf8df44c6-mssq5\" (UID: \"10e6245c-3cae-4bcc-9532-5c9d67d98210\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7cf8df44c6-mssq5"
May 25 12:17:40.417848 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:17:40.417711    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/10e6245c-3cae-4bcc-9532-5c9d67d98210-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-7cf8df44c6-mssq5\" (UID: \"10e6245c-3cae-4bcc-9532-5c9d67d98210\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7cf8df44c6-mssq5"
May 25 12:17:40.417848 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:17:40.417734    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-rd8jk\" (UniqueName: \"kubernetes.io/projected/10e6245c-3cae-4bcc-9532-5c9d67d98210-kube-api-access-rd8jk\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-7cf8df44c6-mssq5\" (UID: \"10e6245c-3cae-4bcc-9532-5c9d67d98210\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7cf8df44c6-mssq5"
May 25 12:17:40.417848 ip-10-0-136-227 kubenswrapper[2561]: E0525 12:17:40.417831    2561 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-serving-cert: secret "isvc-sklearn-s3-tls-global-fail-predictor-serving-cert" not found
May 25 12:17:40.418016 ip-10-0-136-227 kubenswrapper[2561]: E0525 12:17:40.417885    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/10e6245c-3cae-4bcc-9532-5c9d67d98210-proxy-tls podName:10e6245c-3cae-4bcc-9532-5c9d67d98210 nodeName:}" failed. No retries permitted until 2026-05-25 12:17:40.917869834 +0000 UTC m=+3634.463047415 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/10e6245c-3cae-4bcc-9532-5c9d67d98210-proxy-tls") pod "isvc-sklearn-s3-tls-global-fail-predictor-7cf8df44c6-mssq5" (UID: "10e6245c-3cae-4bcc-9532-5c9d67d98210") : secret "isvc-sklearn-s3-tls-global-fail-predictor-serving-cert" not found
May 25 12:17:40.418132 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:17:40.418111    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/10e6245c-3cae-4bcc-9532-5c9d67d98210-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-7cf8df44c6-mssq5\" (UID: \"10e6245c-3cae-4bcc-9532-5c9d67d98210\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7cf8df44c6-mssq5"
May 25 12:17:40.418365 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:17:40.418345    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/10e6245c-3cae-4bcc-9532-5c9d67d98210-isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-7cf8df44c6-mssq5\" (UID: \"10e6245c-3cae-4bcc-9532-5c9d67d98210\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7cf8df44c6-mssq5"
May 25 12:17:40.426175 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:17:40.426155    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rd8jk\" (UniqueName: \"kubernetes.io/projected/10e6245c-3cae-4bcc-9532-5c9d67d98210-kube-api-access-rd8jk\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-7cf8df44c6-mssq5\" (UID: \"10e6245c-3cae-4bcc-9532-5c9d67d98210\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7cf8df44c6-mssq5"
May 25 12:17:40.923584 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:17:40.923550    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/10e6245c-3cae-4bcc-9532-5c9d67d98210-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-7cf8df44c6-mssq5\" (UID: \"10e6245c-3cae-4bcc-9532-5c9d67d98210\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7cf8df44c6-mssq5"
May 25 12:17:40.925968 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:17:40.925935    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/10e6245c-3cae-4bcc-9532-5c9d67d98210-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-7cf8df44c6-mssq5\" (UID: \"10e6245c-3cae-4bcc-9532-5c9d67d98210\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7cf8df44c6-mssq5"
May 25 12:17:41.136106 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:17:41.136071    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7cf8df44c6-mssq5"
May 25 12:17:41.261720 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:17:41.261691    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7cf8df44c6-mssq5"]
May 25 12:17:41.264573 ip-10-0-136-227 kubenswrapper[2561]: W0525 12:17:41.264536    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10e6245c_3cae_4bcc_9532_5c9d67d98210.slice/crio-b967d4af6cdfa1caa8c009e1e34940e82f5aa30e28ba3ea3c5b4b686aae47f62 WatchSource:0}: Error finding container b967d4af6cdfa1caa8c009e1e34940e82f5aa30e28ba3ea3c5b4b686aae47f62: Status 404 returned error can't find the container with id b967d4af6cdfa1caa8c009e1e34940e82f5aa30e28ba3ea3c5b4b686aae47f62
May 25 12:17:41.619744 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:17:41.619711    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7cf8df44c6-mssq5" event={"ID":"10e6245c-3cae-4bcc-9532-5c9d67d98210","Type":"ContainerStarted","Data":"f7e143f930f0db8722f2cfc59c60dba3effec9c6e258f59c462f046ba6d8dac2"}
May 25 12:17:41.619902 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:17:41.619752    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7cf8df44c6-mssq5" event={"ID":"10e6245c-3cae-4bcc-9532-5c9d67d98210","Type":"ContainerStarted","Data":"b967d4af6cdfa1caa8c009e1e34940e82f5aa30e28ba3ea3c5b4b686aae47f62"}
May 25 12:17:42.795551 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:17:42.795530    2561 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-754b5dc854-lmdhs"
May 25 12:17:42.836358 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:17:42.836289    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q4zbp\" (UniqueName: \"kubernetes.io/projected/93f67758-c701-4c1a-98da-8cd15668858e-kube-api-access-q4zbp\") pod \"93f67758-c701-4c1a-98da-8cd15668858e\" (UID: \"93f67758-c701-4c1a-98da-8cd15668858e\") "
May 25 12:17:42.836358 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:17:42.836329    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/93f67758-c701-4c1a-98da-8cd15668858e-isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config\") pod \"93f67758-c701-4c1a-98da-8cd15668858e\" (UID: \"93f67758-c701-4c1a-98da-8cd15668858e\") "
May 25 12:17:42.836558 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:17:42.836370    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/93f67758-c701-4c1a-98da-8cd15668858e-kserve-provision-location\") pod \"93f67758-c701-4c1a-98da-8cd15668858e\" (UID: \"93f67758-c701-4c1a-98da-8cd15668858e\") "
May 25 12:17:42.836558 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:17:42.836437    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/93f67758-c701-4c1a-98da-8cd15668858e-cabundle-cert\") pod \"93f67758-c701-4c1a-98da-8cd15668858e\" (UID: \"93f67758-c701-4c1a-98da-8cd15668858e\") "
May 25 12:17:42.836558 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:17:42.836485    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/93f67758-c701-4c1a-98da-8cd15668858e-proxy-tls\") pod \"93f67758-c701-4c1a-98da-8cd15668858e\" (UID: \"93f67758-c701-4c1a-98da-8cd15668858e\") "
May 25 12:17:42.836798 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:17:42.836770    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93f67758-c701-4c1a-98da-8cd15668858e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "93f67758-c701-4c1a-98da-8cd15668858e" (UID: "93f67758-c701-4c1a-98da-8cd15668858e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 12:17:42.836877 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:17:42.836786    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93f67758-c701-4c1a-98da-8cd15668858e-isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config") pod "93f67758-c701-4c1a-98da-8cd15668858e" (UID: "93f67758-c701-4c1a-98da-8cd15668858e"). InnerVolumeSpecName "isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 12:17:42.836877 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:17:42.836820    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93f67758-c701-4c1a-98da-8cd15668858e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "93f67758-c701-4c1a-98da-8cd15668858e" (UID: "93f67758-c701-4c1a-98da-8cd15668858e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 12:17:42.838444 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:17:42.838418    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93f67758-c701-4c1a-98da-8cd15668858e-kube-api-access-q4zbp" (OuterVolumeSpecName: "kube-api-access-q4zbp") pod "93f67758-c701-4c1a-98da-8cd15668858e" (UID: "93f67758-c701-4c1a-98da-8cd15668858e"). InnerVolumeSpecName "kube-api-access-q4zbp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 12:17:42.838635 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:17:42.838611    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93f67758-c701-4c1a-98da-8cd15668858e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "93f67758-c701-4c1a-98da-8cd15668858e" (UID: "93f67758-c701-4c1a-98da-8cd15668858e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 12:17:42.937074 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:17:42.937048    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q4zbp\" (UniqueName: \"kubernetes.io/projected/93f67758-c701-4c1a-98da-8cd15668858e-kube-api-access-q4zbp\") on node \"ip-10-0-136-227.ec2.internal\" DevicePath \"\""
May 25 12:17:42.937074 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:17:42.937073    2561 reconciler_common.go:299] "Volume detached for volume \"isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/93f67758-c701-4c1a-98da-8cd15668858e-isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-227.ec2.internal\" DevicePath \"\""
May 25 12:17:42.937222 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:17:42.937084    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/93f67758-c701-4c1a-98da-8cd15668858e-kserve-provision-location\") on node \"ip-10-0-136-227.ec2.internal\" DevicePath \"\""
May 25 12:17:42.937222 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:17:42.937094    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/93f67758-c701-4c1a-98da-8cd15668858e-cabundle-cert\") on node \"ip-10-0-136-227.ec2.internal\" DevicePath \"\""
May 25 12:17:42.937222 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:17:42.937103    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/93f67758-c701-4c1a-98da-8cd15668858e-proxy-tls\") on node \"ip-10-0-136-227.ec2.internal\" DevicePath \"\""
May 25 12:17:43.627660 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:17:43.627630    2561 generic.go:358] "Generic (PLEG): container finished" podID="93f67758-c701-4c1a-98da-8cd15668858e" containerID="43e9418b3000c1fabc67fc426873ef34ee59baab776f8fa7d8a73f9b85b8231a" exitCode=0
May 25 12:17:43.627814 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:17:43.627705    2561 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-754b5dc854-lmdhs"
May 25 12:17:43.627814 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:17:43.627709    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-754b5dc854-lmdhs" event={"ID":"93f67758-c701-4c1a-98da-8cd15668858e","Type":"ContainerDied","Data":"43e9418b3000c1fabc67fc426873ef34ee59baab776f8fa7d8a73f9b85b8231a"}
May 25 12:17:43.627814 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:17:43.627748    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-754b5dc854-lmdhs" event={"ID":"93f67758-c701-4c1a-98da-8cd15668858e","Type":"ContainerDied","Data":"90922510307a335a0a6eb369144bb3b5b8e00268417b5cc2a0604f9be7251eb3"}
May 25 12:17:43.627814 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:17:43.627768    2561 scope.go:117] "RemoveContainer" containerID="e1012f262bd7cfe55e2ceee69203b384397a224ac9cc8ade6429930046388a79"
May 25 12:17:43.635426 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:17:43.635385    2561 scope.go:117] "RemoveContainer" containerID="43e9418b3000c1fabc67fc426873ef34ee59baab776f8fa7d8a73f9b85b8231a"
May 25 12:17:43.642441 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:17:43.642422    2561 scope.go:117] "RemoveContainer" containerID="18d225e83120249d65a0373b4b8507c14957be716f95474b7f0f76c78667a0cc"
May 25 12:17:43.645941 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:17:43.645920    2561 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-754b5dc854-lmdhs"]
May 25 12:17:43.650201 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:17:43.650181    2561 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-754b5dc854-lmdhs"]
May 25 12:17:43.651027 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:17:43.651013    2561 scope.go:117] "RemoveContainer" containerID="e1012f262bd7cfe55e2ceee69203b384397a224ac9cc8ade6429930046388a79"
May 25 12:17:43.651262 ip-10-0-136-227 kubenswrapper[2561]: E0525 12:17:43.651247    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1012f262bd7cfe55e2ceee69203b384397a224ac9cc8ade6429930046388a79\": container with ID starting with e1012f262bd7cfe55e2ceee69203b384397a224ac9cc8ade6429930046388a79 not found: ID does not exist" containerID="e1012f262bd7cfe55e2ceee69203b384397a224ac9cc8ade6429930046388a79"
May 25 12:17:43.651432 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:17:43.651290    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1012f262bd7cfe55e2ceee69203b384397a224ac9cc8ade6429930046388a79"} err="failed to get container status \"e1012f262bd7cfe55e2ceee69203b384397a224ac9cc8ade6429930046388a79\": rpc error: code = NotFound desc = could not find container \"e1012f262bd7cfe55e2ceee69203b384397a224ac9cc8ade6429930046388a79\": container with ID starting with e1012f262bd7cfe55e2ceee69203b384397a224ac9cc8ade6429930046388a79 not found: ID does not exist"
May 25 12:17:43.651432 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:17:43.651307    2561 scope.go:117] "RemoveContainer" containerID="43e9418b3000c1fabc67fc426873ef34ee59baab776f8fa7d8a73f9b85b8231a"
May 25 12:17:43.651555 ip-10-0-136-227 kubenswrapper[2561]: E0525 12:17:43.651530    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43e9418b3000c1fabc67fc426873ef34ee59baab776f8fa7d8a73f9b85b8231a\": container with ID starting with 43e9418b3000c1fabc67fc426873ef34ee59baab776f8fa7d8a73f9b85b8231a not found: ID does not exist" containerID="43e9418b3000c1fabc67fc426873ef34ee59baab776f8fa7d8a73f9b85b8231a"
May 25 12:17:43.651595 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:17:43.651550    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43e9418b3000c1fabc67fc426873ef34ee59baab776f8fa7d8a73f9b85b8231a"} err="failed to get container status \"43e9418b3000c1fabc67fc426873ef34ee59baab776f8fa7d8a73f9b85b8231a\": rpc error: code = NotFound desc = could not find container \"43e9418b3000c1fabc67fc426873ef34ee59baab776f8fa7d8a73f9b85b8231a\": container with ID starting with 43e9418b3000c1fabc67fc426873ef34ee59baab776f8fa7d8a73f9b85b8231a not found: ID does not exist"
May 25 12:17:43.651595 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:17:43.651566    2561 scope.go:117] "RemoveContainer" containerID="18d225e83120249d65a0373b4b8507c14957be716f95474b7f0f76c78667a0cc"
May 25 12:17:43.651801 ip-10-0-136-227 kubenswrapper[2561]: E0525 12:17:43.651782    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18d225e83120249d65a0373b4b8507c14957be716f95474b7f0f76c78667a0cc\": container with ID starting with 18d225e83120249d65a0373b4b8507c14957be716f95474b7f0f76c78667a0cc not found: ID does not exist" containerID="18d225e83120249d65a0373b4b8507c14957be716f95474b7f0f76c78667a0cc"
May 25 12:17:43.651850 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:17:43.651807    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18d225e83120249d65a0373b4b8507c14957be716f95474b7f0f76c78667a0cc"} err="failed to get container status \"18d225e83120249d65a0373b4b8507c14957be716f95474b7f0f76c78667a0cc\": rpc error: code = NotFound desc = could not find container \"18d225e83120249d65a0373b4b8507c14957be716f95474b7f0f76c78667a0cc\": container with ID starting with 18d225e83120249d65a0373b4b8507c14957be716f95474b7f0f76c78667a0cc not found: ID does not exist"
May 25 12:17:45.118028 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:17:45.117992    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="93f67758-c701-4c1a-98da-8cd15668858e" path="/var/lib/kubelet/pods/93f67758-c701-4c1a-98da-8cd15668858e/volumes"
May 25 12:17:45.635985 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:17:45.635964    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-7cf8df44c6-mssq5_10e6245c-3cae-4bcc-9532-5c9d67d98210/storage-initializer/0.log"
May 25 12:17:45.636130 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:17:45.635996    2561 generic.go:358] "Generic (PLEG): container finished" podID="10e6245c-3cae-4bcc-9532-5c9d67d98210" containerID="f7e143f930f0db8722f2cfc59c60dba3effec9c6e258f59c462f046ba6d8dac2" exitCode=1
May 25 12:17:45.636130 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:17:45.636073    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7cf8df44c6-mssq5" event={"ID":"10e6245c-3cae-4bcc-9532-5c9d67d98210","Type":"ContainerDied","Data":"f7e143f930f0db8722f2cfc59c60dba3effec9c6e258f59c462f046ba6d8dac2"}
May 25 12:17:46.641451 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:17:46.641421    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-7cf8df44c6-mssq5_10e6245c-3cae-4bcc-9532-5c9d67d98210/storage-initializer/0.log"
May 25 12:17:46.641824 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:17:46.641533    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7cf8df44c6-mssq5" event={"ID":"10e6245c-3cae-4bcc-9532-5c9d67d98210","Type":"ContainerStarted","Data":"3b75df694de5ee7aa2c15ba718875596e655c461314b7f0ff8fbc574a9b17e08"}
May 25 12:17:50.209550 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:17:50.209516    2561 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7cf8df44c6-mssq5"]
May 25 12:17:50.210357 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:17:50.210321    2561 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7cf8df44c6-mssq5" podUID="10e6245c-3cae-4bcc-9532-5c9d67d98210" containerName="storage-initializer" containerID="cri-o://3b75df694de5ee7aa2c15ba718875596e655c461314b7f0ff8fbc574a9b17e08" gracePeriod=30
May 25 12:17:51.368353 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:17:51.368318    2561 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-74788cb89d-fqbr8"]
May 25 12:17:51.368720 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:17:51.368685    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="93f67758-c701-4c1a-98da-8cd15668858e" containerName="kube-rbac-proxy"
May 25 12:17:51.368720 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:17:51.368695    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="93f67758-c701-4c1a-98da-8cd15668858e" containerName="kube-rbac-proxy"
May 25 12:17:51.368720 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:17:51.368706    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="93f67758-c701-4c1a-98da-8cd15668858e" containerName="storage-initializer"
May 25 12:17:51.368720 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:17:51.368712    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="93f67758-c701-4c1a-98da-8cd15668858e" containerName="storage-initializer"
May 25 12:17:51.368720 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:17:51.368719    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="93f67758-c701-4c1a-98da-8cd15668858e" containerName="kserve-container"
May 25 12:17:51.368893 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:17:51.368725    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="93f67758-c701-4c1a-98da-8cd15668858e" containerName="kserve-container"
May 25 12:17:51.368893 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:17:51.368777    2561 memory_manager.go:356] "RemoveStaleState removing state" podUID="93f67758-c701-4c1a-98da-8cd15668858e" containerName="kube-rbac-proxy"
May 25 12:17:51.368893 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:17:51.368788    2561 memory_manager.go:356] "RemoveStaleState removing state" podUID="93f67758-c701-4c1a-98da-8cd15668858e" containerName="kserve-container"
May 25 12:17:51.372196 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:17:51.372175    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-74788cb89d-fqbr8"
May 25 12:17:51.374621 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:17:51.374593    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-pass-predictor-serving-cert\""
May 25 12:17:51.374738 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:17:51.374638    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config\""
May 25 12:17:51.374738 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:17:51.374593    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
May 25 12:17:51.383353 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:17:51.383324    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-74788cb89d-fqbr8"]
May 25 12:17:51.407915 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:17:51.407888    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkzdd\" (UniqueName: \"kubernetes.io/projected/ad2d226a-ee9e-454c-8dee-c501ef1315e9-kube-api-access-rkzdd\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-74788cb89d-fqbr8\" (UID: \"ad2d226a-ee9e-454c-8dee-c501ef1315e9\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-74788cb89d-fqbr8"
May 25 12:17:51.408025 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:17:51.407934    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ad2d226a-ee9e-454c-8dee-c501ef1315e9-cabundle-cert\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-74788cb89d-fqbr8\" (UID: \"ad2d226a-ee9e-454c-8dee-c501ef1315e9\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-74788cb89d-fqbr8"
May 25 12:17:51.408025 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:17:51.407964    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ad2d226a-ee9e-454c-8dee-c501ef1315e9-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-74788cb89d-fqbr8\" (UID: \"ad2d226a-ee9e-454c-8dee-c501ef1315e9\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-74788cb89d-fqbr8"
May 25 12:17:51.408025 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:17:51.408014    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ad2d226a-ee9e-454c-8dee-c501ef1315e9-isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-74788cb89d-fqbr8\" (UID: \"ad2d226a-ee9e-454c-8dee-c501ef1315e9\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-74788cb89d-fqbr8"
May 25 12:17:51.408148 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:17:51.408074    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ad2d226a-ee9e-454c-8dee-c501ef1315e9-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-74788cb89d-fqbr8\" (UID: \"ad2d226a-ee9e-454c-8dee-c501ef1315e9\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-74788cb89d-fqbr8"
May 25 12:17:51.509145 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:17:51.509123    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-rkzdd\" (UniqueName: \"kubernetes.io/projected/ad2d226a-ee9e-454c-8dee-c501ef1315e9-kube-api-access-rkzdd\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-74788cb89d-fqbr8\" (UID: \"ad2d226a-ee9e-454c-8dee-c501ef1315e9\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-74788cb89d-fqbr8"
May 25 12:17:51.509262 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:17:51.509160    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ad2d226a-ee9e-454c-8dee-c501ef1315e9-cabundle-cert\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-74788cb89d-fqbr8\" (UID: \"ad2d226a-ee9e-454c-8dee-c501ef1315e9\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-74788cb89d-fqbr8"
May 25 12:17:51.509262 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:17:51.509184    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ad2d226a-ee9e-454c-8dee-c501ef1315e9-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-74788cb89d-fqbr8\" (UID: \"ad2d226a-ee9e-454c-8dee-c501ef1315e9\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-74788cb89d-fqbr8"
May 25 12:17:51.509262 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:17:51.509215    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ad2d226a-ee9e-454c-8dee-c501ef1315e9-isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-74788cb89d-fqbr8\" (UID: \"ad2d226a-ee9e-454c-8dee-c501ef1315e9\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-74788cb89d-fqbr8"
May 25 12:17:51.509262 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:17:51.509251    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ad2d226a-ee9e-454c-8dee-c501ef1315e9-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-74788cb89d-fqbr8\" (UID: \"ad2d226a-ee9e-454c-8dee-c501ef1315e9\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-74788cb89d-fqbr8"
May 25 12:17:51.509609 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:17:51.509588    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ad2d226a-ee9e-454c-8dee-c501ef1315e9-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-74788cb89d-fqbr8\" (UID: \"ad2d226a-ee9e-454c-8dee-c501ef1315e9\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-74788cb89d-fqbr8"
May 25 12:17:51.509847 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:17:51.509828    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ad2d226a-ee9e-454c-8dee-c501ef1315e9-isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-74788cb89d-fqbr8\" (UID: \"ad2d226a-ee9e-454c-8dee-c501ef1315e9\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-74788cb89d-fqbr8"
May 25 12:17:51.509993 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:17:51.509976    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ad2d226a-ee9e-454c-8dee-c501ef1315e9-cabundle-cert\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-74788cb89d-fqbr8\" (UID: \"ad2d226a-ee9e-454c-8dee-c501ef1315e9\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-74788cb89d-fqbr8"
May 25 12:17:51.511714 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:17:51.511694    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ad2d226a-ee9e-454c-8dee-c501ef1315e9-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-74788cb89d-fqbr8\" (UID: \"ad2d226a-ee9e-454c-8dee-c501ef1315e9\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-74788cb89d-fqbr8"
May 25 12:17:51.518065 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:17:51.518042    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkzdd\" (UniqueName: \"kubernetes.io/projected/ad2d226a-ee9e-454c-8dee-c501ef1315e9-kube-api-access-rkzdd\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-74788cb89d-fqbr8\" (UID: \"ad2d226a-ee9e-454c-8dee-c501ef1315e9\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-74788cb89d-fqbr8"
May 25 12:17:51.683349 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:17:51.683281    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-74788cb89d-fqbr8"
May 25 12:17:51.812391 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:17:51.812008    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-74788cb89d-fqbr8"]
May 25 12:17:51.814690 ip-10-0-136-227 kubenswrapper[2561]: W0525 12:17:51.814661    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad2d226a_ee9e_454c_8dee_c501ef1315e9.slice/crio-7dcb09a50e3cf3c9a6f1e1bd31109d2788fcf7e779f175d305085a655d7c1628 WatchSource:0}: Error finding container 7dcb09a50e3cf3c9a6f1e1bd31109d2788fcf7e779f175d305085a655d7c1628: Status 404 returned error can't find the container with id 7dcb09a50e3cf3c9a6f1e1bd31109d2788fcf7e779f175d305085a655d7c1628
May 25 12:17:52.551891 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:17:52.551867    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-7cf8df44c6-mssq5_10e6245c-3cae-4bcc-9532-5c9d67d98210/storage-initializer/1.log"
May 25 12:17:52.552254 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:17:52.552202    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-7cf8df44c6-mssq5_10e6245c-3cae-4bcc-9532-5c9d67d98210/storage-initializer/0.log"
May 25 12:17:52.552321 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:17:52.552263    2561 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7cf8df44c6-mssq5"
May 25 12:17:52.617895 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:17:52.617873    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/10e6245c-3cae-4bcc-9532-5c9d67d98210-kserve-provision-location\") pod \"10e6245c-3cae-4bcc-9532-5c9d67d98210\" (UID: \"10e6245c-3cae-4bcc-9532-5c9d67d98210\") "
May 25 12:17:52.618050 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:17:52.617951    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/10e6245c-3cae-4bcc-9532-5c9d67d98210-isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config\") pod \"10e6245c-3cae-4bcc-9532-5c9d67d98210\" (UID: \"10e6245c-3cae-4bcc-9532-5c9d67d98210\") "
May 25 12:17:52.618050 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:17:52.618012    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/10e6245c-3cae-4bcc-9532-5c9d67d98210-proxy-tls\") pod \"10e6245c-3cae-4bcc-9532-5c9d67d98210\" (UID: \"10e6245c-3cae-4bcc-9532-5c9d67d98210\") "
May 25 12:17:52.618050 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:17:52.618029    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rd8jk\" (UniqueName: \"kubernetes.io/projected/10e6245c-3cae-4bcc-9532-5c9d67d98210-kube-api-access-rd8jk\") pod \"10e6245c-3cae-4bcc-9532-5c9d67d98210\" (UID: \"10e6245c-3cae-4bcc-9532-5c9d67d98210\") "
May 25 12:17:52.618226 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:17:52.618144    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10e6245c-3cae-4bcc-9532-5c9d67d98210-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "10e6245c-3cae-4bcc-9532-5c9d67d98210" (UID: "10e6245c-3cae-4bcc-9532-5c9d67d98210"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 12:17:52.618327 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:17:52.618292    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/10e6245c-3cae-4bcc-9532-5c9d67d98210-kserve-provision-location\") on node \"ip-10-0-136-227.ec2.internal\" DevicePath \"\""
May 25 12:17:52.618407 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:17:52.618375    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10e6245c-3cae-4bcc-9532-5c9d67d98210-isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config") pod "10e6245c-3cae-4bcc-9532-5c9d67d98210" (UID: "10e6245c-3cae-4bcc-9532-5c9d67d98210"). InnerVolumeSpecName "isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 12:17:52.620035 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:17:52.620008    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10e6245c-3cae-4bcc-9532-5c9d67d98210-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "10e6245c-3cae-4bcc-9532-5c9d67d98210" (UID: "10e6245c-3cae-4bcc-9532-5c9d67d98210"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 12:17:52.620117 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:17:52.620053    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10e6245c-3cae-4bcc-9532-5c9d67d98210-kube-api-access-rd8jk" (OuterVolumeSpecName: "kube-api-access-rd8jk") pod "10e6245c-3cae-4bcc-9532-5c9d67d98210" (UID: "10e6245c-3cae-4bcc-9532-5c9d67d98210"). InnerVolumeSpecName "kube-api-access-rd8jk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 12:17:52.661217 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:17:52.661164    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-7cf8df44c6-mssq5_10e6245c-3cae-4bcc-9532-5c9d67d98210/storage-initializer/1.log"
May 25 12:17:52.661525 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:17:52.661508    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-7cf8df44c6-mssq5_10e6245c-3cae-4bcc-9532-5c9d67d98210/storage-initializer/0.log"
May 25 12:17:52.661617 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:17:52.661548    2561 generic.go:358] "Generic (PLEG): container finished" podID="10e6245c-3cae-4bcc-9532-5c9d67d98210" containerID="3b75df694de5ee7aa2c15ba718875596e655c461314b7f0ff8fbc574a9b17e08" exitCode=1
May 25 12:17:52.661673 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:17:52.661632    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7cf8df44c6-mssq5" event={"ID":"10e6245c-3cae-4bcc-9532-5c9d67d98210","Type":"ContainerDied","Data":"3b75df694de5ee7aa2c15ba718875596e655c461314b7f0ff8fbc574a9b17e08"}
May 25 12:17:52.661673 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:17:52.661656    2561 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7cf8df44c6-mssq5"
May 25 12:17:52.661772 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:17:52.661677    2561 scope.go:117] "RemoveContainer" containerID="3b75df694de5ee7aa2c15ba718875596e655c461314b7f0ff8fbc574a9b17e08"
May 25 12:17:52.661824 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:17:52.661666    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7cf8df44c6-mssq5" event={"ID":"10e6245c-3cae-4bcc-9532-5c9d67d98210","Type":"ContainerDied","Data":"b967d4af6cdfa1caa8c009e1e34940e82f5aa30e28ba3ea3c5b4b686aae47f62"}
May 25 12:17:52.663256 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:17:52.663237    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-74788cb89d-fqbr8" event={"ID":"ad2d226a-ee9e-454c-8dee-c501ef1315e9","Type":"ContainerStarted","Data":"ad31ee3a56c15126df860d90a598aeb367e1bafd4bf980a13181ae9cbedbccdf"}
May 25 12:17:52.663393 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:17:52.663263    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-74788cb89d-fqbr8" event={"ID":"ad2d226a-ee9e-454c-8dee-c501ef1315e9","Type":"ContainerStarted","Data":"7dcb09a50e3cf3c9a6f1e1bd31109d2788fcf7e779f175d305085a655d7c1628"}
May 25 12:17:52.669508 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:17:52.669491    2561 scope.go:117] "RemoveContainer" containerID="f7e143f930f0db8722f2cfc59c60dba3effec9c6e258f59c462f046ba6d8dac2"
May 25 12:17:52.677809 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:17:52.677792    2561 scope.go:117] "RemoveContainer" containerID="3b75df694de5ee7aa2c15ba718875596e655c461314b7f0ff8fbc574a9b17e08"
May 25 12:17:52.678078 ip-10-0-136-227 kubenswrapper[2561]: E0525 12:17:52.678060    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b75df694de5ee7aa2c15ba718875596e655c461314b7f0ff8fbc574a9b17e08\": container with ID starting with 3b75df694de5ee7aa2c15ba718875596e655c461314b7f0ff8fbc574a9b17e08 not found: ID does not exist" containerID="3b75df694de5ee7aa2c15ba718875596e655c461314b7f0ff8fbc574a9b17e08"
May 25 12:17:52.678132 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:17:52.678086    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b75df694de5ee7aa2c15ba718875596e655c461314b7f0ff8fbc574a9b17e08"} err="failed to get container status \"3b75df694de5ee7aa2c15ba718875596e655c461314b7f0ff8fbc574a9b17e08\": rpc error: code = NotFound desc = could not find container \"3b75df694de5ee7aa2c15ba718875596e655c461314b7f0ff8fbc574a9b17e08\": container with ID starting with 3b75df694de5ee7aa2c15ba718875596e655c461314b7f0ff8fbc574a9b17e08 not found: ID does not exist"
May 25 12:17:52.678132 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:17:52.678107    2561 scope.go:117] "RemoveContainer" containerID="f7e143f930f0db8722f2cfc59c60dba3effec9c6e258f59c462f046ba6d8dac2"
May 25 12:17:52.678367 ip-10-0-136-227 kubenswrapper[2561]: E0525 12:17:52.678348    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7e143f930f0db8722f2cfc59c60dba3effec9c6e258f59c462f046ba6d8dac2\": container with ID starting with f7e143f930f0db8722f2cfc59c60dba3effec9c6e258f59c462f046ba6d8dac2 not found: ID does not exist" containerID="f7e143f930f0db8722f2cfc59c60dba3effec9c6e258f59c462f046ba6d8dac2"
May 25 12:17:52.678424 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:17:52.678373    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7e143f930f0db8722f2cfc59c60dba3effec9c6e258f59c462f046ba6d8dac2"} err="failed to get container status \"f7e143f930f0db8722f2cfc59c60dba3effec9c6e258f59c462f046ba6d8dac2\": rpc error: code = NotFound desc = could not find container \"f7e143f930f0db8722f2cfc59c60dba3effec9c6e258f59c462f046ba6d8dac2\": container with ID starting with f7e143f930f0db8722f2cfc59c60dba3effec9c6e258f59c462f046ba6d8dac2 not found: ID does not exist"
May 25 12:17:52.707727 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:17:52.707705    2561 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7cf8df44c6-mssq5"]
May 25 12:17:52.712725 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:17:52.712700    2561 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-7cf8df44c6-mssq5"]
May 25 12:17:52.719621 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:17:52.719586    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/10e6245c-3cae-4bcc-9532-5c9d67d98210-proxy-tls\") on node \"ip-10-0-136-227.ec2.internal\" DevicePath \"\""
May 25 12:17:52.719793 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:17:52.719771    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rd8jk\" (UniqueName: \"kubernetes.io/projected/10e6245c-3cae-4bcc-9532-5c9d67d98210-kube-api-access-rd8jk\") on node \"ip-10-0-136-227.ec2.internal\" DevicePath \"\""
May 25 12:17:52.719886 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:17:52.719802    2561 reconciler_common.go:299] "Volume detached for volume \"isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/10e6245c-3cae-4bcc-9532-5c9d67d98210-isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-227.ec2.internal\" DevicePath \"\""
May 25 12:17:53.116975 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:17:53.116947    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="10e6245c-3cae-4bcc-9532-5c9d67d98210" path="/var/lib/kubelet/pods/10e6245c-3cae-4bcc-9532-5c9d67d98210/volumes"
May 25 12:17:53.667313 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:17:53.667285    2561 generic.go:358] "Generic (PLEG): container finished" podID="ad2d226a-ee9e-454c-8dee-c501ef1315e9" containerID="ad31ee3a56c15126df860d90a598aeb367e1bafd4bf980a13181ae9cbedbccdf" exitCode=0
May 25 12:17:53.667719 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:17:53.667364    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-74788cb89d-fqbr8" event={"ID":"ad2d226a-ee9e-454c-8dee-c501ef1315e9","Type":"ContainerDied","Data":"ad31ee3a56c15126df860d90a598aeb367e1bafd4bf980a13181ae9cbedbccdf"}
May 25 12:17:54.673372 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:17:54.673335    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-74788cb89d-fqbr8" event={"ID":"ad2d226a-ee9e-454c-8dee-c501ef1315e9","Type":"ContainerStarted","Data":"8777e0052b7125793e32d523e39cf6b26a0fa32baeeb3abff9ca6af28915bc46"}
May 25 12:17:54.673729 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:17:54.673376    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-74788cb89d-fqbr8" event={"ID":"ad2d226a-ee9e-454c-8dee-c501ef1315e9","Type":"ContainerStarted","Data":"c93aae11856681bee93bbfc6f3f74a9666e7e9f726222051804a2f5c241ecd36"}
May 25 12:17:54.673729 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:17:54.673481    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-74788cb89d-fqbr8"
May 25 12:17:54.673729 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:17:54.673504    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-74788cb89d-fqbr8"
May 25 12:17:54.674920 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:17:54.674893    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-74788cb89d-fqbr8" podUID="ad2d226a-ee9e-454c-8dee-c501ef1315e9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.59:8080: connect: connection refused"
May 25 12:17:54.692874 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:17:54.692831    2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-74788cb89d-fqbr8" podStartSLOduration=3.692819521 podStartE2EDuration="3.692819521s" podCreationTimestamp="2026-05-25 12:17:51 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 12:17:54.691592029 +0000 UTC m=+3648.236769659" watchObservedRunningTime="2026-05-25 12:17:54.692819521 +0000 UTC m=+3648.237997122"
May 25 12:17:55.677146 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:17:55.677110    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-74788cb89d-fqbr8" podUID="ad2d226a-ee9e-454c-8dee-c501ef1315e9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.59:8080: connect: connection refused"
May 25 12:18:00.681623 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:18:00.681596    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-74788cb89d-fqbr8"
May 25 12:18:00.682299 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:18:00.682250    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-74788cb89d-fqbr8" podUID="ad2d226a-ee9e-454c-8dee-c501ef1315e9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.59:8080: connect: connection refused"
May 25 12:18:10.682181 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:18:10.682137    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-74788cb89d-fqbr8" podUID="ad2d226a-ee9e-454c-8dee-c501ef1315e9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.59:8080: connect: connection refused"
May 25 12:18:20.682735 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:18:20.682692    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-74788cb89d-fqbr8" podUID="ad2d226a-ee9e-454c-8dee-c501ef1315e9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.59:8080: connect: connection refused"
May 25 12:18:30.682551 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:18:30.682476    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-74788cb89d-fqbr8" podUID="ad2d226a-ee9e-454c-8dee-c501ef1315e9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.59:8080: connect: connection refused"
May 25 12:18:40.682553 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:18:40.682511    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-74788cb89d-fqbr8" podUID="ad2d226a-ee9e-454c-8dee-c501ef1315e9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.59:8080: connect: connection refused"
May 25 12:18:50.683088 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:18:50.683049    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-74788cb89d-fqbr8" podUID="ad2d226a-ee9e-454c-8dee-c501ef1315e9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.59:8080: connect: connection refused"
May 25 12:19:00.682923 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:19:00.682893    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-74788cb89d-fqbr8"
May 25 12:19:01.344749 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:19:01.344720    2561 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-74788cb89d-fqbr8"]
May 25 12:19:01.345020 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:19:01.344979    2561 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-74788cb89d-fqbr8" podUID="ad2d226a-ee9e-454c-8dee-c501ef1315e9" containerName="kserve-container" containerID="cri-o://c93aae11856681bee93bbfc6f3f74a9666e7e9f726222051804a2f5c241ecd36" gracePeriod=30
May 25 12:19:01.345085 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:19:01.345006    2561 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-74788cb89d-fqbr8" podUID="ad2d226a-ee9e-454c-8dee-c501ef1315e9" containerName="kube-rbac-proxy" containerID="cri-o://8777e0052b7125793e32d523e39cf6b26a0fa32baeeb3abff9ca6af28915bc46" gracePeriod=30
May 25 12:19:01.892734 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:19:01.892701    2561 generic.go:358] "Generic (PLEG): container finished" podID="ad2d226a-ee9e-454c-8dee-c501ef1315e9" containerID="8777e0052b7125793e32d523e39cf6b26a0fa32baeeb3abff9ca6af28915bc46" exitCode=2
May 25 12:19:01.893087 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:19:01.892774    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-74788cb89d-fqbr8" event={"ID":"ad2d226a-ee9e-454c-8dee-c501ef1315e9","Type":"ContainerDied","Data":"8777e0052b7125793e32d523e39cf6b26a0fa32baeeb3abff9ca6af28915bc46"}
May 25 12:19:02.428165 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:19:02.428125    2561 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6f99b9d644-7g7ld"]
May 25 12:19:02.428712 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:19:02.428679    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10e6245c-3cae-4bcc-9532-5c9d67d98210" containerName="storage-initializer"
May 25 12:19:02.428712 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:19:02.428700    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="10e6245c-3cae-4bcc-9532-5c9d67d98210" containerName="storage-initializer"
May 25 12:19:02.428903 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:19:02.428806    2561 memory_manager.go:356] "RemoveStaleState removing state" podUID="10e6245c-3cae-4bcc-9532-5c9d67d98210" containerName="storage-initializer"
May 25 12:19:02.428903 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:19:02.428822    2561 memory_manager.go:356] "RemoveStaleState removing state" podUID="10e6245c-3cae-4bcc-9532-5c9d67d98210" containerName="storage-initializer"
May 25 12:19:02.428983 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:19:02.428929    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10e6245c-3cae-4bcc-9532-5c9d67d98210" containerName="storage-initializer"
May 25 12:19:02.428983 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:19:02.428941    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="10e6245c-3cae-4bcc-9532-5c9d67d98210" containerName="storage-initializer"
May 25 12:19:02.432170 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:19:02.432152    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6f99b9d644-7g7ld"
May 25 12:19:02.434467 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:19:02.434446    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config\""
May 25 12:19:02.434573 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:19:02.434535    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-fail-predictor-serving-cert\""
May 25 12:19:02.446070 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:19:02.446045    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6f99b9d644-7g7ld"]
May 25 12:19:02.557794 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:19:02.557758    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhnf7\" (UniqueName: \"kubernetes.io/projected/6614cea8-75df-4ce2-8970-44928df2dda7-kube-api-access-vhnf7\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-6f99b9d644-7g7ld\" (UID: \"6614cea8-75df-4ce2-8970-44928df2dda7\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6f99b9d644-7g7ld"
May 25 12:19:02.557960 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:19:02.557814    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6614cea8-75df-4ce2-8970-44928df2dda7-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-6f99b9d644-7g7ld\" (UID: \"6614cea8-75df-4ce2-8970-44928df2dda7\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6f99b9d644-7g7ld"
May 25 12:19:02.557960 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:19:02.557903    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6614cea8-75df-4ce2-8970-44928df2dda7-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-6f99b9d644-7g7ld\" (UID: \"6614cea8-75df-4ce2-8970-44928df2dda7\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6f99b9d644-7g7ld"
May 25 12:19:02.558061 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:19:02.557965    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6614cea8-75df-4ce2-8970-44928df2dda7-isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-6f99b9d644-7g7ld\" (UID: \"6614cea8-75df-4ce2-8970-44928df2dda7\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6f99b9d644-7g7ld"
May 25 12:19:02.659108 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:19:02.659067    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6614cea8-75df-4ce2-8970-44928df2dda7-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-6f99b9d644-7g7ld\" (UID: \"6614cea8-75df-4ce2-8970-44928df2dda7\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6f99b9d644-7g7ld"
May 25 12:19:02.659329 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:19:02.659138    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6614cea8-75df-4ce2-8970-44928df2dda7-isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-6f99b9d644-7g7ld\" (UID: \"6614cea8-75df-4ce2-8970-44928df2dda7\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6f99b9d644-7g7ld"
May 25 12:19:02.659329 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:19:02.659188    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-vhnf7\" (UniqueName: \"kubernetes.io/projected/6614cea8-75df-4ce2-8970-44928df2dda7-kube-api-access-vhnf7\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-6f99b9d644-7g7ld\" (UID: \"6614cea8-75df-4ce2-8970-44928df2dda7\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6f99b9d644-7g7ld"
May 25 12:19:02.659329 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:19:02.659233    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6614cea8-75df-4ce2-8970-44928df2dda7-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-6f99b9d644-7g7ld\" (UID: \"6614cea8-75df-4ce2-8970-44928df2dda7\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6f99b9d644-7g7ld"
May 25 12:19:02.659547 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:19:02.659513    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6614cea8-75df-4ce2-8970-44928df2dda7-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-6f99b9d644-7g7ld\" (UID: \"6614cea8-75df-4ce2-8970-44928df2dda7\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6f99b9d644-7g7ld"
May 25 12:19:02.659840 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:19:02.659820    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6614cea8-75df-4ce2-8970-44928df2dda7-isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-6f99b9d644-7g7ld\" (UID: \"6614cea8-75df-4ce2-8970-44928df2dda7\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6f99b9d644-7g7ld"
May 25 12:19:02.661616 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:19:02.661594    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6614cea8-75df-4ce2-8970-44928df2dda7-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-6f99b9d644-7g7ld\" (UID: \"6614cea8-75df-4ce2-8970-44928df2dda7\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6f99b9d644-7g7ld"
May 25 12:19:02.667561 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:19:02.667540    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhnf7\" (UniqueName: \"kubernetes.io/projected/6614cea8-75df-4ce2-8970-44928df2dda7-kube-api-access-vhnf7\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-6f99b9d644-7g7ld\" (UID: \"6614cea8-75df-4ce2-8970-44928df2dda7\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6f99b9d644-7g7ld"
May 25 12:19:02.748421 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:19:02.748346    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6f99b9d644-7g7ld"
May 25 12:19:02.877479 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:19:02.877452    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6f99b9d644-7g7ld"]
May 25 12:19:02.879630 ip-10-0-136-227 kubenswrapper[2561]: W0525 12:19:02.879604    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6614cea8_75df_4ce2_8970_44928df2dda7.slice/crio-e1ab630b2d412fb61fc4e5af1cfdb0a6ab6e708f58d674d9542717c2ed9f57be WatchSource:0}: Error finding container e1ab630b2d412fb61fc4e5af1cfdb0a6ab6e708f58d674d9542717c2ed9f57be: Status 404 returned error can't find the container with id e1ab630b2d412fb61fc4e5af1cfdb0a6ab6e708f58d674d9542717c2ed9f57be
May 25 12:19:02.897047 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:19:02.897017    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6f99b9d644-7g7ld" event={"ID":"6614cea8-75df-4ce2-8970-44928df2dda7","Type":"ContainerStarted","Data":"e1ab630b2d412fb61fc4e5af1cfdb0a6ab6e708f58d674d9542717c2ed9f57be"}
May 25 12:19:03.901989 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:19:03.901952    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6f99b9d644-7g7ld" event={"ID":"6614cea8-75df-4ce2-8970-44928df2dda7","Type":"ContainerStarted","Data":"1c46c140511bf3d7bb332f8f0d42ddee2b401d606b229bad13d12700d3f18386"}
May 25 12:19:05.496060 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:19:05.496039    2561 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-74788cb89d-fqbr8"
May 25 12:19:05.587595 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:19:05.587565    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ad2d226a-ee9e-454c-8dee-c501ef1315e9-cabundle-cert\") pod \"ad2d226a-ee9e-454c-8dee-c501ef1315e9\" (UID: \"ad2d226a-ee9e-454c-8dee-c501ef1315e9\") "
May 25 12:19:05.587774 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:19:05.587614    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ad2d226a-ee9e-454c-8dee-c501ef1315e9-proxy-tls\") pod \"ad2d226a-ee9e-454c-8dee-c501ef1315e9\" (UID: \"ad2d226a-ee9e-454c-8dee-c501ef1315e9\") "
May 25 12:19:05.587774 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:19:05.587636    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ad2d226a-ee9e-454c-8dee-c501ef1315e9-isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config\") pod \"ad2d226a-ee9e-454c-8dee-c501ef1315e9\" (UID: \"ad2d226a-ee9e-454c-8dee-c501ef1315e9\") "
May 25 12:19:05.587774 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:19:05.587666    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ad2d226a-ee9e-454c-8dee-c501ef1315e9-kserve-provision-location\") pod \"ad2d226a-ee9e-454c-8dee-c501ef1315e9\" (UID: \"ad2d226a-ee9e-454c-8dee-c501ef1315e9\") "
May 25 12:19:05.587774 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:19:05.587686    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rkzdd\" (UniqueName: \"kubernetes.io/projected/ad2d226a-ee9e-454c-8dee-c501ef1315e9-kube-api-access-rkzdd\") pod \"ad2d226a-ee9e-454c-8dee-c501ef1315e9\" (UID: \"ad2d226a-ee9e-454c-8dee-c501ef1315e9\") "
May 25 12:19:05.588053 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:19:05.588026    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad2d226a-ee9e-454c-8dee-c501ef1315e9-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ad2d226a-ee9e-454c-8dee-c501ef1315e9" (UID: "ad2d226a-ee9e-454c-8dee-c501ef1315e9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 12:19:05.588053 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:19:05.588041    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad2d226a-ee9e-454c-8dee-c501ef1315e9-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ad2d226a-ee9e-454c-8dee-c501ef1315e9" (UID: "ad2d226a-ee9e-454c-8dee-c501ef1315e9"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 12:19:05.588147 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:19:05.588049    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad2d226a-ee9e-454c-8dee-c501ef1315e9-isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config") pod "ad2d226a-ee9e-454c-8dee-c501ef1315e9" (UID: "ad2d226a-ee9e-454c-8dee-c501ef1315e9"). InnerVolumeSpecName "isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 12:19:05.589751 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:19:05.589731    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad2d226a-ee9e-454c-8dee-c501ef1315e9-kube-api-access-rkzdd" (OuterVolumeSpecName: "kube-api-access-rkzdd") pod "ad2d226a-ee9e-454c-8dee-c501ef1315e9" (UID: "ad2d226a-ee9e-454c-8dee-c501ef1315e9"). InnerVolumeSpecName "kube-api-access-rkzdd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 12:19:05.589851 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:19:05.589836    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad2d226a-ee9e-454c-8dee-c501ef1315e9-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ad2d226a-ee9e-454c-8dee-c501ef1315e9" (UID: "ad2d226a-ee9e-454c-8dee-c501ef1315e9"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 12:19:05.689293 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:19:05.689245    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ad2d226a-ee9e-454c-8dee-c501ef1315e9-cabundle-cert\") on node \"ip-10-0-136-227.ec2.internal\" DevicePath \"\""
May 25 12:19:05.689293 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:19:05.689290    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ad2d226a-ee9e-454c-8dee-c501ef1315e9-proxy-tls\") on node \"ip-10-0-136-227.ec2.internal\" DevicePath \"\""
May 25 12:19:05.689293 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:19:05.689299    2561 reconciler_common.go:299] "Volume detached for volume \"isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ad2d226a-ee9e-454c-8dee-c501ef1315e9-isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-227.ec2.internal\" DevicePath \"\""
May 25 12:19:05.689488 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:19:05.689309    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ad2d226a-ee9e-454c-8dee-c501ef1315e9-kserve-provision-location\") on node \"ip-10-0-136-227.ec2.internal\" DevicePath \"\""
May 25 12:19:05.689488 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:19:05.689318    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rkzdd\" (UniqueName: \"kubernetes.io/projected/ad2d226a-ee9e-454c-8dee-c501ef1315e9-kube-api-access-rkzdd\") on node \"ip-10-0-136-227.ec2.internal\" DevicePath \"\""
May 25 12:19:05.913308 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:19:05.913199    2561 generic.go:358] "Generic (PLEG): container finished" podID="ad2d226a-ee9e-454c-8dee-c501ef1315e9" containerID="c93aae11856681bee93bbfc6f3f74a9666e7e9f726222051804a2f5c241ecd36" exitCode=0
May 25 12:19:05.913308 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:19:05.913246    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-74788cb89d-fqbr8" event={"ID":"ad2d226a-ee9e-454c-8dee-c501ef1315e9","Type":"ContainerDied","Data":"c93aae11856681bee93bbfc6f3f74a9666e7e9f726222051804a2f5c241ecd36"}
May 25 12:19:05.913308 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:19:05.913303    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-74788cb89d-fqbr8" event={"ID":"ad2d226a-ee9e-454c-8dee-c501ef1315e9","Type":"ContainerDied","Data":"7dcb09a50e3cf3c9a6f1e1bd31109d2788fcf7e779f175d305085a655d7c1628"}
May 25 12:19:05.913561 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:19:05.913309    2561 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-74788cb89d-fqbr8"
May 25 12:19:05.913561 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:19:05.913327    2561 scope.go:117] "RemoveContainer" containerID="8777e0052b7125793e32d523e39cf6b26a0fa32baeeb3abff9ca6af28915bc46"
May 25 12:19:05.922545 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:19:05.922526    2561 scope.go:117] "RemoveContainer" containerID="c93aae11856681bee93bbfc6f3f74a9666e7e9f726222051804a2f5c241ecd36"
May 25 12:19:05.930233 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:19:05.930211    2561 scope.go:117] "RemoveContainer" containerID="ad31ee3a56c15126df860d90a598aeb367e1bafd4bf980a13181ae9cbedbccdf"
May 25 12:19:05.936248 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:19:05.936224    2561 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-74788cb89d-fqbr8"]
May 25 12:19:05.939039 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:19:05.939013    2561 scope.go:117] "RemoveContainer" containerID="8777e0052b7125793e32d523e39cf6b26a0fa32baeeb3abff9ca6af28915bc46"
May 25 12:19:05.939353 ip-10-0-136-227 kubenswrapper[2561]: E0525 12:19:05.939329    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8777e0052b7125793e32d523e39cf6b26a0fa32baeeb3abff9ca6af28915bc46\": container with ID starting with 8777e0052b7125793e32d523e39cf6b26a0fa32baeeb3abff9ca6af28915bc46 not found: ID does not exist" containerID="8777e0052b7125793e32d523e39cf6b26a0fa32baeeb3abff9ca6af28915bc46"
May 25 12:19:05.939409 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:19:05.939355    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8777e0052b7125793e32d523e39cf6b26a0fa32baeeb3abff9ca6af28915bc46"} err="failed to get container status \"8777e0052b7125793e32d523e39cf6b26a0fa32baeeb3abff9ca6af28915bc46\": rpc error: code = NotFound desc = could not find container \"8777e0052b7125793e32d523e39cf6b26a0fa32baeeb3abff9ca6af28915bc46\": container with ID starting with 8777e0052b7125793e32d523e39cf6b26a0fa32baeeb3abff9ca6af28915bc46 not found: ID does not exist"
May 25 12:19:05.939409 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:19:05.939375    2561 scope.go:117] "RemoveContainer" containerID="c93aae11856681bee93bbfc6f3f74a9666e7e9f726222051804a2f5c241ecd36"
May 25 12:19:05.939622 ip-10-0-136-227 kubenswrapper[2561]: E0525 12:19:05.939603    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c93aae11856681bee93bbfc6f3f74a9666e7e9f726222051804a2f5c241ecd36\": container with ID starting with c93aae11856681bee93bbfc6f3f74a9666e7e9f726222051804a2f5c241ecd36 not found: ID does not exist" containerID="c93aae11856681bee93bbfc6f3f74a9666e7e9f726222051804a2f5c241ecd36"
May 25 12:19:05.939666 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:19:05.939629    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c93aae11856681bee93bbfc6f3f74a9666e7e9f726222051804a2f5c241ecd36"} err="failed to get container status \"c93aae11856681bee93bbfc6f3f74a9666e7e9f726222051804a2f5c241ecd36\": rpc error: code = NotFound desc = could not find container \"c93aae11856681bee93bbfc6f3f74a9666e7e9f726222051804a2f5c241ecd36\": container with ID starting with c93aae11856681bee93bbfc6f3f74a9666e7e9f726222051804a2f5c241ecd36 not found: ID does not exist"
May 25 12:19:05.939666 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:19:05.939646    2561 scope.go:117] "RemoveContainer" containerID="ad31ee3a56c15126df860d90a598aeb367e1bafd4bf980a13181ae9cbedbccdf"
May 25 12:19:05.939910 ip-10-0-136-227 kubenswrapper[2561]: E0525 12:19:05.939893    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad31ee3a56c15126df860d90a598aeb367e1bafd4bf980a13181ae9cbedbccdf\": container with ID starting with ad31ee3a56c15126df860d90a598aeb367e1bafd4bf980a13181ae9cbedbccdf not found: ID does not exist" containerID="ad31ee3a56c15126df860d90a598aeb367e1bafd4bf980a13181ae9cbedbccdf"
May 25 12:19:05.939973 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:19:05.939914    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad31ee3a56c15126df860d90a598aeb367e1bafd4bf980a13181ae9cbedbccdf"} err="failed to get container status \"ad31ee3a56c15126df860d90a598aeb367e1bafd4bf980a13181ae9cbedbccdf\": rpc error: code = NotFound desc = could not find container \"ad31ee3a56c15126df860d90a598aeb367e1bafd4bf980a13181ae9cbedbccdf\": container with ID starting with ad31ee3a56c15126df860d90a598aeb367e1bafd4bf980a13181ae9cbedbccdf not found: ID does not exist"
May 25 12:19:05.940014 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:19:05.939996    2561 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-74788cb89d-fqbr8"]
May 25 12:19:06.919278 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:19:06.919252    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-6f99b9d644-7g7ld_6614cea8-75df-4ce2-8970-44928df2dda7/storage-initializer/0.log"
May 25 12:19:06.919634 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:19:06.919312    2561 generic.go:358] "Generic (PLEG): container finished" podID="6614cea8-75df-4ce2-8970-44928df2dda7" containerID="1c46c140511bf3d7bb332f8f0d42ddee2b401d606b229bad13d12700d3f18386" exitCode=1
May 25 12:19:06.919634 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:19:06.919390    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6f99b9d644-7g7ld" event={"ID":"6614cea8-75df-4ce2-8970-44928df2dda7","Type":"ContainerDied","Data":"1c46c140511bf3d7bb332f8f0d42ddee2b401d606b229bad13d12700d3f18386"}
May 25 12:19:07.117982 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:19:07.117948    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ad2d226a-ee9e-454c-8dee-c501ef1315e9" path="/var/lib/kubelet/pods/ad2d226a-ee9e-454c-8dee-c501ef1315e9/volumes"
May 25 12:19:07.924584 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:19:07.924558    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-6f99b9d644-7g7ld_6614cea8-75df-4ce2-8970-44928df2dda7/storage-initializer/0.log"
May 25 12:19:07.924931 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:19:07.924642    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6f99b9d644-7g7ld" event={"ID":"6614cea8-75df-4ce2-8970-44928df2dda7","Type":"ContainerStarted","Data":"ed49a0363e701a1f470f6e60f543c79c646cf692548a1a09eebffe7864801af6"}
May 25 12:19:11.937995 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:19:11.937969    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-6f99b9d644-7g7ld_6614cea8-75df-4ce2-8970-44928df2dda7/storage-initializer/1.log"
May 25 12:19:11.938460 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:19:11.938404    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-6f99b9d644-7g7ld_6614cea8-75df-4ce2-8970-44928df2dda7/storage-initializer/0.log"
May 25 12:19:11.938460 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:19:11.938444    2561 generic.go:358] "Generic (PLEG): container finished" podID="6614cea8-75df-4ce2-8970-44928df2dda7" containerID="ed49a0363e701a1f470f6e60f543c79c646cf692548a1a09eebffe7864801af6" exitCode=1
May 25 12:19:11.938571 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:19:11.938524    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6f99b9d644-7g7ld" event={"ID":"6614cea8-75df-4ce2-8970-44928df2dda7","Type":"ContainerDied","Data":"ed49a0363e701a1f470f6e60f543c79c646cf692548a1a09eebffe7864801af6"}
May 25 12:19:11.938571 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:19:11.938567    2561 scope.go:117] "RemoveContainer" containerID="1c46c140511bf3d7bb332f8f0d42ddee2b401d606b229bad13d12700d3f18386"
May 25 12:19:11.938943 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:19:11.938927    2561 scope.go:117] "RemoveContainer" containerID="1c46c140511bf3d7bb332f8f0d42ddee2b401d606b229bad13d12700d3f18386"
May 25 12:19:11.950136 ip-10-0-136-227 kubenswrapper[2561]: E0525 12:19:11.950107    2561 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-sklearn-s3-tls-custom-fail-predictor-6f99b9d644-7g7ld_kserve-ci-e2e-test_6614cea8-75df-4ce2-8970-44928df2dda7_0 in pod sandbox e1ab630b2d412fb61fc4e5af1cfdb0a6ab6e708f58d674d9542717c2ed9f57be from index: no such id: '1c46c140511bf3d7bb332f8f0d42ddee2b401d606b229bad13d12700d3f18386'" containerID="1c46c140511bf3d7bb332f8f0d42ddee2b401d606b229bad13d12700d3f18386"
May 25 12:19:11.950237 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:19:11.950143    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c46c140511bf3d7bb332f8f0d42ddee2b401d606b229bad13d12700d3f18386"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-sklearn-s3-tls-custom-fail-predictor-6f99b9d644-7g7ld_kserve-ci-e2e-test_6614cea8-75df-4ce2-8970-44928df2dda7_0 in pod sandbox e1ab630b2d412fb61fc4e5af1cfdb0a6ab6e708f58d674d9542717c2ed9f57be from index: no such id: '1c46c140511bf3d7bb332f8f0d42ddee2b401d606b229bad13d12700d3f18386'"
May 25 12:19:11.950342 ip-10-0-136-227 kubenswrapper[2561]: E0525 12:19:11.950320    2561 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-sklearn-s3-tls-custom-fail-predictor-6f99b9d644-7g7ld_kserve-ci-e2e-test(6614cea8-75df-4ce2-8970-44928df2dda7)\"" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6f99b9d644-7g7ld" podUID="6614cea8-75df-4ce2-8970-44928df2dda7"
May 25 12:19:12.419856 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:19:12.419826    2561 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6f99b9d644-7g7ld"]
May 25 12:19:12.943108 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:19:12.943077    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-6f99b9d644-7g7ld_6614cea8-75df-4ce2-8970-44928df2dda7/storage-initializer/1.log"
May 25 12:19:13.081847 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:19:13.081827    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-6f99b9d644-7g7ld_6614cea8-75df-4ce2-8970-44928df2dda7/storage-initializer/1.log"
May 25 12:19:13.081967 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:19:13.081890    2561 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6f99b9d644-7g7ld"
May 25 12:19:13.147238 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:19:13.147213    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vhnf7\" (UniqueName: \"kubernetes.io/projected/6614cea8-75df-4ce2-8970-44928df2dda7-kube-api-access-vhnf7\") pod \"6614cea8-75df-4ce2-8970-44928df2dda7\" (UID: \"6614cea8-75df-4ce2-8970-44928df2dda7\") "
May 25 12:19:13.147392 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:19:13.147298    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6614cea8-75df-4ce2-8970-44928df2dda7-proxy-tls\") pod \"6614cea8-75df-4ce2-8970-44928df2dda7\" (UID: \"6614cea8-75df-4ce2-8970-44928df2dda7\") "
May 25 12:19:13.147392 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:19:13.147337    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6614cea8-75df-4ce2-8970-44928df2dda7-kserve-provision-location\") pod \"6614cea8-75df-4ce2-8970-44928df2dda7\" (UID: \"6614cea8-75df-4ce2-8970-44928df2dda7\") "
May 25 12:19:13.147392 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:19:13.147362    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6614cea8-75df-4ce2-8970-44928df2dda7-isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config\") pod \"6614cea8-75df-4ce2-8970-44928df2dda7\" (UID: \"6614cea8-75df-4ce2-8970-44928df2dda7\") "
May 25 12:19:13.147602 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:19:13.147577    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6614cea8-75df-4ce2-8970-44928df2dda7-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6614cea8-75df-4ce2-8970-44928df2dda7" (UID: "6614cea8-75df-4ce2-8970-44928df2dda7"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 12:19:13.147759 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:19:13.147736    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6614cea8-75df-4ce2-8970-44928df2dda7-isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config") pod "6614cea8-75df-4ce2-8970-44928df2dda7" (UID: "6614cea8-75df-4ce2-8970-44928df2dda7"). InnerVolumeSpecName "isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 12:19:13.149224 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:19:13.149205    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6614cea8-75df-4ce2-8970-44928df2dda7-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6614cea8-75df-4ce2-8970-44928df2dda7" (UID: "6614cea8-75df-4ce2-8970-44928df2dda7"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 12:19:13.149332 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:19:13.149315    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6614cea8-75df-4ce2-8970-44928df2dda7-kube-api-access-vhnf7" (OuterVolumeSpecName: "kube-api-access-vhnf7") pod "6614cea8-75df-4ce2-8970-44928df2dda7" (UID: "6614cea8-75df-4ce2-8970-44928df2dda7"). InnerVolumeSpecName "kube-api-access-vhnf7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 12:19:13.248905 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:19:13.248846    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vhnf7\" (UniqueName: \"kubernetes.io/projected/6614cea8-75df-4ce2-8970-44928df2dda7-kube-api-access-vhnf7\") on node \"ip-10-0-136-227.ec2.internal\" DevicePath \"\""
May 25 12:19:13.248905 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:19:13.248867    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6614cea8-75df-4ce2-8970-44928df2dda7-proxy-tls\") on node \"ip-10-0-136-227.ec2.internal\" DevicePath \"\""
May 25 12:19:13.248905 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:19:13.248877    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6614cea8-75df-4ce2-8970-44928df2dda7-kserve-provision-location\") on node \"ip-10-0-136-227.ec2.internal\" DevicePath \"\""
May 25 12:19:13.248905 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:19:13.248888    2561 reconciler_common.go:299] "Volume detached for volume \"isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6614cea8-75df-4ce2-8970-44928df2dda7-isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-227.ec2.internal\" DevicePath \"\""
May 25 12:19:13.491133 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:19:13.491102    2561 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7c48986569-c7fxx"]
May 25 12:19:13.491520 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:19:13.491504    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad2d226a-ee9e-454c-8dee-c501ef1315e9" containerName="kserve-container"
May 25 12:19:13.491583 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:19:13.491522    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad2d226a-ee9e-454c-8dee-c501ef1315e9" containerName="kserve-container"
May 25 12:19:13.491583 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:19:13.491536    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad2d226a-ee9e-454c-8dee-c501ef1315e9" containerName="kube-rbac-proxy"
May 25 12:19:13.491583 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:19:13.491541    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad2d226a-ee9e-454c-8dee-c501ef1315e9" containerName="kube-rbac-proxy"
May 25 12:19:13.491583 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:19:13.491557    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad2d226a-ee9e-454c-8dee-c501ef1315e9" containerName="storage-initializer"
May 25 12:19:13.491583 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:19:13.491563    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad2d226a-ee9e-454c-8dee-c501ef1315e9" containerName="storage-initializer"
May 25 12:19:13.491583 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:19:13.491568    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6614cea8-75df-4ce2-8970-44928df2dda7" containerName="storage-initializer"
May 25 12:19:13.491583 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:19:13.491573    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="6614cea8-75df-4ce2-8970-44928df2dda7" containerName="storage-initializer"
May 25 12:19:13.491583 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:19:13.491581    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6614cea8-75df-4ce2-8970-44928df2dda7" containerName="storage-initializer"
May 25 12:19:13.491583 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:19:13.491586    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="6614cea8-75df-4ce2-8970-44928df2dda7" containerName="storage-initializer"
May 25 12:19:13.491852 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:19:13.491644    2561 memory_manager.go:356] "RemoveStaleState removing state" podUID="ad2d226a-ee9e-454c-8dee-c501ef1315e9" containerName="kserve-container"
May 25 12:19:13.491852 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:19:13.491652    2561 memory_manager.go:356] "RemoveStaleState removing state" podUID="6614cea8-75df-4ce2-8970-44928df2dda7" containerName="storage-initializer"
May 25 12:19:13.491852 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:19:13.491659    2561 memory_manager.go:356] "RemoveStaleState removing state" podUID="6614cea8-75df-4ce2-8970-44928df2dda7" containerName="storage-initializer"
May 25 12:19:13.491852 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:19:13.491666    2561 memory_manager.go:356] "RemoveStaleState removing state" podUID="ad2d226a-ee9e-454c-8dee-c501ef1315e9" containerName="kube-rbac-proxy"
May 25 12:19:13.496350 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:19:13.496329    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7c48986569-c7fxx"
May 25 12:19:13.498979 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:19:13.498919    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-pass-predictor-serving-cert\""
May 25 12:19:13.498979 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:19:13.498944    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config\""
May 25 12:19:13.499110 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:19:13.498922    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
May 25 12:19:13.502613 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:19:13.502592    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7c48986569-c7fxx"]
May 25 12:19:13.551055 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:19:13.551028    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4ba4a439-2e5f-47e6-8154-882197640493-isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-7c48986569-c7fxx\" (UID: \"4ba4a439-2e5f-47e6-8154-882197640493\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7c48986569-c7fxx"
May 25 12:19:13.551182 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:19:13.551154    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4ba4a439-2e5f-47e6-8154-882197640493-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-7c48986569-c7fxx\" (UID: \"4ba4a439-2e5f-47e6-8154-882197640493\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7c48986569-c7fxx"
May 25 12:19:13.551230 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:19:13.551192    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4ba4a439-2e5f-47e6-8154-882197640493-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-7c48986569-c7fxx\" (UID: \"4ba4a439-2e5f-47e6-8154-882197640493\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7c48986569-c7fxx"
May 25 12:19:13.551291 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:19:13.551224    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lp6cq\" (UniqueName: \"kubernetes.io/projected/4ba4a439-2e5f-47e6-8154-882197640493-kube-api-access-lp6cq\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-7c48986569-c7fxx\" (UID: \"4ba4a439-2e5f-47e6-8154-882197640493\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7c48986569-c7fxx"
May 25 12:19:13.551291 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:19:13.551253    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4ba4a439-2e5f-47e6-8154-882197640493-cabundle-cert\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-7c48986569-c7fxx\" (UID: \"4ba4a439-2e5f-47e6-8154-882197640493\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7c48986569-c7fxx"
May 25 12:19:13.652648 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:19:13.652623    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4ba4a439-2e5f-47e6-8154-882197640493-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-7c48986569-c7fxx\" (UID: \"4ba4a439-2e5f-47e6-8154-882197640493\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7c48986569-c7fxx"
May 25 12:19:13.652786 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:19:13.652654    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4ba4a439-2e5f-47e6-8154-882197640493-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-7c48986569-c7fxx\" (UID: \"4ba4a439-2e5f-47e6-8154-882197640493\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7c48986569-c7fxx"
May 25 12:19:13.652786 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:19:13.652680    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-lp6cq\" (UniqueName: \"kubernetes.io/projected/4ba4a439-2e5f-47e6-8154-882197640493-kube-api-access-lp6cq\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-7c48986569-c7fxx\" (UID: \"4ba4a439-2e5f-47e6-8154-882197640493\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7c48986569-c7fxx"
May 25 12:19:13.652786 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:19:13.652724    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4ba4a439-2e5f-47e6-8154-882197640493-cabundle-cert\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-7c48986569-c7fxx\" (UID: \"4ba4a439-2e5f-47e6-8154-882197640493\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7c48986569-c7fxx"
May 25 12:19:13.652935 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:19:13.652805    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4ba4a439-2e5f-47e6-8154-882197640493-isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-7c48986569-c7fxx\" (UID: \"4ba4a439-2e5f-47e6-8154-882197640493\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7c48986569-c7fxx"
May 25 12:19:13.653208 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:19:13.653181    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4ba4a439-2e5f-47e6-8154-882197640493-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-7c48986569-c7fxx\" (UID: \"4ba4a439-2e5f-47e6-8154-882197640493\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7c48986569-c7fxx"
May 25 12:19:13.653340 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:19:13.653315    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4ba4a439-2e5f-47e6-8154-882197640493-cabundle-cert\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-7c48986569-c7fxx\" (UID: \"4ba4a439-2e5f-47e6-8154-882197640493\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7c48986569-c7fxx"
May 25 12:19:13.653407 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:19:13.653391    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4ba4a439-2e5f-47e6-8154-882197640493-isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-7c48986569-c7fxx\" (UID: \"4ba4a439-2e5f-47e6-8154-882197640493\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7c48986569-c7fxx"
May 25 12:19:13.654931 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:19:13.654909    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4ba4a439-2e5f-47e6-8154-882197640493-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-7c48986569-c7fxx\" (UID: \"4ba4a439-2e5f-47e6-8154-882197640493\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7c48986569-c7fxx"
May 25 12:19:13.661415 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:19:13.661393    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lp6cq\" (UniqueName: \"kubernetes.io/projected/4ba4a439-2e5f-47e6-8154-882197640493-kube-api-access-lp6cq\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-7c48986569-c7fxx\" (UID: \"4ba4a439-2e5f-47e6-8154-882197640493\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7c48986569-c7fxx"
May 25 12:19:13.808565 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:19:13.808489    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7c48986569-c7fxx"
May 25 12:19:13.931722 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:19:13.931700    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7c48986569-c7fxx"]
May 25 12:19:13.934104 ip-10-0-136-227 kubenswrapper[2561]: W0525 12:19:13.934070    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ba4a439_2e5f_47e6_8154_882197640493.slice/crio-012fc448d60a5914efa8c8492a148d0cbdc3d310f84ddb8c81039470f7026629 WatchSource:0}: Error finding container 012fc448d60a5914efa8c8492a148d0cbdc3d310f84ddb8c81039470f7026629: Status 404 returned error can't find the container with id 012fc448d60a5914efa8c8492a148d0cbdc3d310f84ddb8c81039470f7026629
May 25 12:19:13.948058 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:19:13.948029    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7c48986569-c7fxx" event={"ID":"4ba4a439-2e5f-47e6-8154-882197640493","Type":"ContainerStarted","Data":"012fc448d60a5914efa8c8492a148d0cbdc3d310f84ddb8c81039470f7026629"}
May 25 12:19:13.949286 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:19:13.949250    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-6f99b9d644-7g7ld_6614cea8-75df-4ce2-8970-44928df2dda7/storage-initializer/1.log"
May 25 12:19:13.949381 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:19:13.949358    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6f99b9d644-7g7ld" event={"ID":"6614cea8-75df-4ce2-8970-44928df2dda7","Type":"ContainerDied","Data":"e1ab630b2d412fb61fc4e5af1cfdb0a6ab6e708f58d674d9542717c2ed9f57be"}
May 25 12:19:13.949431 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:19:13.949394    2561 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6f99b9d644-7g7ld"
May 25 12:19:13.949431 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:19:13.949406    2561 scope.go:117] "RemoveContainer" containerID="ed49a0363e701a1f470f6e60f543c79c646cf692548a1a09eebffe7864801af6"
May 25 12:19:13.998064 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:19:13.998043    2561 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6f99b9d644-7g7ld"]
May 25 12:19:14.002644 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:19:14.002620    2561 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-6f99b9d644-7g7ld"]
May 25 12:19:14.955553 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:19:14.955516    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7c48986569-c7fxx" event={"ID":"4ba4a439-2e5f-47e6-8154-882197640493","Type":"ContainerStarted","Data":"421f3fdc75263d2ae33a1f6c83d686ffc0b1834459bf98017e83aeaf11b228cd"}
May 25 12:19:15.117117 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:19:15.117090    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6614cea8-75df-4ce2-8970-44928df2dda7" path="/var/lib/kubelet/pods/6614cea8-75df-4ce2-8970-44928df2dda7/volumes"
May 25 12:19:15.961079 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:19:15.961047    2561 generic.go:358] "Generic (PLEG): container finished" podID="4ba4a439-2e5f-47e6-8154-882197640493" containerID="421f3fdc75263d2ae33a1f6c83d686ffc0b1834459bf98017e83aeaf11b228cd" exitCode=0
May 25 12:19:15.961079 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:19:15.961081    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7c48986569-c7fxx" event={"ID":"4ba4a439-2e5f-47e6-8154-882197640493","Type":"ContainerDied","Data":"421f3fdc75263d2ae33a1f6c83d686ffc0b1834459bf98017e83aeaf11b228cd"}
May 25 12:19:16.966847 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:19:16.966814    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7c48986569-c7fxx" event={"ID":"4ba4a439-2e5f-47e6-8154-882197640493","Type":"ContainerStarted","Data":"9238a203a5cec17a2dd359875208c145d0b2968e7c9e29ce98df4779594bcb5b"}
May 25 12:19:16.966847 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:19:16.966848    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7c48986569-c7fxx" event={"ID":"4ba4a439-2e5f-47e6-8154-882197640493","Type":"ContainerStarted","Data":"5b87a17c607e81573333f9d84ee2f16f7c00a6f56691a9b490f01d72bf9a3c3b"}
May 25 12:19:16.967290 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:19:16.966951    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7c48986569-c7fxx"
May 25 12:19:16.987281 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:19:16.987210    2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7c48986569-c7fxx" podStartSLOduration=3.987192773 podStartE2EDuration="3.987192773s" podCreationTimestamp="2026-05-25 12:19:13 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 12:19:16.9843678 +0000 UTC m=+3730.529545406" watchObservedRunningTime="2026-05-25 12:19:16.987192773 +0000 UTC m=+3730.532370379"
May 25 12:19:17.969899 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:19:17.969864    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7c48986569-c7fxx"
May 25 12:19:17.971190 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:19:17.971167    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7c48986569-c7fxx" podUID="4ba4a439-2e5f-47e6-8154-882197640493" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.61:8080: connect: connection refused"
May 25 12:19:18.973418 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:19:18.973375    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7c48986569-c7fxx" podUID="4ba4a439-2e5f-47e6-8154-882197640493" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.61:8080: connect: connection refused"
May 25 12:19:23.977569 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:19:23.977543    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7c48986569-c7fxx"
May 25 12:19:23.978025 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:19:23.977997    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7c48986569-c7fxx" podUID="4ba4a439-2e5f-47e6-8154-882197640493" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.61:8080: connect: connection refused"
May 25 12:19:33.978457 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:19:33.978415    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7c48986569-c7fxx" podUID="4ba4a439-2e5f-47e6-8154-882197640493" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.61:8080: connect: connection refused"
May 25 12:19:43.978049 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:19:43.978005    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7c48986569-c7fxx" podUID="4ba4a439-2e5f-47e6-8154-882197640493" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.61:8080: connect: connection refused"
May 25 12:19:53.978486 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:19:53.978448    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7c48986569-c7fxx" podUID="4ba4a439-2e5f-47e6-8154-882197640493" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.61:8080: connect: connection refused"
May 25 12:20:03.978294 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:20:03.978193    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7c48986569-c7fxx" podUID="4ba4a439-2e5f-47e6-8154-882197640493" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.61:8080: connect: connection refused"
May 25 12:20:13.978752 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:20:13.978711    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7c48986569-c7fxx" podUID="4ba4a439-2e5f-47e6-8154-882197640493" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.61:8080: connect: connection refused"
May 25 12:20:23.978940 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:20:23.978909    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7c48986569-c7fxx"
May 25 12:20:33.526410 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:20:33.526376    2561 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7c48986569-c7fxx"]
May 25 12:20:33.526907 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:20:33.526699    2561 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7c48986569-c7fxx" podUID="4ba4a439-2e5f-47e6-8154-882197640493" containerName="kserve-container" containerID="cri-o://5b87a17c607e81573333f9d84ee2f16f7c00a6f56691a9b490f01d72bf9a3c3b" gracePeriod=30
May 25 12:20:33.526907 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:20:33.526757    2561 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7c48986569-c7fxx" podUID="4ba4a439-2e5f-47e6-8154-882197640493" containerName="kube-rbac-proxy" containerID="cri-o://9238a203a5cec17a2dd359875208c145d0b2968e7c9e29ce98df4779594bcb5b" gracePeriod=30
May 25 12:20:33.974323 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:20:33.974284    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7c48986569-c7fxx" podUID="4ba4a439-2e5f-47e6-8154-882197640493" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.61:8643/healthz\": dial tcp 10.132.0.61:8643: connect: connection refused"
May 25 12:20:33.978918 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:20:33.978894    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7c48986569-c7fxx" podUID="4ba4a439-2e5f-47e6-8154-882197640493" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.61:8080: connect: connection refused"
May 25 12:20:34.217591 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:20:34.217561    2561 generic.go:358] "Generic (PLEG): container finished" podID="4ba4a439-2e5f-47e6-8154-882197640493" containerID="9238a203a5cec17a2dd359875208c145d0b2968e7c9e29ce98df4779594bcb5b" exitCode=2
May 25 12:20:34.217707 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:20:34.217601    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7c48986569-c7fxx" event={"ID":"4ba4a439-2e5f-47e6-8154-882197640493","Type":"ContainerDied","Data":"9238a203a5cec17a2dd359875208c145d0b2968e7c9e29ce98df4779594bcb5b"}
May 25 12:20:34.598509 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:20:34.598474    2561 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-bbc48787f-vlxgk"]
May 25 12:20:34.602190 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:20:34.602170    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-bbc48787f-vlxgk"
May 25 12:20:34.604453 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:20:34.604432    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-fail-predictor-serving-cert\""
May 25 12:20:34.604547 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:20:34.604454    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config\""
May 25 12:20:34.612633 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:20:34.612609    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-bbc48787f-vlxgk"]
May 25 12:20:34.698088 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:20:34.698055    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4731dcf7-0cc8-462a-918b-bfc36f489a36-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-bbc48787f-vlxgk\" (UID: \"4731dcf7-0cc8-462a-918b-bfc36f489a36\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-bbc48787f-vlxgk"
May 25 12:20:34.698316 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:20:34.698116    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g758k\" (UniqueName: \"kubernetes.io/projected/4731dcf7-0cc8-462a-918b-bfc36f489a36-kube-api-access-g758k\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-bbc48787f-vlxgk\" (UID: \"4731dcf7-0cc8-462a-918b-bfc36f489a36\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-bbc48787f-vlxgk"
May 25 12:20:34.698316 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:20:34.698187    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4731dcf7-0cc8-462a-918b-bfc36f489a36-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-bbc48787f-vlxgk\" (UID: \"4731dcf7-0cc8-462a-918b-bfc36f489a36\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-bbc48787f-vlxgk"
May 25 12:20:34.698316 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:20:34.698242    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4731dcf7-0cc8-462a-918b-bfc36f489a36-isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-bbc48787f-vlxgk\" (UID: \"4731dcf7-0cc8-462a-918b-bfc36f489a36\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-bbc48787f-vlxgk"
May 25 12:20:34.799628 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:20:34.799601    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4731dcf7-0cc8-462a-918b-bfc36f489a36-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-bbc48787f-vlxgk\" (UID: \"4731dcf7-0cc8-462a-918b-bfc36f489a36\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-bbc48787f-vlxgk"
May 25 12:20:34.799821 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:20:34.799647    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-g758k\" (UniqueName: \"kubernetes.io/projected/4731dcf7-0cc8-462a-918b-bfc36f489a36-kube-api-access-g758k\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-bbc48787f-vlxgk\" (UID: \"4731dcf7-0cc8-462a-918b-bfc36f489a36\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-bbc48787f-vlxgk"
May 25 12:20:34.799821 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:20:34.799673    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4731dcf7-0cc8-462a-918b-bfc36f489a36-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-bbc48787f-vlxgk\" (UID: \"4731dcf7-0cc8-462a-918b-bfc36f489a36\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-bbc48787f-vlxgk"
May 25 12:20:34.799821 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:20:34.799694    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4731dcf7-0cc8-462a-918b-bfc36f489a36-isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-bbc48787f-vlxgk\" (UID: \"4731dcf7-0cc8-462a-918b-bfc36f489a36\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-bbc48787f-vlxgk"
May 25 12:20:34.799821 ip-10-0-136-227 kubenswrapper[2561]: E0525 12:20:34.799760    2561 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-serving-cert: secret "isvc-sklearn-s3-tls-serving-fail-predictor-serving-cert" not found
May 25 12:20:34.800066 ip-10-0-136-227 kubenswrapper[2561]: E0525 12:20:34.799831    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4731dcf7-0cc8-462a-918b-bfc36f489a36-proxy-tls podName:4731dcf7-0cc8-462a-918b-bfc36f489a36 nodeName:}" failed. No retries permitted until 2026-05-25 12:20:35.299810434 +0000 UTC m=+3808.844988016 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/4731dcf7-0cc8-462a-918b-bfc36f489a36-proxy-tls") pod "isvc-sklearn-s3-tls-serving-fail-predictor-bbc48787f-vlxgk" (UID: "4731dcf7-0cc8-462a-918b-bfc36f489a36") : secret "isvc-sklearn-s3-tls-serving-fail-predictor-serving-cert" not found
May 25 12:20:34.800419 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:20:34.800389    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4731dcf7-0cc8-462a-918b-bfc36f489a36-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-bbc48787f-vlxgk\" (UID: \"4731dcf7-0cc8-462a-918b-bfc36f489a36\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-bbc48787f-vlxgk"
May 25 12:20:34.800527 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:20:34.800448    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4731dcf7-0cc8-462a-918b-bfc36f489a36-isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-bbc48787f-vlxgk\" (UID: \"4731dcf7-0cc8-462a-918b-bfc36f489a36\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-bbc48787f-vlxgk"
May 25 12:20:34.808661 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:20:34.808641    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g758k\" (UniqueName: \"kubernetes.io/projected/4731dcf7-0cc8-462a-918b-bfc36f489a36-kube-api-access-g758k\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-bbc48787f-vlxgk\" (UID: \"4731dcf7-0cc8-462a-918b-bfc36f489a36\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-bbc48787f-vlxgk"
May 25 12:20:35.305399 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:20:35.305367    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4731dcf7-0cc8-462a-918b-bfc36f489a36-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-bbc48787f-vlxgk\" (UID: \"4731dcf7-0cc8-462a-918b-bfc36f489a36\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-bbc48787f-vlxgk"
May 25 12:20:35.307565 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:20:35.307541    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4731dcf7-0cc8-462a-918b-bfc36f489a36-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-bbc48787f-vlxgk\" (UID: \"4731dcf7-0cc8-462a-918b-bfc36f489a36\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-bbc48787f-vlxgk"
May 25 12:20:35.513231 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:20:35.513194    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-bbc48787f-vlxgk"
May 25 12:20:35.641155 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:20:35.641119    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-bbc48787f-vlxgk"]
May 25 12:20:35.644709 ip-10-0-136-227 kubenswrapper[2561]: W0525 12:20:35.644677    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4731dcf7_0cc8_462a_918b_bfc36f489a36.slice/crio-5021d039fdc45707340376c1b6d3d13b639c9058b54a44cf15c0eaac3449b254 WatchSource:0}: Error finding container 5021d039fdc45707340376c1b6d3d13b639c9058b54a44cf15c0eaac3449b254: Status 404 returned error can't find the container with id 5021d039fdc45707340376c1b6d3d13b639c9058b54a44cf15c0eaac3449b254
May 25 12:20:35.646708 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:20:35.646691    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 25 12:20:36.227548 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:20:36.227510    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-bbc48787f-vlxgk" event={"ID":"4731dcf7-0cc8-462a-918b-bfc36f489a36","Type":"ContainerStarted","Data":"a29af3f9c7a927843a8d8954599e84376071c5dd2d16c5c1676ca98ec279c5d4"}
May 25 12:20:36.227548 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:20:36.227548    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-bbc48787f-vlxgk" event={"ID":"4731dcf7-0cc8-462a-918b-bfc36f489a36","Type":"ContainerStarted","Data":"5021d039fdc45707340376c1b6d3d13b639c9058b54a44cf15c0eaac3449b254"}
May 25 12:20:37.470232 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:20:37.470210    2561 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7c48986569-c7fxx"
May 25 12:20:37.627307 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:20:37.627263    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4ba4a439-2e5f-47e6-8154-882197640493-cabundle-cert\") pod \"4ba4a439-2e5f-47e6-8154-882197640493\" (UID: \"4ba4a439-2e5f-47e6-8154-882197640493\") "
May 25 12:20:37.627447 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:20:37.627313    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lp6cq\" (UniqueName: \"kubernetes.io/projected/4ba4a439-2e5f-47e6-8154-882197640493-kube-api-access-lp6cq\") pod \"4ba4a439-2e5f-47e6-8154-882197640493\" (UID: \"4ba4a439-2e5f-47e6-8154-882197640493\") "
May 25 12:20:37.627447 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:20:37.627339    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4ba4a439-2e5f-47e6-8154-882197640493-isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config\") pod \"4ba4a439-2e5f-47e6-8154-882197640493\" (UID: \"4ba4a439-2e5f-47e6-8154-882197640493\") "
May 25 12:20:37.627447 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:20:37.627415    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4ba4a439-2e5f-47e6-8154-882197640493-proxy-tls\") pod \"4ba4a439-2e5f-47e6-8154-882197640493\" (UID: \"4ba4a439-2e5f-47e6-8154-882197640493\") "
May 25 12:20:37.627447 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:20:37.627434    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4ba4a439-2e5f-47e6-8154-882197640493-kserve-provision-location\") pod \"4ba4a439-2e5f-47e6-8154-882197640493\" (UID: \"4ba4a439-2e5f-47e6-8154-882197640493\") "
May 25 12:20:37.627734 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:20:37.627708    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ba4a439-2e5f-47e6-8154-882197640493-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "4ba4a439-2e5f-47e6-8154-882197640493" (UID: "4ba4a439-2e5f-47e6-8154-882197640493"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 12:20:37.627862 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:20:37.627836    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ba4a439-2e5f-47e6-8154-882197640493-isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config") pod "4ba4a439-2e5f-47e6-8154-882197640493" (UID: "4ba4a439-2e5f-47e6-8154-882197640493"). InnerVolumeSpecName "isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 12:20:37.627922 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:20:37.627856    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ba4a439-2e5f-47e6-8154-882197640493-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4ba4a439-2e5f-47e6-8154-882197640493" (UID: "4ba4a439-2e5f-47e6-8154-882197640493"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 12:20:37.629473 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:20:37.629453    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ba4a439-2e5f-47e6-8154-882197640493-kube-api-access-lp6cq" (OuterVolumeSpecName: "kube-api-access-lp6cq") pod "4ba4a439-2e5f-47e6-8154-882197640493" (UID: "4ba4a439-2e5f-47e6-8154-882197640493"). InnerVolumeSpecName "kube-api-access-lp6cq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 12:20:37.629541 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:20:37.629498    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ba4a439-2e5f-47e6-8154-882197640493-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4ba4a439-2e5f-47e6-8154-882197640493" (UID: "4ba4a439-2e5f-47e6-8154-882197640493"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 12:20:37.728034 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:20:37.728008    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4ba4a439-2e5f-47e6-8154-882197640493-proxy-tls\") on node \"ip-10-0-136-227.ec2.internal\" DevicePath \"\""
May 25 12:20:37.728034 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:20:37.728033    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4ba4a439-2e5f-47e6-8154-882197640493-kserve-provision-location\") on node \"ip-10-0-136-227.ec2.internal\" DevicePath \"\""
May 25 12:20:37.728173 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:20:37.728047    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4ba4a439-2e5f-47e6-8154-882197640493-cabundle-cert\") on node \"ip-10-0-136-227.ec2.internal\" DevicePath \"\""
May 25 12:20:37.728173 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:20:37.728059    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lp6cq\" (UniqueName: \"kubernetes.io/projected/4ba4a439-2e5f-47e6-8154-882197640493-kube-api-access-lp6cq\") on node \"ip-10-0-136-227.ec2.internal\" DevicePath \"\""
May 25 12:20:37.728173 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:20:37.728074    2561 reconciler_common.go:299] "Volume detached for volume \"isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4ba4a439-2e5f-47e6-8154-882197640493-isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-227.ec2.internal\" DevicePath \"\""
May 25 12:20:38.235136 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:20:38.235098    2561 generic.go:358] "Generic (PLEG): container finished" podID="4ba4a439-2e5f-47e6-8154-882197640493" containerID="5b87a17c607e81573333f9d84ee2f16f7c00a6f56691a9b490f01d72bf9a3c3b" exitCode=0
May 25 12:20:38.235331 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:20:38.235174    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7c48986569-c7fxx" event={"ID":"4ba4a439-2e5f-47e6-8154-882197640493","Type":"ContainerDied","Data":"5b87a17c607e81573333f9d84ee2f16f7c00a6f56691a9b490f01d72bf9a3c3b"}
May 25 12:20:38.235331 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:20:38.235190    2561 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7c48986569-c7fxx"
May 25 12:20:38.235331 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:20:38.235209    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7c48986569-c7fxx" event={"ID":"4ba4a439-2e5f-47e6-8154-882197640493","Type":"ContainerDied","Data":"012fc448d60a5914efa8c8492a148d0cbdc3d310f84ddb8c81039470f7026629"}
May 25 12:20:38.235331 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:20:38.235224    2561 scope.go:117] "RemoveContainer" containerID="9238a203a5cec17a2dd359875208c145d0b2968e7c9e29ce98df4779594bcb5b"
May 25 12:20:38.243545 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:20:38.243531    2561 scope.go:117] "RemoveContainer" containerID="5b87a17c607e81573333f9d84ee2f16f7c00a6f56691a9b490f01d72bf9a3c3b"
May 25 12:20:38.250477 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:20:38.250461    2561 scope.go:117] "RemoveContainer" containerID="421f3fdc75263d2ae33a1f6c83d686ffc0b1834459bf98017e83aeaf11b228cd"
May 25 12:20:38.257974 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:20:38.257950    2561 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7c48986569-c7fxx"]
May 25 12:20:38.259069 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:20:38.259052    2561 scope.go:117] "RemoveContainer" containerID="9238a203a5cec17a2dd359875208c145d0b2968e7c9e29ce98df4779594bcb5b"
May 25 12:20:38.259349 ip-10-0-136-227 kubenswrapper[2561]: E0525 12:20:38.259325    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9238a203a5cec17a2dd359875208c145d0b2968e7c9e29ce98df4779594bcb5b\": container with ID starting with 9238a203a5cec17a2dd359875208c145d0b2968e7c9e29ce98df4779594bcb5b not found: ID does not exist" containerID="9238a203a5cec17a2dd359875208c145d0b2968e7c9e29ce98df4779594bcb5b"
May 25 12:20:38.259414 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:20:38.259360    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9238a203a5cec17a2dd359875208c145d0b2968e7c9e29ce98df4779594bcb5b"} err="failed to get container status \"9238a203a5cec17a2dd359875208c145d0b2968e7c9e29ce98df4779594bcb5b\": rpc error: code = NotFound desc = could not find container \"9238a203a5cec17a2dd359875208c145d0b2968e7c9e29ce98df4779594bcb5b\": container with ID starting with 9238a203a5cec17a2dd359875208c145d0b2968e7c9e29ce98df4779594bcb5b not found: ID does not exist"
May 25 12:20:38.259414 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:20:38.259384    2561 scope.go:117] "RemoveContainer" containerID="5b87a17c607e81573333f9d84ee2f16f7c00a6f56691a9b490f01d72bf9a3c3b"
May 25 12:20:38.259618 ip-10-0-136-227 kubenswrapper[2561]: E0525 12:20:38.259601    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b87a17c607e81573333f9d84ee2f16f7c00a6f56691a9b490f01d72bf9a3c3b\": container with ID starting with 5b87a17c607e81573333f9d84ee2f16f7c00a6f56691a9b490f01d72bf9a3c3b not found: ID does not exist" containerID="5b87a17c607e81573333f9d84ee2f16f7c00a6f56691a9b490f01d72bf9a3c3b"
May 25 12:20:38.259662 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:20:38.259625    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b87a17c607e81573333f9d84ee2f16f7c00a6f56691a9b490f01d72bf9a3c3b"} err="failed to get container status \"5b87a17c607e81573333f9d84ee2f16f7c00a6f56691a9b490f01d72bf9a3c3b\": rpc error: code = NotFound desc = could not find container \"5b87a17c607e81573333f9d84ee2f16f7c00a6f56691a9b490f01d72bf9a3c3b\": container with ID starting with 5b87a17c607e81573333f9d84ee2f16f7c00a6f56691a9b490f01d72bf9a3c3b not found: ID does not exist"
May 25 12:20:38.259662 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:20:38.259641    2561 scope.go:117] "RemoveContainer" containerID="421f3fdc75263d2ae33a1f6c83d686ffc0b1834459bf98017e83aeaf11b228cd"
May 25 12:20:38.259899 ip-10-0-136-227 kubenswrapper[2561]: E0525 12:20:38.259884    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"421f3fdc75263d2ae33a1f6c83d686ffc0b1834459bf98017e83aeaf11b228cd\": container with ID starting with 421f3fdc75263d2ae33a1f6c83d686ffc0b1834459bf98017e83aeaf11b228cd not found: ID does not exist" containerID="421f3fdc75263d2ae33a1f6c83d686ffc0b1834459bf98017e83aeaf11b228cd"
May 25 12:20:38.259950 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:20:38.259902    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"421f3fdc75263d2ae33a1f6c83d686ffc0b1834459bf98017e83aeaf11b228cd"} err="failed to get container status \"421f3fdc75263d2ae33a1f6c83d686ffc0b1834459bf98017e83aeaf11b228cd\": rpc error: code = NotFound desc = could not find container \"421f3fdc75263d2ae33a1f6c83d686ffc0b1834459bf98017e83aeaf11b228cd\": container with ID starting with 421f3fdc75263d2ae33a1f6c83d686ffc0b1834459bf98017e83aeaf11b228cd not found: ID does not exist"
May 25 12:20:38.260589 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:20:38.260572    2561 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7c48986569-c7fxx"]
May 25 12:20:39.116679 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:20:39.116648    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4ba4a439-2e5f-47e6-8154-882197640493" path="/var/lib/kubelet/pods/4ba4a439-2e5f-47e6-8154-882197640493/volumes"
May 25 12:20:40.245857 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:20:40.245834    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-bbc48787f-vlxgk_4731dcf7-0cc8-462a-918b-bfc36f489a36/storage-initializer/0.log"
May 25 12:20:40.246129 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:20:40.245867    2561 generic.go:358] "Generic (PLEG): container finished" podID="4731dcf7-0cc8-462a-918b-bfc36f489a36" containerID="a29af3f9c7a927843a8d8954599e84376071c5dd2d16c5c1676ca98ec279c5d4" exitCode=1
May 25 12:20:40.246129 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:20:40.245949    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-bbc48787f-vlxgk" event={"ID":"4731dcf7-0cc8-462a-918b-bfc36f489a36","Type":"ContainerDied","Data":"a29af3f9c7a927843a8d8954599e84376071c5dd2d16c5c1676ca98ec279c5d4"}
May 25 12:20:41.250216 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:20:41.250184    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-bbc48787f-vlxgk_4731dcf7-0cc8-462a-918b-bfc36f489a36/storage-initializer/0.log"
May 25 12:20:41.250634 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:20:41.250300    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-bbc48787f-vlxgk" event={"ID":"4731dcf7-0cc8-462a-918b-bfc36f489a36","Type":"ContainerStarted","Data":"6b38a813b28d4b8dcf9d4f8140bdfd5d4d6de858e8ecd8bff9cace4fc437f2c0"}
May 25 12:20:44.582850 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:20:44.582815    2561 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-bbc48787f-vlxgk"]
May 25 12:20:44.583245 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:20:44.583114    2561 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-bbc48787f-vlxgk" podUID="4731dcf7-0cc8-462a-918b-bfc36f489a36" containerName="storage-initializer" containerID="cri-o://6b38a813b28d4b8dcf9d4f8140bdfd5d4d6de858e8ecd8bff9cace4fc437f2c0" gracePeriod=30
May 25 12:20:45.523013 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:20:45.522993    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-bbc48787f-vlxgk_4731dcf7-0cc8-462a-918b-bfc36f489a36/storage-initializer/1.log"
May 25 12:20:45.523371 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:20:45.523356    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-bbc48787f-vlxgk_4731dcf7-0cc8-462a-918b-bfc36f489a36/storage-initializer/0.log"
May 25 12:20:45.523426 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:20:45.523419    2561 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-bbc48787f-vlxgk"
May 25 12:20:45.688096 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:20:45.688023    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4731dcf7-0cc8-462a-918b-bfc36f489a36-proxy-tls\") pod \"4731dcf7-0cc8-462a-918b-bfc36f489a36\" (UID: \"4731dcf7-0cc8-462a-918b-bfc36f489a36\") "
May 25 12:20:45.688464 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:20:45.688107    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g758k\" (UniqueName: \"kubernetes.io/projected/4731dcf7-0cc8-462a-918b-bfc36f489a36-kube-api-access-g758k\") pod \"4731dcf7-0cc8-462a-918b-bfc36f489a36\" (UID: \"4731dcf7-0cc8-462a-918b-bfc36f489a36\") "
May 25 12:20:45.688464 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:20:45.688132    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4731dcf7-0cc8-462a-918b-bfc36f489a36-kserve-provision-location\") pod \"4731dcf7-0cc8-462a-918b-bfc36f489a36\" (UID: \"4731dcf7-0cc8-462a-918b-bfc36f489a36\") "
May 25 12:20:45.688464 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:20:45.688167    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4731dcf7-0cc8-462a-918b-bfc36f489a36-isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config\") pod \"4731dcf7-0cc8-462a-918b-bfc36f489a36\" (UID: \"4731dcf7-0cc8-462a-918b-bfc36f489a36\") "
May 25 12:20:45.688464 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:20:45.688436    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4731dcf7-0cc8-462a-918b-bfc36f489a36-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4731dcf7-0cc8-462a-918b-bfc36f489a36" (UID: "4731dcf7-0cc8-462a-918b-bfc36f489a36"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 12:20:45.688629 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:20:45.688530    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4731dcf7-0cc8-462a-918b-bfc36f489a36-isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config") pod "4731dcf7-0cc8-462a-918b-bfc36f489a36" (UID: "4731dcf7-0cc8-462a-918b-bfc36f489a36"). InnerVolumeSpecName "isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 12:20:45.690035 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:20:45.690018    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4731dcf7-0cc8-462a-918b-bfc36f489a36-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4731dcf7-0cc8-462a-918b-bfc36f489a36" (UID: "4731dcf7-0cc8-462a-918b-bfc36f489a36"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 12:20:45.690113 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:20:45.690093    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4731dcf7-0cc8-462a-918b-bfc36f489a36-kube-api-access-g758k" (OuterVolumeSpecName: "kube-api-access-g758k") pod "4731dcf7-0cc8-462a-918b-bfc36f489a36" (UID: "4731dcf7-0cc8-462a-918b-bfc36f489a36"). InnerVolumeSpecName "kube-api-access-g758k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 12:20:45.789162 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:20:45.789137    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4731dcf7-0cc8-462a-918b-bfc36f489a36-proxy-tls\") on node \"ip-10-0-136-227.ec2.internal\" DevicePath \"\""
May 25 12:20:45.789162 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:20:45.789160    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g758k\" (UniqueName: \"kubernetes.io/projected/4731dcf7-0cc8-462a-918b-bfc36f489a36-kube-api-access-g758k\") on node \"ip-10-0-136-227.ec2.internal\" DevicePath \"\""
May 25 12:20:45.789394 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:20:45.789171    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4731dcf7-0cc8-462a-918b-bfc36f489a36-kserve-provision-location\") on node \"ip-10-0-136-227.ec2.internal\" DevicePath \"\""
May 25 12:20:45.789394 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:20:45.789181    2561 reconciler_common.go:299] "Volume detached for volume \"isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4731dcf7-0cc8-462a-918b-bfc36f489a36-isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config\") on node \"ip-10-0-136-227.ec2.internal\" DevicePath \"\""
May 25 12:20:46.268397 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:20:46.268353    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-bbc48787f-vlxgk_4731dcf7-0cc8-462a-918b-bfc36f489a36/storage-initializer/1.log"
May 25 12:20:46.268716 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:20:46.268694    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-bbc48787f-vlxgk_4731dcf7-0cc8-462a-918b-bfc36f489a36/storage-initializer/0.log"
May 25 12:20:46.268780 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:20:46.268730    2561 generic.go:358] "Generic (PLEG): container finished" podID="4731dcf7-0cc8-462a-918b-bfc36f489a36" containerID="6b38a813b28d4b8dcf9d4f8140bdfd5d4d6de858e8ecd8bff9cace4fc437f2c0" exitCode=1
May 25 12:20:46.268835 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:20:46.268808    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-bbc48787f-vlxgk" event={"ID":"4731dcf7-0cc8-462a-918b-bfc36f489a36","Type":"ContainerDied","Data":"6b38a813b28d4b8dcf9d4f8140bdfd5d4d6de858e8ecd8bff9cace4fc437f2c0"}
May 25 12:20:46.268835 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:20:46.268820    2561 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-bbc48787f-vlxgk"
May 25 12:20:46.268910 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:20:46.268842    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-bbc48787f-vlxgk" event={"ID":"4731dcf7-0cc8-462a-918b-bfc36f489a36","Type":"ContainerDied","Data":"5021d039fdc45707340376c1b6d3d13b639c9058b54a44cf15c0eaac3449b254"}
May 25 12:20:46.268910 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:20:46.268862    2561 scope.go:117] "RemoveContainer" containerID="6b38a813b28d4b8dcf9d4f8140bdfd5d4d6de858e8ecd8bff9cace4fc437f2c0"
May 25 12:20:46.278100 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:20:46.278081    2561 scope.go:117] "RemoveContainer" containerID="a29af3f9c7a927843a8d8954599e84376071c5dd2d16c5c1676ca98ec279c5d4"
May 25 12:20:46.285775 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:20:46.285756    2561 scope.go:117] "RemoveContainer" containerID="6b38a813b28d4b8dcf9d4f8140bdfd5d4d6de858e8ecd8bff9cace4fc437f2c0"
May 25 12:20:46.286022 ip-10-0-136-227 kubenswrapper[2561]: E0525 12:20:46.286003    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b38a813b28d4b8dcf9d4f8140bdfd5d4d6de858e8ecd8bff9cace4fc437f2c0\": container with ID starting with 6b38a813b28d4b8dcf9d4f8140bdfd5d4d6de858e8ecd8bff9cace4fc437f2c0 not found: ID does not exist" containerID="6b38a813b28d4b8dcf9d4f8140bdfd5d4d6de858e8ecd8bff9cace4fc437f2c0"
May 25 12:20:46.286103 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:20:46.286035    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b38a813b28d4b8dcf9d4f8140bdfd5d4d6de858e8ecd8bff9cace4fc437f2c0"} err="failed to get container status \"6b38a813b28d4b8dcf9d4f8140bdfd5d4d6de858e8ecd8bff9cace4fc437f2c0\": rpc error: code = NotFound desc = could not find container \"6b38a813b28d4b8dcf9d4f8140bdfd5d4d6de858e8ecd8bff9cace4fc437f2c0\": container with ID starting with 6b38a813b28d4b8dcf9d4f8140bdfd5d4d6de858e8ecd8bff9cace4fc437f2c0 not found: ID does not exist"
May 25 12:20:46.286103 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:20:46.286073    2561 scope.go:117] "RemoveContainer" containerID="a29af3f9c7a927843a8d8954599e84376071c5dd2d16c5c1676ca98ec279c5d4"
May 25 12:20:46.286353 ip-10-0-136-227 kubenswrapper[2561]: E0525 12:20:46.286335    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a29af3f9c7a927843a8d8954599e84376071c5dd2d16c5c1676ca98ec279c5d4\": container with ID starting with a29af3f9c7a927843a8d8954599e84376071c5dd2d16c5c1676ca98ec279c5d4 not found: ID does not exist" containerID="a29af3f9c7a927843a8d8954599e84376071c5dd2d16c5c1676ca98ec279c5d4"
May 25 12:20:46.286401 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:20:46.286359    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a29af3f9c7a927843a8d8954599e84376071c5dd2d16c5c1676ca98ec279c5d4"} err="failed to get container status \"a29af3f9c7a927843a8d8954599e84376071c5dd2d16c5c1676ca98ec279c5d4\": rpc error: code = NotFound desc = could not find container \"a29af3f9c7a927843a8d8954599e84376071c5dd2d16c5c1676ca98ec279c5d4\": container with ID starting with a29af3f9c7a927843a8d8954599e84376071c5dd2d16c5c1676ca98ec279c5d4 not found: ID does not exist"
May 25 12:20:46.305295 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:20:46.305253    2561 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-bbc48787f-vlxgk"]
May 25 12:20:46.311432 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:20:46.311407    2561 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-bbc48787f-vlxgk"]
May 25 12:20:47.117532 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:20:47.117503    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4731dcf7-0cc8-462a-918b-bfc36f489a36" path="/var/lib/kubelet/pods/4731dcf7-0cc8-462a-918b-bfc36f489a36/volumes"
May 25 12:21:21.144283 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:21:21.144241    2561 ???:1] "http: TLS handshake error from 10.0.136.227:60244: EOF"
May 25 12:21:21.156077 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:21:21.156039    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-dq8s4_3e92c2a7-1021-4ee8-aded-8ae5ddc6b81c/global-pull-secret-syncer/0.log"
May 25 12:21:21.364130 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:21:21.364101    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-mr48p_ae10dbbc-8226-48df-a0f7-39aed3d95c58/konnectivity-agent/0.log"
May 25 12:21:21.441389 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:21:21.441312    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-136-227.ec2.internal_f96ecd3e8e97719fd70e0de354f8eb4d/haproxy/0.log"
May 25 12:21:25.154177 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:21:25.154149    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5c807139-8565-45a2-98d3-aed4e9ba4d37/alertmanager/0.log"
May 25 12:21:25.177736 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:21:25.177710    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5c807139-8565-45a2-98d3-aed4e9ba4d37/config-reloader/0.log"
May 25 12:21:25.200612 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:21:25.200587    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5c807139-8565-45a2-98d3-aed4e9ba4d37/kube-rbac-proxy-web/0.log"
May 25 12:21:25.223520 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:21:25.223497    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5c807139-8565-45a2-98d3-aed4e9ba4d37/kube-rbac-proxy/0.log"
May 25 12:21:25.242413 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:21:25.242396    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5c807139-8565-45a2-98d3-aed4e9ba4d37/kube-rbac-proxy-metric/0.log"
May 25 12:21:25.263465 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:21:25.263450    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5c807139-8565-45a2-98d3-aed4e9ba4d37/prom-label-proxy/0.log"
May 25 12:21:25.284924 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:21:25.284901    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5c807139-8565-45a2-98d3-aed4e9ba4d37/init-config-reloader/0.log"
May 25 12:21:25.342231 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:21:25.342211    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6c75959455-tfzq9_82fb3b0a-798a-406b-99b4-548782582352/cluster-monitoring-operator/0.log"
May 25 12:21:25.373216 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:21:25.373184    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-54fd6bc5c4-n7x6m_4efe0bc1-af7a-4e08-bb46-9cac545bc309/kube-state-metrics/0.log"
May 25 12:21:25.390062 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:21:25.390041    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-54fd6bc5c4-n7x6m_4efe0bc1-af7a-4e08-bb46-9cac545bc309/kube-rbac-proxy-main/0.log"
May 25 12:21:25.408214 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:21:25.408155    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-54fd6bc5c4-n7x6m_4efe0bc1-af7a-4e08-bb46-9cac545bc309/kube-rbac-proxy-self/0.log"
May 25 12:21:25.579830 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:21:25.579808    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-r8hl4_744866a4-40aa-4412-9df2-8d1a2f4d1ea2/node-exporter/0.log"
May 25 12:21:25.599807 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:21:25.599787    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-r8hl4_744866a4-40aa-4412-9df2-8d1a2f4d1ea2/kube-rbac-proxy/0.log"
May 25 12:21:25.622053 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:21:25.622036    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-r8hl4_744866a4-40aa-4412-9df2-8d1a2f4d1ea2/init-textfile/0.log"
May 25 12:21:26.148839 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:21:26.148803    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5fc4d84cc6-jh48n_0df23aa9-66a4-415d-9c86-402e09fc814a/telemeter-client/0.log"
May 25 12:21:26.174179 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:21:26.174159    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5fc4d84cc6-jh48n_0df23aa9-66a4-415d-9c86-402e09fc814a/reload/0.log"
May 25 12:21:26.196414 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:21:26.196394    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5fc4d84cc6-jh48n_0df23aa9-66a4-415d-9c86-402e09fc814a/kube-rbac-proxy/0.log"
May 25 12:21:27.359690 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:21:27.359662    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-d4d8f94f9-8ktvq_1aff1fe9-36dd-4aed-8407-73bf7f9c439e/networking-console-plugin/0.log"
May 25 12:21:27.720618 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:21:27.720523    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-9lj95_10a66344-3304-49a1-bbc1-b885800f5acf/console-operator/1.log"
May 25 12:21:27.728858 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:21:27.728836    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-9lj95_10a66344-3304-49a1-bbc1-b885800f5acf/console-operator/2.log"
May 25 12:21:28.054922 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:21:28.054842    2561 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-must-gather-7t57z/perf-node-gather-daemonset-924fd"]
May 25 12:21:28.055181 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:21:28.055169    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ba4a439-2e5f-47e6-8154-882197640493" containerName="storage-initializer"
May 25 12:21:28.055225 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:21:28.055182    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ba4a439-2e5f-47e6-8154-882197640493" containerName="storage-initializer"
May 25 12:21:28.055225 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:21:28.055191    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ba4a439-2e5f-47e6-8154-882197640493" containerName="kube-rbac-proxy"
May 25 12:21:28.055225 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:21:28.055196    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ba4a439-2e5f-47e6-8154-882197640493" containerName="kube-rbac-proxy"
May 25 12:21:28.055225 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:21:28.055202    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4731dcf7-0cc8-462a-918b-bfc36f489a36" containerName="storage-initializer"
May 25 12:21:28.055225 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:21:28.055209    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="4731dcf7-0cc8-462a-918b-bfc36f489a36" containerName="storage-initializer"
May 25 12:21:28.055225 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:21:28.055223    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ba4a439-2e5f-47e6-8154-882197640493" containerName="kserve-container"
May 25 12:21:28.055441 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:21:28.055229    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ba4a439-2e5f-47e6-8154-882197640493" containerName="kserve-container"
May 25 12:21:28.055441 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:21:28.055307    2561 memory_manager.go:356] "RemoveStaleState removing state" podUID="4ba4a439-2e5f-47e6-8154-882197640493" containerName="kserve-container"
May 25 12:21:28.055441 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:21:28.055317    2561 memory_manager.go:356] "RemoveStaleState removing state" podUID="4ba4a439-2e5f-47e6-8154-882197640493" containerName="kube-rbac-proxy"
May 25 12:21:28.055441 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:21:28.055324    2561 memory_manager.go:356] "RemoveStaleState removing state" podUID="4731dcf7-0cc8-462a-918b-bfc36f489a36" containerName="storage-initializer"
May 25 12:21:28.058261 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:21:28.058246    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-7t57z/perf-node-gather-daemonset-924fd"
May 25 12:21:28.060785 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:21:28.060744    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-7t57z\"/\"openshift-service-ca.crt\""
May 25 12:21:28.060951 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:21:28.060799    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-7t57z\"/\"kube-root-ca.crt\""
May 25 12:21:28.061733 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:21:28.061695    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-7t57z\"/\"default-dockercfg-5jlg4\""
May 25 12:21:28.066835 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:21:28.066814    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-7t57z/perf-node-gather-daemonset-924fd"]
May 25 12:21:28.097743 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:21:28.097723    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-78c764b57c-98fm8_db40b82d-e321-4af8-925c-710fff638c2d/console/0.log"
May 25 12:21:28.109226 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:21:28.109201    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/40ab5826-9d8f-4d61-a610-f8c10aa1ed7a-proc\") pod \"perf-node-gather-daemonset-924fd\" (UID: \"40ab5826-9d8f-4d61-a610-f8c10aa1ed7a\") " pod="openshift-must-gather-7t57z/perf-node-gather-daemonset-924fd"
May 25 12:21:28.109359 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:21:28.109234    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/40ab5826-9d8f-4d61-a610-f8c10aa1ed7a-lib-modules\") pod \"perf-node-gather-daemonset-924fd\" (UID: \"40ab5826-9d8f-4d61-a610-f8c10aa1ed7a\") " pod="openshift-must-gather-7t57z/perf-node-gather-daemonset-924fd"
May 25 12:21:28.109359 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:21:28.109260    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4mtg\" (UniqueName: \"kubernetes.io/projected/40ab5826-9d8f-4d61-a610-f8c10aa1ed7a-kube-api-access-v4mtg\") pod \"perf-node-gather-daemonset-924fd\" (UID: \"40ab5826-9d8f-4d61-a610-f8c10aa1ed7a\") " pod="openshift-must-gather-7t57z/perf-node-gather-daemonset-924fd"
May 25 12:21:28.109359 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:21:28.109336    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/40ab5826-9d8f-4d61-a610-f8c10aa1ed7a-sys\") pod \"perf-node-gather-daemonset-924fd\" (UID: \"40ab5826-9d8f-4d61-a610-f8c10aa1ed7a\") " pod="openshift-must-gather-7t57z/perf-node-gather-daemonset-924fd"
May 25 12:21:28.109513 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:21:28.109393    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/40ab5826-9d8f-4d61-a610-f8c10aa1ed7a-podres\") pod \"perf-node-gather-daemonset-924fd\" (UID: \"40ab5826-9d8f-4d61-a610-f8c10aa1ed7a\") " pod="openshift-must-gather-7t57z/perf-node-gather-daemonset-924fd"
May 25 12:21:28.210416 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:21:28.210392    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/40ab5826-9d8f-4d61-a610-f8c10aa1ed7a-proc\") pod \"perf-node-gather-daemonset-924fd\" (UID: \"40ab5826-9d8f-4d61-a610-f8c10aa1ed7a\") " pod="openshift-must-gather-7t57z/perf-node-gather-daemonset-924fd"
May 25 12:21:28.210581 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:21:28.210421    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/40ab5826-9d8f-4d61-a610-f8c10aa1ed7a-lib-modules\") pod \"perf-node-gather-daemonset-924fd\" (UID: \"40ab5826-9d8f-4d61-a610-f8c10aa1ed7a\") " pod="openshift-must-gather-7t57z/perf-node-gather-daemonset-924fd"
May 25 12:21:28.210581 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:21:28.210437    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-v4mtg\" (UniqueName: \"kubernetes.io/projected/40ab5826-9d8f-4d61-a610-f8c10aa1ed7a-kube-api-access-v4mtg\") pod \"perf-node-gather-daemonset-924fd\" (UID: \"40ab5826-9d8f-4d61-a610-f8c10aa1ed7a\") " pod="openshift-must-gather-7t57z/perf-node-gather-daemonset-924fd"
May 25 12:21:28.210581 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:21:28.210466    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/40ab5826-9d8f-4d61-a610-f8c10aa1ed7a-sys\") pod \"perf-node-gather-daemonset-924fd\" (UID: \"40ab5826-9d8f-4d61-a610-f8c10aa1ed7a\") " pod="openshift-must-gather-7t57z/perf-node-gather-daemonset-924fd"
May 25 12:21:28.210581 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:21:28.210495    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/40ab5826-9d8f-4d61-a610-f8c10aa1ed7a-proc\") pod \"perf-node-gather-daemonset-924fd\" (UID: \"40ab5826-9d8f-4d61-a610-f8c10aa1ed7a\") " pod="openshift-must-gather-7t57z/perf-node-gather-daemonset-924fd"
May 25 12:21:28.210581 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:21:28.210516    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/40ab5826-9d8f-4d61-a610-f8c10aa1ed7a-podres\") pod \"perf-node-gather-daemonset-924fd\" (UID: \"40ab5826-9d8f-4d61-a610-f8c10aa1ed7a\") " pod="openshift-must-gather-7t57z/perf-node-gather-daemonset-924fd"
May 25 12:21:28.210812 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:21:28.210579    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/40ab5826-9d8f-4d61-a610-f8c10aa1ed7a-lib-modules\") pod \"perf-node-gather-daemonset-924fd\" (UID: \"40ab5826-9d8f-4d61-a610-f8c10aa1ed7a\") " pod="openshift-must-gather-7t57z/perf-node-gather-daemonset-924fd"
May 25 12:21:28.210812 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:21:28.210691    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/40ab5826-9d8f-4d61-a610-f8c10aa1ed7a-sys\") pod \"perf-node-gather-daemonset-924fd\" (UID: \"40ab5826-9d8f-4d61-a610-f8c10aa1ed7a\") " pod="openshift-must-gather-7t57z/perf-node-gather-daemonset-924fd"
May 25 12:21:28.210812 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:21:28.210704    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/40ab5826-9d8f-4d61-a610-f8c10aa1ed7a-podres\") pod \"perf-node-gather-daemonset-924fd\" (UID: \"40ab5826-9d8f-4d61-a610-f8c10aa1ed7a\") " pod="openshift-must-gather-7t57z/perf-node-gather-daemonset-924fd"
May 25 12:21:28.225393 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:21:28.225364    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4mtg\" (UniqueName: \"kubernetes.io/projected/40ab5826-9d8f-4d61-a610-f8c10aa1ed7a-kube-api-access-v4mtg\") pod \"perf-node-gather-daemonset-924fd\" (UID: \"40ab5826-9d8f-4d61-a610-f8c10aa1ed7a\") " pod="openshift-must-gather-7t57z/perf-node-gather-daemonset-924fd"
May 25 12:21:28.368952 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:21:28.368923    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-7t57z/perf-node-gather-daemonset-924fd"
May 25 12:21:28.493969 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:21:28.493939    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-7t57z/perf-node-gather-daemonset-924fd"]
May 25 12:21:28.496643 ip-10-0-136-227 kubenswrapper[2561]: W0525 12:21:28.496608    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod40ab5826_9d8f_4d61_a610_f8c10aa1ed7a.slice/crio-852847af839b7d63df25a88a5486d59baec494d362381c9c336c8e2d020d76cb WatchSource:0}: Error finding container 852847af839b7d63df25a88a5486d59baec494d362381c9c336c8e2d020d76cb: Status 404 returned error can't find the container with id 852847af839b7d63df25a88a5486d59baec494d362381c9c336c8e2d020d76cb
May 25 12:21:28.593387 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:21:28.593353    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-5cf8c89d64-fwgjs_aa041c38-250b-4342-9951-7f04c9daf30c/volume-data-source-validator/0.log"
May 25 12:21:29.247203 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:21:29.247173    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-d6xph_ffe11530-9960-4b99-862e-093a4377bd45/dns/0.log"
May 25 12:21:29.266548 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:21:29.266523    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-d6xph_ffe11530-9960-4b99-862e-093a4377bd45/kube-rbac-proxy/0.log"
May 25 12:21:29.369891 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:21:29.369869    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-xvcpl_b55f6df2-1dbc-4fc5-a4e6-e679d289e9f5/dns-node-resolver/0.log"
May 25 12:21:29.407743 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:21:29.407721    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7t57z/perf-node-gather-daemonset-924fd" event={"ID":"40ab5826-9d8f-4d61-a610-f8c10aa1ed7a","Type":"ContainerStarted","Data":"822df3b3db0788cca6e52f9d3aee73b32068864b5604215e4d6c2dbd821d2718"}
May 25 12:21:29.407864 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:21:29.407747    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7t57z/perf-node-gather-daemonset-924fd" event={"ID":"40ab5826-9d8f-4d61-a610-f8c10aa1ed7a","Type":"ContainerStarted","Data":"852847af839b7d63df25a88a5486d59baec494d362381c9c336c8e2d020d76cb"}
May 25 12:21:29.407864 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:21:29.407769    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-7t57z/perf-node-gather-daemonset-924fd"
May 25 12:21:29.424108 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:21:29.424065    2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-must-gather-7t57z/perf-node-gather-daemonset-924fd" podStartSLOduration=1.424052892 podStartE2EDuration="1.424052892s" podCreationTimestamp="2026-05-25 12:21:28 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 12:21:29.422223275 +0000 UTC m=+3862.967400880" watchObservedRunningTime="2026-05-25 12:21:29.424052892 +0000 UTC m=+3862.969230535"
May 25 12:21:29.833834 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:21:29.833809    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-q8ng6_9a250955-746b-416a-b798-283a0ea46864/node-ca/0.log"
May 25 12:21:30.501971 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:21:30.501940    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-846bb98c44-b4vb2_c6cac71b-58b3-43ba-8a3a-efb15e35c401/router/0.log"
May 25 12:21:30.792942 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:21:30.792869    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-66tdv_039e2ad5-6518-454a-99b9-33221aaef30a/serve-healthcheck-canary/0.log"
May 25 12:21:31.159777 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:21:31.159744    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-78bdfb888c-nd5f5_13894946-8385-4b6f-a82a-4188f46d576c/insights-operator/0.log"
May 25 12:21:31.161310 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:21:31.161262    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-78bdfb888c-nd5f5_13894946-8385-4b6f-a82a-4188f46d576c/insights-operator/1.log"
May 25 12:21:31.180224 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:21:31.180203    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-7nvl9_c18099d5-e3a6-4e23-b792-8fc91dcb9343/kube-rbac-proxy/0.log"
May 25 12:21:31.197630 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:21:31.197607    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-7nvl9_c18099d5-e3a6-4e23-b792-8fc91dcb9343/exporter/0.log"
May 25 12:21:31.215958 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:21:31.215940    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-7nvl9_c18099d5-e3a6-4e23-b792-8fc91dcb9343/extractor/0.log"
May 25 12:21:33.370067 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:21:33.370040    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_kserve-controller-manager-559cc6dd6f-bksd2_199e7c1d-de93-4bcb-98b9-5d91f052f6d4/manager/0.log"
May 25 12:21:33.811449 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:21:33.811373    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-696fc77849-8hh2g_43d9a307-9385-49d0-b3cf-258c9ffdc164/manager/0.log"
May 25 12:21:33.899332 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:21:33.899291    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-79f457656b-p72l9_e3cf2ca6-1c50-43b7-a6a3-7009420c743c/seaweedfs/0.log"
May 25 12:21:33.921449 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:21:33.921425    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-tls-custom-5c88b85bb7-mhx2w_e9009c16-ff47-44bc-93c9-0deeeacf1c71/seaweedfs-tls-custom/0.log"
May 25 12:21:35.420024 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:21:35.420000    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-7t57z/perf-node-gather-daemonset-924fd"
May 25 12:21:38.445604 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:21:38.445529    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-565644f49-88l9x_9b9cd68b-6d33-40ae-b547-43f4ef7a2f5c/kube-storage-version-migrator-operator/1.log"
May 25 12:21:38.446949 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:21:38.446902    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-565644f49-88l9x_9b9cd68b-6d33-40ae-b547-43f4ef7a2f5c/kube-storage-version-migrator-operator/0.log"
May 25 12:21:39.751325 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:21:39.751297    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dmzql_7c97daaa-b9c1-4ee0-ade1-e8de392e496f/kube-multus-additional-cni-plugins/0.log"
May 25 12:21:39.769631 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:21:39.769607    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dmzql_7c97daaa-b9c1-4ee0-ade1-e8de392e496f/egress-router-binary-copy/0.log"
May 25 12:21:39.791566 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:21:39.791504    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dmzql_7c97daaa-b9c1-4ee0-ade1-e8de392e496f/cni-plugins/0.log"
May 25 12:21:39.811997 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:21:39.811979    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dmzql_7c97daaa-b9c1-4ee0-ade1-e8de392e496f/bond-cni-plugin/0.log"
May 25 12:21:39.830988 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:21:39.830963    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dmzql_7c97daaa-b9c1-4ee0-ade1-e8de392e496f/routeoverride-cni/0.log"
May 25 12:21:39.850004 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:21:39.849983    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dmzql_7c97daaa-b9c1-4ee0-ade1-e8de392e496f/whereabouts-cni-bincopy/0.log"
May 25 12:21:39.869233 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:21:39.869208    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dmzql_7c97daaa-b9c1-4ee0-ade1-e8de392e496f/whereabouts-cni/0.log"
May 25 12:21:39.909019 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:21:39.909004    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-bxzz6_a3d75818-4ca3-4bb1-a3dd-28ae4ce71d09/kube-multus/0.log"
May 25 12:21:39.986981 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:21:39.986962    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-444mb_ead20bcd-df93-4d45-b5bb-470a8fe373e7/network-metrics-daemon/0.log"
May 25 12:21:40.001123 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:21:40.001103    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-444mb_ead20bcd-df93-4d45-b5bb-470a8fe373e7/kube-rbac-proxy/0.log"
May 25 12:21:41.558984 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:21:41.558952    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-thvg5_5b2bf5ae-a8a4-48e9-9afc-ee576988791e/ovn-controller/0.log"
May 25 12:21:41.611479 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:21:41.611455    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-thvg5_5b2bf5ae-a8a4-48e9-9afc-ee576988791e/ovn-acl-logging/0.log"
May 25 12:21:41.631775 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:21:41.631757    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-thvg5_5b2bf5ae-a8a4-48e9-9afc-ee576988791e/kube-rbac-proxy-node/0.log"
May 25 12:21:41.654900 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:21:41.654884    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-thvg5_5b2bf5ae-a8a4-48e9-9afc-ee576988791e/kube-rbac-proxy-ovn-metrics/0.log"
May 25 12:21:41.678648 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:21:41.678628    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-thvg5_5b2bf5ae-a8a4-48e9-9afc-ee576988791e/northd/0.log"
May 25 12:21:41.699346 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:21:41.699323    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-thvg5_5b2bf5ae-a8a4-48e9-9afc-ee576988791e/nbdb/0.log"
May 25 12:21:41.720168 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:21:41.720140    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-thvg5_5b2bf5ae-a8a4-48e9-9afc-ee576988791e/sbdb/0.log"
May 25 12:21:41.897508 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:21:41.897486    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-thvg5_5b2bf5ae-a8a4-48e9-9afc-ee576988791e/ovnkube-controller/0.log"
May 25 12:21:42.818661 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:21:42.818609    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-source-7f44b6dd87-7b6rg_bf1cc7fa-a86f-43ee-ab68-3b6b57050611/check-endpoints/0.log"
May 25 12:21:42.898127 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:21:42.898099    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-rl7hd_2b05b78f-6ab1-4f94-8273-e5fa70b78d15/network-check-target-container/0.log"
May 25 12:21:43.742124 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:21:43.742097    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-4xz76_a8a3a45a-3dba-4919-895a-ef467fbd4343/iptables-alerter/0.log"
May 25 12:21:44.439441 ip-10-0-136-227 kubenswrapper[2561]: I0525 12:21:44.439414    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-k2fsn_b0e8a1a4-c0cb-443e-8c38-9abcf242fc84/tuned/0.log"